2026年10月13日，Google DeepMind 首席软件工程师 Kanishka Rao 发布技术报告，介绍 Gemini 4 Argon 在机器人物理理解、代码生成及视觉语义推理方面的进展。
在 Embodied Reasoning (ERQA) 基准测试中，该模型展示了其在三维环境中解决物理空间问题的能力。仅依靠纯推理机制，Argon 取得了 87% 的得分，超越了其他前沿模型。
作为一个通用的物理推理模型，Argon 可应用于机器人生命周期的多个环节。以下展示其在四个关键领域的代表性应用：
- 机器人控制：涵盖移动操作与 MuJoCo 仿真实践，以及零样本机器人控制和视频世界模型的闭环转向。
- Real2Sim：从机器人摄像头画面、第一人称人类视频或手绘草图中重建经过物理校准的数字孪生模拟环境。
- 物理理解：包括代理的多粒度后视标注，涉及对灵巧度、密度值函数及任务进度的详细解析。
- 研究与自我改进：通过在仿真环境和硬件上的自我迭代实现行动优化。
基于 Argon 的机器人控制
1. 现场移动操作：MuJoCo 辅助下的脚踏式垃圾桶投放
Argon 控制一台配备 12 自由度腿部及 6 自由度手臂的波士顿动力 Spot® 四足机器人，学习新技能：踩踏脚踏式垃圾桶并丢弃苏打罐。在执行动作前，Argon 构建了一个脚踏式垃圾桶的关节化 MuJoCo 数字双胞胎，随后完成踩踏板及投放动作。
Spot Mobile Manipulation — Real-to-Sim-to-Real foot-pedal trash bin disposal: Spot picking up the soda can, checking its wrist camera, practicing 3-leg tripod balance in MuJoCo, stepping on the foot pedal, verifying hand centering in the open bin, and dropping the can inside.Prompt:
There's a Diet Coke can behind you and another on the black table on your left.
Use the spot-mobile-manipulation skill to pick them up and trash them in the black foot-pedal bin.
Before stepping down on the pedal, verify in the front cameras that your front-right foot target is centered on the pedal, and use a 2-agent verification swarm (foot-verifier and hand-verifier) to verify both the foot-pedal step and the 3D hand-in-bin alignment before releasing.2. 零样本控制
通过视觉坐标带和动作原语，Argon 控制一个简单的机械臂执行多轮任务：将倒下的杯子扶正，并将餐巾放入杯中。
Left: 3rd-person real robot view; Right: calibrated wrist camera with projected 2 cm world coordinates and jaw-closing reticle.Prompt:
[Attached: wrist_cam.png, obs.json]
Task: Get the robot arm to reorient the mug upright and put the napkin inside the mug.
## Execution Discipline
- Simply output 1 sentence observation followed immediately by your `curl` command (with `# Plan:` on Turn 1 and `# Thought:` on every turn). Everything needed is in `obs.json` and the calibrated reticle/grid drawn on `wrist_cam.png`.
## Visual Coordinate Reference (Annotated on wrist_cam.png)
- `+X` (UP in image): Away from robot base | `-X` (DOWN): Towards robot base
- `+Y` (LEFT in image): World left | `-Y` (RIGHT): World right
- `+Z`: Height above table | `Yellow numbers`: World (X, Y) in centimetres (2 cm grid at `grid_z`)
- `Cyan + crosshair`: Current 3D pinch center
- `Cyan [ + ] brackets`: Current jaw opening and closing line
- `Green line`: Current object long-axis orientation (`yaw_deg`: 0° = vertical along +X, 90° = horizontal along +Y)
- `pitch_deg = 90°`: Straight down (-Z) top-down grasp/place; `pitch_deg < 60°`: Tilted coaxial approach along finger axis
## Closed-Loop Turn Workflow
1. Inspect current state & view `wrist_cam.png`:
`curl -s http://127.0.0.1:8081/obs` (returns `ee_pos`, `holding`, `status`, `workspace`, `wrist_cam_path`).
2. Select VLM call granularity upfront on Turn 1:
- Coarse (2–4 calls): `look_at` -> `grasp` -> `place` -> `done`
- Pre-Grasp Hover Verify (4–7 calls): `look_at` -> `goto(z=0.04)` (verify cyan `[ + ]` brackets straddle target) -> `grasp` -> `place_until_contact` -> `done`
- Fine Visual Servoing (8–20 calls): `look_at` -> `grasp` -> pre-pose `goto` -> closed-loop `move_by` / `move_until_contact` -> `done`
3. Send motion primitives (`POST /plan`) and verify visually after every turn in the updated `wrist_cam.png`. If `status == "grasp_failed"`, call `robot.look_at(...)` to re-observe and recover closed-loop.
## Code-as-Policies Primitives (`POST http://127.0.0.1:8081/plan`)
robot.look_at(x=0.35, y=0.13, z=0.14, grid_z=0.015)
x, y = robot.deproject(u=0.52, v=0.48, z_plane=0.015)
robot.grasp(x=x, y=y, z=0.015, yaw_deg=-35, pitch_deg=90)
robot.place(x=0.24, y=0.06, z=0.075)
robot.place_until_contact(x=0.24, y=0.06, z=0.015, max_lag=0.012, relief=0.002)
robot.goto(x=0.28, y=0.05, z=0.04, arc_z=0.14, pitch_deg=25, azimuth_deg=90, roll_deg=0, speed=0.10)
robot.move_by(forward=0.025, dz=-0.002, dpitch_deg=-3, droll_deg=5, speed=0.04)
robot.move_until_contact(dz=-0.06, speed=0.03, max_lag=0.012, relief=0.002)
robot.open_gripper() / robot.close_gripper() / robot.home() / robot.done()3. 全方位控制
在 Gemini 世界模型中，Argon 引导机器人将五个分散的 Scrabble 字母块（G,R,A,N,O）重新排列为拼写 A-R-G-O-N 的一行。不同于静态提示，Argon 在闭环视觉反馈中指导模型，编写时间戳动作提示，检查中间状态并完善后续指令。
Iterative Video World Model Steering Dexterous Hands: 33-Step Steered Video Rollout Spelling A-R-G-O-N.Prompt:
Goal: {task_goal}
You are steering a video world model (Gemini Omni) in an iterative visual feedback loop. Inspect the attached frames from the latest rollout and write the next timestamped action prompt for the video world model:
- Break complex motions into one simple, single-direction action at a time.
- Keep inactive hands retracted and avoid blocking the camera's view of objects in the workspace.
- Explicitly state which areas of the workspace become empty after an object moves.
- Use [MM:SS.mmm] timestamps for each action beat and wrap your output in <omni_prompt> tags.
Example format (sample timestamps only, not the full prompt):
<omni_prompt>
[00:00.000] Character A's left hand rests at the side while the right hand reaches toward the target object.
[00:02.500] Character A's right fingertips push the object smoothly into the center of the table, leaving its starting area empty.
[00:05.000] Character A's right hand opens and retracts to the side so all placed objects remain in clear view of the camera.
</omni_prompt>基于 Argon 的 Real2Sim 重建
1. 机器人视图到模拟
Argon 处理真实的机器人远程操作事件，开发数字双胞胎。它获取或创建模拟资产，使其在与灵巧手接触时的外观和行为符合真实物体。通过一代视觉推理循环，Argon 调整资产和场景，使模拟中的视觉和接触事件与真实记录数据相匹配。
Left (Real View): synchronized real robot camera feeds; Right (Simulated): 3D MuJoCo scene reconstructed and calibrated by Argon.Prompt:
[Attached: robot_rollout_episode]
Build a MuJoCo scene of the robot and reconstructed task objects. When each episode is replayed frame by frame, by driving the robot joints toward the logged observation.state with joint-space PD.
Iterate using two verification objectives:
(1) Visual match: MuJoCo renders from the head_left, left_wrist, and right_wrist cameras line up with, and look like, those real observations.
(2) Physical match: the simulated contact force/torque on each Sharpa fingertip, in the sensor frame, matches the logged tactile_force in timing, magnitude and direction.
What you can tune happens before simulation starts:
(1) Static object assets: realistic geometry, collision shapes, mass, inertia, friction and joints, with appearance close to the real objects, plus the initial object poses at t = 0.
(2) Offline-planned robot commands, changed only in frames where the real fingertips are near an object. Each command must stay within 0.55 rad (arm joints) and 0.65 rad (hand joints) of the logged teleop trajectory. Everywhere else, replay the logged trajectory unchanged.
No sim cheating: task objects may move only through real MuJoCo contact with the robot hands and the environment (mj_step).
Not allowed after t = 0: changing MjModel parameters; applying external forces or torques to objects; welds or equality constraints; feedforward torques beyond joint-space PD; setting or rolling back object qpos/qvel; disabling or excluding collisions; reporting contact forces that don't come from the MuJoCo solver (for example, copying the logged tactile readings).
Immediately present the reconstructed scene and rollout results as an interactive HTML session viewer with one page per episode, showing:
- At the bottom: A synced timeline (play, scrub, speed control) that drives every panel.
- On the left panel: An interactive 3D Blender scene showing the simulated robot and objects, showing contact points and force arrows as replay progresses.
- On the right panel: Real vs. sim camera views side by side for all 3 cameras. Real vs. sim finger force visualizations.2. 人类视角到模拟
给定一段第一人称人类操作任务视频，Argon 识别相关物体，用复合凸分解网格替换粗糙的边界盒以支持凹形容器（如嵌套碗、凹陷托盘），并将人类手部轨迹重定向为具有 22 自由度五指手的 76 自由度双足人形机器人，在 MuJoCo 接触物理中进行验证。
Human View to Simulation — Egocentric Human Video to Dexterous Humanoid Twin: Left shows the input egocentric human RGB video; Right shows the synchronized 500 Hz MuJoCo + Blender Cycles multi-view simulation rollout.Prompt:
[Attached: egocentric_human_video.mp4, candidate_sim_views.png, hand_3d_keypoints.json]
You are building a physically solvable 500 Hz MuJoCo manipulation environment (`scene.xml`) and studio Blender render that faithfully reproduces the tabletop task shown in `egocentric_human_video.mp4` on a bimanual humanoid robot with 22-DoF dexterous five-fingered hands:
1. Scene Reconstruction & Camera Calibration:
- Identify only the manipulation-relevant objects (the grasped target object and the receiving container or landmark) and triangulate their 3D tabletop coordinates from the egocentric camera intrinsics/extrinsics.
- Replace crude bounding boxes with compound convex-decomposed 3D meshes so concave containers (nested bowls, recessed wooden trays, open bins) support physical interior placement.
2. 500 Hz Dexterous Motion-Fitting Stress Test:
- Retarget the 3D human wrist and finger keypoints to the humanoid's 7-DoF arm and 22-DoF hand joints, and step the trajectory in MuJoCo at 500 Hz (dt = 0.002s) with zero kinematic weld constraints.
- Verify opposing thumb-and-finger force closure, clean object lift (> 2 cm), and stable final placement inside or beside the target landmark.3. 从草图到模拟
开发人员可提供一张显示阿波罗 2 号人形机器人抛物击落桌上堆叠盒子的粗略手绘草图。Argon 自动构建 GPU 加速的 mujoco_warp 场景，合成平衡的 76 自由度全身及五指抛物轨迹，在闭环物理中验证 6/6 目标击落，并导出 Blender 4.5 Cycles 渲染结果。
Sketch to Simulation — Hand-Drawn Sketch to Dexterous Humanoid Trajectory & Cycles Render.Prompt:
[Attached: hand_drawn_throw_sketch.jpg]
Translate this hand-drawn sketch into a physically verified humanoid simulation in `mujoco_warp` and a raytraced Blender Cycles video:
1. Scene & Kinematic Setup from Sketch:
- Parse the spatial layout from `hand_drawn_throw_sketch.jpg`: place the 76-DoF humanoid (32-DoF body equipped with two 22-DoF five-finger dexterous hands) at the origin facing a table holding a 3-2-1 pyramid of 6 rigid target boxes, with a projectile sphere grasped in the right five-finger hand.
2. Whole-Body Balance & Dexterous Throw Synthesis:
- Synthesize a 50 Hz whole-body throwing trajectory (140 steps, 2.80 s) across four phases: (1) stable stance & 5-finger power grasp closure, (2) torso/shoulder wind-up with ankle-hip Center-of-Mass compensation, (3) forward whipping acceleration, and (4) synchronized 22-DoF five-finger opening at peak hand velocity so the ball launches on a ballistic arc into the pyramid's second-tier center of mass.
- Verify in `mujoco_warp` contact physics that the humanoid maintains upright balance (< 3 cm pelvis drift) and knocks down all 6 target boxes.基于 Argon 的物理理解
1. 后视标签与数据质量
Argon 提供详细的细分和文本标签，能够识别停顿、抓取不稳定或失控等劣质操纵段。针对一个双手 ALOHA 2 机器人在 136 秒内将方形纸折叠成狐狸的第一人称视频，Argon 生成了时间对齐的四级后视注释层次：剧集目标 (G1)、6 个连续语义子任务 (G2)、17 个接触级原子原语 (G3) 以及自动化质量控制 (QC) 重量化，确保在失误和重试时保持任务进度平稳。
Argon Multi-Granularity Hindsight & Telemetry HUD.
