Action Images: raw pose → multi-view pixel-grounded actions

arXiv 2604.06168 run on FreeTacMan motherboard_0510_episode_001, frames 719–1019 (10 s, sustained motion), three calibrated RealSense views (left / middle / right) at 640×480. Instead of feeding a policy a low-dimensional action vector, each action is converted into points that live in the same 2D pixel space as the observation, in every camera view.

Step 0 — raw action

The raw 7-DoF pose from the parquet

Each frame carries an absolute pose per tactile sensor: [x, y, z] in metres (OptiTrack mocap frame) and an xyzw quaternion. This is the "raw robot action" to be converted.

Step 1 — 7-DoF → three semantic 3D points

Pose becomes q_pos, q_normal, q_up (ℓ = 100 mm)

Following the paper: q_pos is the end-effector point, q_normal = q_pos + ℓ·(surface normal), q_up = q_pos + ℓ·(orthogonal in-plane axis). One deliberate departure: this rig is calibrated, so the normal is the measured gel axis (T_gel_to_rigid) and q_pos the real gel centre, rather than the paper's canonical gripper axes.

Trajectory of q_pos (red, = the position channel) with normal (green) and up (blue) triads drawn every second, in mocap millimetres.

Step 2 — multi-view projection

Project into each calibrated camera view

Chain: pose (m) → ×1000 → gel centre in mocap mm → T_mocap_to_cam → pinhole intrinsics → pixels. Calibration rmse is 4.7 / 5.3 / 7.5 mm for left / middle / right. Over the whole episode 97–100% of points land in-frame with positive depth; the demo segment is 100% visible in all three views.

Filled dot = projected gel centre (blue = left sensor, orange = right), thick arrow → normal point, thin arrow → up point, label = depth. The dot staying on the physical sensor in all three views is the calibration check.

Step 3 — action images

Render as RGB Gaussian heatmaps (the paper's format)

Each projected triple becomes an H×W×3 image: R = position, G = normal, B = up, σ = 0.05·min(H, W). Stacked over time (6 fps grid) this is the action video the paper pairs with the observation video per view. Two departures, documented in build_action_images.py: one image per sensor (two sensors would be indistinguishable in one RGB image), and no gripper-openness term (handheld rig).

Rows = views. Columns: observation · left-sensor action image · right-sensor action image · blend. In the blend, the Gaussian triads must sit on the sensors — and they do. Yellow = R+G overlap where the normal point projects near the position point.

Raw action video, middle view, left sensor.

Raw action video, middle view, right sensor.

Step 4 — does it track real motion?

Validation against AllTracker optical flow

The representation claims to explicitly track robot motion in image space. Two checks against AllTracker on the real videos:

A. Drift-free local-motion check (the decisive one). At every frame (30 fps) the dense adjacent flow (t → t+1) is sampled at the projected gel-centre pixel and compared with the projected point's own displacement. No long-range tracking is involved, so nothing can drift:

view / sensor	median EPE (px/frame)	p95	median EPE, moving	cosine (moving)	gain
left / left	0.77	2.65	0.90	0.910	0.94
left / right	0.20	0.90	0.44	0.948	0.97
middle / left	0.85	2.99	0.97	0.912	0.97
middle / right	0.24	1.27	0.57	0.854	1.02
right / left	0.76	4.31	0.92	0.868	0.69
right / right	0.19	1.74	0.52	0.824	1.06

Per-frame displacement of the projected point vs AllTracker flow under it (u and v pooled). Points hug the dashed y=x line in every view: the projected action moves the way the pixels under it move.

B. Long-range reference tracks. AllTracker reference mode (query = segment start) tracking the physical gel-centre pixel:

Filled dot + colored trail = pose-projected; white ring + trail = AllTracker.

Where the tracker stays locked (view_right / right sensor: flat ~8 px for 10 s — about the 7.5 mm calibration rmse at 0.9 m depth) the two tracks agree throughout. The large reference-track errors elsewhere onset abruptly when a sensor starts moving and never recover — the signature of tracker drift under strong rotation and occlusion, not projection error. The drift-free check in A isolates the projection itself, and it passes in all six view×sensor combinations.

Reproduce

Scripts

world_model/data_preprocessing/project_pose_to_camera.py (geometry) · render_pose_overlay.py (step 2) · build_action_images.py (step 3) · action_images_demo/alltracker_validate.py + step4_metrics.json (step 4). Demo artifacts live in mode1_v2_videos/motherboard_0510_episode_001/action_images_demo/.