VLA Visualization Demo¶

This demo provides a complete pipeline for egocentric video hand action recognition and LeRobot dataset export. It extracts frames from ego-view videos, estimates camera intrinsics and poses, reconstructs 3D hands, computes hand actions, and exports the results as a LeRobot v2.0 dataset.

Pipeline Overview¶

Video Input
  │
  ▼
VideoExtractFramesMapper          # Extract video keyframes
  │
  ▼
VideoCameraCalibrationMogeMapper  # MoGe-2 camera calibration + depth estimation
  │
  ▼
VideoHandReconstructionHaworMapper # HaWoR 3D hand reconstruction
  │
  ▼
VideoCameraPoseMegaSaMMapper      # MegaSaM camera pose estimation (⚠️ requires separate conda env)
  │
  ▼
VideoHandActionComputeMapper      # Compute 7-DoF actions + 8-dim states
  │
  ▼
VideoActionCaptioningMapper       # action instruction captioning
  │
  ▼
ExportToLeRobotMapper            # Export to LeRobot v2.0 dataset

Output Format¶

Action: 7-dim [dx, dy, dz, droll, dpitch, dyaw, gripper]
State: 8-dim [x, y, z, roll, pitch, yaw, pad, gripper]
Gripper: 1.0 (open) to -1.0 (closed), estimated from finger joint angles

Prerequisites¶

1. Base Environment¶

Create an image based on the Dockerfile.

The VideoCameraPoseMegaSaMMapper operator depends on MegaSaM (based on DROID-SLAM). Its CUDA compiled components (droid_backends, lietorch, torch-scatter) conflict with the main environment and must run in a separate conda environment.

Note: This environment is automatically activated at runtime via Ray’s runtime_env={"conda": "mega-sam"} mechanism. You do not need to manually switch environments. All other operators run in the default environment.

2. Ray Cluster¶

The pipeline runs on Ray. You need to start a Ray cluster.

3. MANO Hand Model¶

Download MANO v1.2 from the MANO website. Update the mano_right_path and mano_left_path in the config or script to point to your MANO_RIGHT.pkl and MANO_LEFT.pkl files.

Running the Demo¶

Option 1: Python Script (Recommended)¶

cd demos/ego_hand_action_annotation
python vla_pipeline.py

Option 2: YAML Config¶

python tools/process_data.py --config demos/ego_hand_action_annotation/configs/vla_pipeline.yaml

Input Data Format¶

Each sample is a JSON object containing a video path list:

{
    "videos": ["./data/1018.mp4"],
    "text": "",
    "__dj__meta__": {}
}

The demo includes two sample videos: data/1018.mp4 and data/1034.mp4.

Output Structure¶

output/
├── frames/                    # Extracted video frames
├── lerobot_dataset/           # LeRobot v2.0 dataset
│   ├── data/
│   │   └── chunk-000/
│   │       ├── episode_000000.parquet
│   │       └── ...
│   ├── videos/
│   │   └── chunk-000/
│   │       ├── observation.images.main/
│   │       │   ├── episode_000000.mp4
│   │       │   └── ...
│   ├── meta/
│   │   ├── info.json
│   │   ├── episodes.jsonl
│   │   ├── stats.json
│   │   └── tasks.jsonl
│   └── modality.json
└── *.parquet                  # Ray output results

Visualization Tools¶

Two visualization scripts are provided for inspecting processing results:

Action Annotation Verification (vis_hand_action_demo.py)¶

Verify hand action annotations with hand trajectory, state, and action value overlays:

python vis_hand_action_demo.py --data_path output/xxx.parquet

Troubleshooting¶

MegaSaM compilation fails¶

Ensure the mega-sam conda environment has a CUDA toolkit matching your PyTorch version. Verify with nvcc --version.

MANO model loading fails¶

Check that mano_right_path and mano_left_path point to valid files. MANO models must be downloaded separately from the official website.

Ray GPU resource exhaustion¶

Multiple operators require GPU. By default each uses 0.1 GPU (10 operators can share 1 GPU). Adjust num_gpus or add more GPUs if needed.