video_hand_reconstruction_mapper#

Use the WiLoR model for hand localization and reconstruction.

使用 WiLoR 模型进行手部定位与重建。

Type 算子类型: mapper

Tags 标签: gpu, video

🔧 Parameter Configuration 参数配置#

name 参数名

type 类型

default 默认值

desc 说明

wilor_model_path

<class ‘str’>

'wilor_final.ckpt'

The path to ‘wilor_final.ckpt’.

wilor_model_config

<class ‘str’>

'model_config.yaml'

The path to ‘model_config.yaml’ for the WiLOR model.

detector_model_path

<class ‘str’>

'detector.pt'

The path to ‘detector.pt’ for the WiLOR model.

mano_right_path

<class ‘str’>

'path_to_mano_right_pkl'

The path to ‘MANO_RIGHT.pkl’. Users need to download this file from https://mano.is.tue.mpg.de/ and comply with the MANO license.

frame_num

typing.Annotated[int, Gt(gt=0)]

3

The number of frames to be extracted uniformly from the video. If it’s 1, only the middle frame will be extracted. If it’s 2, only the first and the last frames will be extracted. If it’s larger than 2, in addition to the first and the last frames, other frames will be extracted uniformly within the video duration. If “duration” > 0, frame_num is the number of frames per segment.

duration

<class ‘float’>

0

The duration of each segment in seconds. If 0, frames are extracted from the entire video. If duration > 0, the video is segmented into multiple segments based on duration, and frames are extracted from each segment.

batch_size

<class ‘int’>

16

Batch size for simultaneous hand inference.

tag_field_name

<class ‘str’>

'hand_reconstruction_tags'

The field name to store the tags. It’s “hand_reconstruction_tags” in default.

frame_dir

<class ‘str’>

DATA_JUICER_ASSETS_CACHE

Output directory to save extracted frames.

if_save_visualization

<class ‘bool’>

True

Whether to save overlay images.

save_visualization_dir

<class ‘str’>

DATA_JUICER_ASSETS_CACHE

The path for saving overlay images.

if_save_mesh

<class ‘bool’>

True

Whether to save images of the hand mesh.

save_mesh_dir

<class ‘str’>

DATA_JUICER_ASSETS_CACHE

The path for saving images of the hand mesh.

args

''

extra args

kwargs

''

extra args

📊 Effect demonstration 效果演示#

test#

VideoHandReconstructionMapper(wilor_model_path='wilor_final.ckpt', wilor_model_config='model_config.yaml', detector_model_path='detector.pt', mano_right_path='path_to_mano_right_pkl', frame_num=1, duration=1, frame_dir=DATA_JUICER_ASSETS_CACHE, if_save_visualization=True, save_visualization_dir=DATA_JUICER_ASSETS_CACHE, if_save_mesh=True, save_mesh_dir=DATA_JUICER_ASSETS_CACHE)

📥 input data 输入数据#

Sample 1: 1 video
video3.mp4:
Sample 2: 1 video
video4.mp4:

📤 output data 输出数据#

Sample 1: empty
frame_nums49
vertices_shape
[2, 778, 3]
camera_translation_shape
[2, 3]
if_right_hand_shape
[2]
joints_shape
[2, 21, 3]
keypoints_shape
[2, 778, 2]
Sample 2: empty
frame_nums22
vertices_shape
[1, 778, 3]
camera_translation_shape
[1, 3]
if_right_hand_shape
[1]
joints_shape
[1, 21, 3]
keypoints_shape
[1, 778, 2]

test_mul_proc#

VideoHandReconstructionMapper(wilor_model_path='wilor_final.ckpt', wilor_model_config='model_config.yaml', detector_model_path='detector.pt', mano_right_path='path_to_mano_right_pkl', frame_num=1, duration=1, frame_dir=DATA_JUICER_ASSETS_CACHE, if_save_visualization=True, save_visualization_dir=DATA_JUICER_ASSETS_CACHE, if_save_mesh=True, save_mesh_dir=DATA_JUICER_ASSETS_CACHE)

📥 input data 输入数据#

Sample 1: 1 video
video3.mp4:
Sample 2: 1 video
video4.mp4:

📤 output data 输出数据#

Sample 1: empty
frame_nums49
vertices_shape
[2, 778, 3]
camera_translation_shape
[2, 3]
if_right_hand_shape
[2]
joints_shape
[2, 21, 3]
keypoints_shape
[2, 778, 2]
Sample 2: empty
frame_nums22
vertices_shape
[1, 778, 3]
camera_translation_shape
[1, 3]
if_right_hand_shape
[1]
joints_shape
[1, 21, 3]
keypoints_shape
[1, 778, 2]