data_juicer.ops.mapper.video_whole_body_pose_estimation_mapper module#

class data_juicer.ops.mapper.video_whole_body_pose_estimation_mapper.VideoWholeBodyPoseEstimationMapper(onnx_det_model: str = 'yolox_l.onnx', onnx_pose_model: str = 'dw-ll_ucoco_384.onnx', frame_num: Annotated[int, Gt(gt=0)] = 3, duration: float = 0, tag_field_name: str = 'pose_estimation_tags', frame_dir: str = '/home/runner/.cache/data_juicer/assets', if_save_visualization: bool = False, save_visualization_dir: str = '/home/runner/.cache/data_juicer/assets', *args, **kwargs)[source]#

Bases: Mapper

Input a video containing people, and use the DWPose model to extract the body, hand, feet, and face keypoints of the human subjects in the video, i.e., 2D Whole-body Pose Estimation.

__init__(onnx_det_model: str = 'yolox_l.onnx', onnx_pose_model: str = 'dw-ll_ucoco_384.onnx', frame_num: Annotated[int, Gt(gt=0)] = 3, duration: float = 0, tag_field_name: str = 'pose_estimation_tags', frame_dir: str = '/home/runner/.cache/data_juicer/assets', if_save_visualization: bool = False, save_visualization_dir: str = '/home/runner/.cache/data_juicer/assets', *args, **kwargs)[source]#

Initialization method.

Parameters:

  • onnx_det_model – The path to ‘yolox_l.onnx’.

  • onnx_pose_model – The path to ‘dw-ll_ucoco_384.onnx’.

  • frame_num – The number of frames to be extracted uniformly from the video. If it’s 1, only the middle frame will be extracted. If it’s 2, only the first and the last frames will be extracted. If it’s larger than 2, in addition to the first and the last frames, other frames will be extracted uniformly within the video duration. If “duration” > 0, frame_num is the number of frames per segment.

  • duration – The duration of each segment in seconds. If 0, frames are extracted from the entire video. If duration > 0, the video is segmented into multiple segments based on duration, and frames are extracted from each segment.

  • tag_field_name – The field name to store the tags. It’s “pose_estimation_tags” in default.

  • frame_dir – Output directory to save extracted frames.

  • if_save_visualization – Whether to save visualization results.

  • save_visualization_dir – The path for saving visualization results.

  • args – extra args

  • kwargs – extra args

process_single(sample=None, rank=None)[source]#

For sample level, sample –> sample

Parameters:

sample – sample to process

Returns:

processed sample