HumanVBench Operators Demo#

This is the operator contribution page for the paper: HumanVBench: Probing Human-Centric Video Understanding in MLLMs with Automatically Synthesized Benchmarks (CVPR’26).

Paper#

HumanVBench (CVPR 2026)
arXiv: https://arxiv.org/abs/2412.17574
Project: https://openaccess.thecvf.com/content/CVPR2026/html/Zhou_HumanVBench_Probing_Human-Centric_Video_Understanding_in_MLLMs_with_Automatically_Synthesized_CVPR_2026_paper.html

Operators#

Operator	Type	Description
`video_face_ratio_filter`	filter	Filter videos by face-to-frame ratio.
`video_human_tracks_extraction_mapper`	mapper	Extract face and human bounding box tracks.
`video_human_tracks_face_demographic_mapper`	mapper	Detect face demographics via DeepFace.
`video_tagging_from_audio_mapper`	mapper	Generate video tags from audio (existing).
`video_audio_detect_age_gender_mapper`	mapper	Detect age/gender from speech audio.
`video_audio_ASR_mapper`	mapper	Automatic speech recognition.
`video_audio_speech_emotion_mapper`	mapper	Speech emotion recognition.
`video_captioning_from_human_tracks_mapper`	mapper	Per-person captioning via VideoLLaMA3.
`video_captioning_face_attribute_emotion_mapper`	mapper	Face attribute/emotion captioning via VideoLLaMA3.
`video_active_speaker_detect_mapper`	mapper	Active speaker detection via Light-ASD.

Installation#

Note: These OPs need third-party patches/models under thirdparty/humanvbench_models/, so a source install is required for now.

git clone https://github.com/datajuicer/data-juicer.git
cd data-juicer
pip install -e .

Quick Start#

Since HumanVBench operators involve modifications to external repositories, these adjusted repositories are stored in thirdparty/humanvbench_models.

Option 1: Automatic Mode (Recommended)#

Run directly — the operators handle git clone and .diff patch merging automatically:

dj-process --config demos/video_humanvbench_simple/analyzer.yaml

Option 2: Manual Mode#

Follow the instructions in thirdparty/humanvbench_models/README.md to manually complete the git clone and .diff patch merging, then run:

dj-process --config demos/video_humanvbench_simple/analyzer.yaml

Pipeline Dependency#

The operators form a processing pipeline. The recommended execution order is:

video_face_ratio_filter — filter human-centric videos
video_human_tracks_extraction_mapper — extract face/human tracks
video_human_tracks_face_demographic_mapper — face demographics
video_tagging_from_audio_mapper — audio tags (Speech/Music/EMPTY)
video_audio_detect_age_gender_mapper — speaker age/gender
video_captioning_from_human_tracks_mapper — per-person captions
video_captioning_face_attribute_emotion_mapper — face attributes/emotions
video_active_speaker_detect_mapper — active speaker detection
video_audio_ASR_mapper — speech transcription
video_audio_speech_emotion_mapper — speech emotion