data_juicer.ops.deduplicator.video_deduplicator module

class data_juicer.ops.deduplicator.video_deduplicator.VideoDeduplicator(consider_text: bool = False, *args, **kwargs)

Bases: Deduplicator

Deduplicator to deduplicate samples at document-level using exact matching of videos between documents.

__init__(consider_text: bool = False, *args, **kwargs)

Initialization.

Parameters:

• consider_text – whether to consider text hash together with video hash when applying deduplication.

• args – extra args

• kwargs – extra args

compute_hash(sample, context=False)

Compute hash values for the sample.

Parameters:

sample – input sample

Returns:

sample with computed hash value.

process(dataset, show_num=0)

For doc-level, dataset –> dataset.

Parameters:

• dataset – input dataset

• show_num – number of traced samples used when tracer is open.

Returns:

deduplicated dataset and the sampled duplicate pairs.