data_juicer.ops.deduplicator.image_deduplicator module#

data_juicer.ops.deduplicator.image_deduplicator.get_hash_method(method_name)[source]#
class data_juicer.ops.deduplicator.image_deduplicator.ImageDeduplicator(method: str = 'phash', consider_text: bool = False, *args, **kwargs)[source]#

Bases: Deduplicator

Deduplicator to deduplicate samples at document-level using exact matching of images between documents.

__init__(method: str = 'phash', consider_text: bool = False, *args, **kwargs)[source]#

Initialization method.

Parameters:

  • method – hash method for image

  • consider_text – whether to consider text hash together with image hash when applying deduplication.

  • args – extra args

  • kwargs – extra args

compute_hash(sample, context=False)[source]#

Compute hash values for the sample.

Parameters:

sample – input sample

Returns:

sample with computed hash value.

process(dataset, show_num=0)[source]#

For doc-level, dataset –> dataset.

Parameters:

  • dataset – input dataset

  • show_num – number of traced samples used when tracer is open.

Returns:

deduplicated dataset and the sampled duplicate pairs.