data_juicer.ops.mapper.fix_unicode_mapper module#

class data_juicer.ops.mapper.fix_unicode_mapper.FixUnicodeMapper(normalization: str = None, *args, **kwargs)[source]#

Bases: Mapper

Mapper to fix unicode errors in text samples.

__init__(normalization: str = None, *args, **kwargs)[source]#

Initialization method.

Parameters:

  • normalization – the specified form of Unicode normalization mode, which can be one of [‘NFC’, ‘NFKC’, ‘NFD’, and ‘NFKD’], default ‘NFC’.

  • args – extra args

  • kwargs – extra args

process_batched(samples)[source]#