data_juicer.ops.mapper.remove_non_chinese_character_mapper module#

class data_juicer.ops.mapper.remove_non_chinese_character_mapper.RemoveNonChineseCharacterlMapper(keep_alphabet: bool = True, keep_number: bool = True, keep_punc: bool = True, *args, **kwargs)[源代码]#

基类：Mapper

Mapper to remove non chinese Character in text samples.

__init__(keep_alphabet: bool = True, keep_number: bool = True, keep_punc: bool = True, *args, **kwargs)[源代码]#

Initialization method.

参数:

  • keep_alphabet -- whether to keep alphabet

  • keep_number -- whether to keep number

  • keep_punc -- whether to keep punctuation

  • args -- extra args

  • kwargs -- extra args

process_batched(samples)[源代码]#