data_juicer.format.json_formatter module#

class data_juicer.format.json_formatter.JsonFormatter(dataset_path, suffixes=None, **kwargs)[source]#

Bases: LocalFormatter

The class is used to load and format json-type files.

Default suffixes is [‘.json’, ‘.jsonl’, ‘.json.gz’, ‘.jsonl.gz’, ‘.json.zst’, ‘.jsonl.zst’]

SUFFIXES = ['.json', '.jsonl', '.json.gz', '.jsonl.gz', '.json.zst', '.jsonl.zst']#
__init__(dataset_path, suffixes=None, **kwargs)[source]#

Initialization method.

Parameters:

  • dataset_path – a dataset file or a dataset directory

  • suffixes – files with specified suffixes to be processed

  • kwargs – extra args