data_juicer.core.ray_exporter module#

class data_juicer.core.ray_exporter.RayExporter(export_path, export_type=None, keep_stats_in_res_ds=True, keep_hashes_in_res_ds=False, **kwargs)[source]#

Bases: object

The Exporter class is used to export a ray dataset to files of specific format.

__init__(export_path, export_type=None, keep_stats_in_res_ds=True, keep_hashes_in_res_ds=False, **kwargs)[source]#

Initialization method.

Parameters:

  • export_path – the path to export datasets.

  • keep_stats_in_res_ds – whether to keep stats in the result dataset.

  • keep_hashes_in_res_ds – whether to keep hashes in the result dataset.

export(dataset, columns=None)[source]#

Export method for a dataset.

Parameters:

  • dataset – the dataset to export.

  • columns – the columns to export.

Returns:

static write_json(dataset, export_path, **kwargs)[source]#

Export method for json/jsonl target files.

Parameters:

  • dataset – the dataset to export.

  • export_path – the path to store the exported dataset.

  • kwargs – extra arguments.

Returns:

static write_webdataset(dataset, export_path, **kwargs)[source]#

Export method for webdataset target files.

Parameters:

  • dataset – the dataset to export.

  • export_path – the path to store the exported dataset.

  • kwargs – extra arguments.

Returns:

static write_others(dataset, export_path, **kwargs)[source]#

Export method for other target files.

Parameters:

  • dataset – the dataset to export.

  • export_path – the path to store the exported dataset.

  • kwargs – extra arguments.

Returns: