data-juicer

Dataset Configuration

This guide covers how to configure input datasets in your Data-Juicer recipe. You will learn how to point at local files, remote Hugging Face or arXiv datasets, mix multiple sources, validate data, and handle edge cases.

Supported dataset formats

Data-Juicer auto-detects file formats for local files. Supported formats include parquet, jsonl, json, csv, tsv, txt, and jsonl.gz.

Local dataset

Point at a file or directory on your local filesystem. The format field is optional — Data-Juicer detects it from the file extension.

dataset:
  configs:
    - type: local
      path: path/to/your/local/dataset.json
      format: json    # optional

dataset:
  configs:
    - type: local
      path: path/to/your/local/dataset.parquet
      format: parquet

See local_json.yaml for a complete example.

Remote Hugging Face dataset

Load any dataset from the Hugging Face Hub. Set type to remote and source to huggingface.

dataset:
  configs:
    - type: 'remote'
      source: 'huggingface'
      path: "HuggingFaceFW/fineweb"
      name: "CC-MAIN-2024-10"   # optional: dataset config name
      split: "train"             # optional: which split to load
      limit: 1000                # optional: cap the number of samples

See remote_huggingface.yaml for a complete example.

arXiv data

For arXiv papers, use the preprocessing tools to download and convert arXiv tar archives into JSONL format that Data-Juicer can process directly.

Other formats

For the full list of supported formats and loading strategies, see load_strategy.py.

Data mixture

The default executor combines sources listed in dataset.configs in two ways:

  • With dataset.max_sample_num: allocate the total sample budget by source weight, then sample and combine the results. An allocation larger than a source is filled by repeating samples.

  • With dataset.max_sample_num omitted: concatenate all rows from each source.

The Ray executor loads a single source through this configuration.

dataset:
  max_sample_num: 10000    # total samples allocated by source weight
  configs:
    - type: 'local'
      weight: 1.0
      path: 'path/to/json/file'
    - type: 'local'
      weight: 1.0
      path: 'path/to/csv/file'

See mixture.yaml for a complete example.

Data validation

Validate your dataset before processing by adding validators to your recipe. Each validator checks a specific aspect of the data.

dataset:
  configs:
    - type: local
      path: path/to/data.json

validators:
  - type: swift_messages
    min_turns: 2
    max_turns: 20
    sample_size: 1000
  - type: required_fields
    required_fields:
      - "text"
      - "metadata"
      - "language"
    field_types:
      text: "str"
      metadata: "dict"
      language: "str"

See data_validator.py for the full list of supported validators.

Reader Options

For local processing and analysis, use load_dataset_kwargs to choose format options. For example, load selected columns from Parquet:

load_dataset_kwargs:
  columns: ['text', 'meta']

For a tab-separated CSV file, use load_dataset_kwargs: {delimiter: "\t"} instead. Choose options that match your input format.

Loading Workers

np sets the default number of workers for both loading and processing. To use fewer workers for loading:

np: 16
load_dataset_kwargs:
  num_proc: 4

This loads with 4 workers and processes with a default of 16. In Python, an explicit DatasetBuilder.load_dataset(num_proc=...) argument takes precedence over these settings.

Ray Input

For ray, ray_partitioned, and RayAnalyzer, use override_num_blocks to set the requested number of input blocks. For large JSON records, increase block_size:

read_options:
  block_size: 268435456  # 256 MiB
override_num_blocks: 64

read_options accepts PyArrow JSON reader options. Leave it empty to use the defaults. Ray distributes loading across the cluster; load_dataset_kwargs.num_proc controls local loading only.

Troubleshooting

JSONL per-line fault tolerance

For local processing and analysis, set load_jsonl_lenient: true in your recipe to skip malformed JSONL lines and process the remaining data:

load_jsonl_lenient: true

You can also enable this option for one command:

DATA_JUICER_JSONL_LENIENT=1 dj-process --config path/to/config.yaml

Note: Only .jsonl / .jsonl.gz / .jsonl.zst shards are read. Other files in the same directory (e.g. .json) are skipped with a warning. Search logs for [lenient jsonl] to see which lines were skipped.

Value is too big! error

When loading local JSONL, HuggingFace datasets may parse with ujson, which cannot handle very large integers. If you see ValueError: Value is too big!:

Fix

How

Use stdlib json (recommended)

DATA_JUICER_USE_STDLIB_JSON=1 dj-process --config path/to/config.yaml

Export as strings

Quote the problematic numeric fields in your JSON source.

Switch to Parquet

Parquet uses Arrow, which avoids this code path entirely.

Legacy dataset_path configuration

The dataset_path key is the original, simpler way to specify input. It works but lacks the flexibility of the dataset.configs approach above.

# YAML
dataset_path: path/to/your/dataset.json

# CLI
dj-process --dataset_path path/to/your/dataset.json

# CLI with mixture weights
dj-process --dataset_path "0.5 path/to/dataset1.json 0.5 path/to/dataset2.json"

What’s next

↵ to select ↑↓ to navigate esc to close