Cache Management

This document describes DataJuicer’s cache management system, including HuggingFace dataset caching, cache directory configuration, cache compression, and temporary storage.

Overview

DataJuicer provides a caching mechanism based on HuggingFace Datasets to avoid redundant computation. When enabled, each operator generates cache files with a unique fingerprint based on:

• The fingerprint of the input data

• The operator name and parameters

• The hash of the processing function

That is: same input + same operator configuration = same fingerprint = cache hit. Therefore, re-running the same pipeline on the same data will skip already-computed steps. For more details, please refer to our paper: Data-Juicer: A One-Stop Data Processing System for Large Language Models.

The cache system also provides:

• Configurable cache directories via environment variables or config options

• Cache compression to reduce disk usage for large-scale datasets

• Temporary storage for intermediate files in non-cache mode

• Fine-grained cache control via context managers and decorators

Configuration

Basic Cache Settings

use_cache: true           # Enable/disable HuggingFace dataset caching
ds_cache_dir: null         # Custom cache directory (overrides HF_DATASETS_CACHE)
cache_compress: null       # Compression method: 'gzip', 'zstd', 'lz4', or null
temp_dir: null             # Temp directory for intermediate files when cache is disabled

Command Line

# Enable caching (default)
dj-process --config config.yaml --use_cache true

# Disable caching
dj-process --config config.yaml --use_cache false

# Enable cache compression
dj-process --config config.yaml --cache_compress zstd

# Custom cache directory
dj-process --config config.yaml --ds_cache_dir /fast-storage/dj-cache

Cache Directory Structure

DataJuicer organizes cache files in a hierarchical directory structure controlled by environment variables:

~/.cache/                              # CACHE_HOME (default)
└── data_juicer/                       # DATA_JUICER_CACHE_HOME
    ├── assets/                        # DATA_JUICER_ASSETS_CACHE
    │   └── (extracted frames, stopwords, flagged words, etc.)
    └── models/                        # DATA_JUICER_MODELS_CACHE
        └── (downloaded model files)

Environment Variables

Variable	Default	Description
CACHE_HOME	~/.cache	Root cache directory
DATA_JUICER_CACHE_HOME	$CACHE_HOME/data_juicer	DataJuicer cache root
DATA_JUICER_ASSETS_CACHE	$DATA_JUICER_CACHE_HOME/assets	Assets cache (frames, word lists, etc.)
DATA_JUICER_MODELS_CACHE	$DATA_JUICER_CACHE_HOME/models	Downloaded models cache
DATA_JUICER_EXTERNAL_MODELS_HOME	None	External models directory

Override defaults by setting environment variables:

export DATA_JUICER_CACHE_HOME=/data/dj-cache
export DATA_JUICER_MODELS_CACHE=/models/dj-models
dj-process --config config.yaml

Cache Compression

For large-scale datasets (tens of GB or more), cache files can consume significant disk space. Cache compression reduces storage requirements by compressing intermediate cache files after each operator completes.

Supported Algorithms

Algorithm	Library	Speed	Compression Ratio	Recommended For
zstd	zstandard	Fast	High	General use (default)
lz4	lz4	Fastest	Moderate	Speed-critical workloads
gzip	gzip	Slow	High	Compatibility needs

Configuration

use_cache: true
cache_compress: zstd    # Enable zstd compression

dj-process --config config.yaml --cache_compress zstd

Multi-Process Compression

Cache compression supports parallel processing. The number of compression worker processes is controlled by the np parameter:

np: 4                   # Number of parallel workers (also used for compression)
cache_compress: zstd

Cache Control API

DatasetCacheControl

A context manager to temporarily enable or disable HuggingFace dataset caching within a specific scope:

from data_juicer.utils.cache_utils import DatasetCacheControl

# Temporarily disable caching
with DatasetCacheControl(on=False):
    # Operations here will not use cache
    result = dataset.map(my_function)

# Temporarily enable caching
with DatasetCacheControl(on=True):
    # Operations here will use cache
    result = dataset.map(my_function)

dataset_cache_control Decorator

A decorator for functions that need to control cache state:

from data_juicer.utils.cache_utils import dataset_cache_control

@dataset_cache_control(on=False)
def process_without_cache(dataset):
    return dataset.map(my_function)

CompressionOff

A context manager to temporarily disable cache compression:

from data_juicer.utils.compress import CompressionOff

with CompressionOff():
    # Cache compression is disabled in this scope
    result = dataset.map(my_function)

CompressManager

Low-level API for manual compression/decompression:

from data_juicer.utils.compress import CompressManager

manager = CompressManager(compressor_format="zstd")

# Compress a file
manager.compress("input.arrow", "input.arrow.zstd")

# Decompress a file
manager.decompress("input.arrow.zstd", "input.arrow")

CacheCompressManager

High-level API for managing HuggingFace dataset cache compression:

from data_juicer.utils.compress import CacheCompressManager

manager = CacheCompressManager(compressor_format="zstd")

# Compress previous dataset's cache files
manager.compress(prev_ds=previous_dataset, this_ds=current_dataset, num_proc=4)

# Decompress cache files for a dataset
manager.decompress(ds=dataset, num_proc=4)

# Clean up all compressed cache files
manager.cleanup_cache_files(ds=dataset)

Cache vs Checkpoint

Cache and checkpoint are mutually exclusive — enabling checkpoint automatically disables cache:

Feature	Cache	Checkpoint
Purpose	Accelerate repeated runs with same configuration	Fault recovery and resumption
Granularity	Per-operator result	Full dataset snapshot
Storage Location	HuggingFace cache directory	Work directory
Recovery Method	Automatic (hash-based)	Manual (config-based)
Compression	Supported (cache_compress)	Not applicable
Scenario	Iterative development, parameter tuning	Long-running production tasks

# Cache mode (default)
use_cache: true
use_checkpoint: false

# Checkpoint mode (cache auto-disabled)
use_cache: true           # Will be overridden to false
use_checkpoint: true

Disabling Cache and Temporary Directory

When use_cache: false or checkpoint mode is enabled (use_checkpoint: true), HuggingFace dataset caching is fully disabled. In this mode, DataJuicer writes intermediate files produced during operator processing to a temporary directory, and cleans them up automatically when processing completes. The temp_dir parameter controls where these intermediate files are stored.

Behavior

• Defaults to null: When not set, the operating system determines the temporary directory location (typically /tmp), equivalent to Python’s tempfile.gettempdir().

• Takes effect automatically when cache is disabled: Once caching is disabled, temp_dir is applied as the global temporary directory for the entire process via Python’s tempfile.tempdir, affecting all temporary files created through tempfile in the process.

• Cache compression is automatically disabled: When caching is disabled, cache_compress is automatically ignored and reset to null.

Configuration

use_cache: false
temp_dir: /data/dj-temp    # Custom temp directory; null means system default

dj-process --config config.yaml --use_cache false --temp_dir /data/dj-temp

Safety Notes

Set temp_dir with caution — an unsafe path can cause unexpected program behavior.

• Do not point to critical system directories (e.g., /, /usr, /etc). Automatic cleanup of temporary files may accidentally delete important files.

• Do not point to directories containing important data. Temporary file writes and cleanup operations may conflict with existing files.

• Ensure sufficient disk space: When cache is disabled, intermediate files are written and deleted dynamically during processing. Peak disk usage is approximately equal to the output size of a single operator.

• The directory is created automatically if it does not exist: DataJuicer calls os.makedirs to create the specified path if it is missing.

• temp_dir affects the entire process’s tempfile behavior: Because it sets the global tempfile.tempdir variable, this setting influences all components in the process that rely on tempfile, including third-party libraries.

Performance Considerations

When to Enable Cache

• Enable: For iterative development where you frequently re-run pipelines with minor changes

• Enable: When operators are computationally expensive and you want to skip already-computed steps

• Disable: For one-shot processing to avoid disk overhead

When to Enable Compression

• Enable: When dataset size exceeds tens of GB and disk space is limited

• Enable zstd: For the best balance of speed and compression ratio

• Enable lz4: When compression speed is critical

• Disable: When disk space is abundant and you want maximum processing speed

Troubleshooting

Cache files consuming too much disk space:

# Check cache directory size
du -sh ~/.cache/data_juicer/

# Enable compression
dj-process --config config.yaml --cache_compress zstd

Stale cache causing unexpected results:

# Clear HuggingFace dataset cache
rm -rf ~/.cache/huggingface/datasets/

# Or specify a fresh cache directory
dj-process --config config.yaml --ds_cache_dir /tmp/fresh-cache