Documentation¶
Welcome to the Data-Juicer documentation.
New to Data-Juicer? Follow Getting Started in order: install it, walk through the Quick Start, then explore more resources in the DJ-Cookbook.
Building your own recipes? The Guides cover processing, analysis, dataset configuration, export, caching, and tracing.
Looking up configuration or operators? Use Lookup Materials for global options and the operator overview; use the top-level API tab for Python interfaces.
Processing at scale? See Distributed Processing for Ray mode, partitioning/checkpointing, and job management.
Extending Data-Juicer? Extension & Development covers operator plugins, the API service, and the developer guide.
Getting Started
Guides
Lookup Materials
Distributed Processing
Extension & Development
Resources
demos
- Demos
- Agent Quality & Bad-Case Pipeline
- VLA Visualization Demo
- Elastic Multi-Node Sharding on Shared Storage
- Overview
- Key advantages
- Comparison with existing approaches
- Files
- Important path concepts
- Requirements and current scope
- PAI-DLC Worker-broadcast quick start
- GPU smoke test: one recipe with CPU and GPU operators
- Important: use an existing Mapper/Filter recipe and your own JSONL
dlc_job.pyparameter reference- Complete
shard_job.pyparameter reference - Generic manual or scheduler workflow
- Ray execution modes
- Job directory layout
- Failure semantics
- Exit codes
- Troubleshooting
- Tests
- Note for dataset path
- HumanVBench Operators Demo
tools
- Distributed Fuzzy Deduplication Tools
- Auto Evaluation Toolkit
- GPT EVAL: Evaluate your model with OpenAI API
- Evaluation Results Recorder
- Format Conversion Tools
- Multimodal Tools
- Post Tuning Tools
- Label Studio Service Utility
- Metrics for video generation
- VBench metrics
- Postprocess tools
- Preprocess Tools