DOCS

Tutorial

• DJ-Cookbook
  • Curated Resources
  • Coding with Data-Juicer (DJ)
  • Use Cases & Data Recipes
  • Interactive Examples
• Installation Guide
  • Prerequisites
  • Basic DJ Installation
  • Scenario-based Installation
  • Common Installation Patterns
  • Installation From Source
  • Installation for Specific OPs
  • Installation Using Docker
  • Notes & Troubleshooting
• Quick Start
  • Dataset Configuration
  • Data Processing
  • Distributed Data Processing
  • Data Analysis
  • Data Visualization
  • Build Up Config Files
  • Sandbox
  • Preprocess Raw Data (Optional)
  • For Docker Users

docs

• Operator Schemas 算子提要
  • Overview 概览
  • aggregator
  • deduplicator
  • filter
  • formatter
  • grouper
  • mapper
  • pipeline
  • selector
  • Contributing 贡献
• Dataset Configuration Guide
  • Supported Dataset Formats
  • Other features
• “Bad” Data Exhibition
  • Table of Contents
  • Involved OPs
  • Multimodal Datasets
  • Text-only Datasets
• Cache Management
  • Overview
  • Configuration
  • Cache Directory Structure
  • Cache Compression
  • Cache Control API
  • Cache vs Checkpoint
  • Disabling Cache and Temporary Directory
  • Performance Considerations
  • Troubleshooting
• DJ-SORA
  • Motivation
  • Roadmap
• DJ_service
  • API Service
  • MCP Server
• How-to Guide for Developers
  • 1. Build Your Own OPs Quickly
  • 2. Build Your Own Data Recipes and Configs
  • 3. Dependency Management
  • 4. Contribution to the Open-source Community
• Distributed Data Processing in Data-Juicer
  • Overview
  • Implementation and Optimizations
  • Performance Results
  • Quick Start
• Dataset Export
  • Overview
  • Configuration
  • Supported Formats
  • Shard Export
  • Parallel Export
  • S3 Export
  • Stats and Hash Management
  • WebDataset Export (Ray Mode)
  • API Reference
  • Troubleshooting
• Job Management
  • Processing Snapshot
  • Resource-Aware Partitioning
  • Logging
  • API Reference
  • Troubleshooting
• Partitioned Processing with Checkpointing
  • Overview
  • Directory Structure
  • Configuration
  • Usage
  • Auto-Configuration
  • Job Management Utilities
  • Event Types
  • Performance Considerations
  • Troubleshooting
• Data Tracing
  • Overview
  • Configuration
  • Output Structure
  • Traced Operator Types
  • Sample Collection Behavior
  • trace_keys
  • API Reference
  • Performance Considerations
  • Troubleshooting
• Awesome Data-Model Co-Development of MLLMs
  • News
  • Candidate Co-Development Tags
  • Paper List
  • Contribution to This Survey
  • “Section - Mentioned Papers” Retrieval List

demos

• Demos
  • Usage
  • Available Demos
• Note for dataset path

tools

• Distributed Fuzzy Deduplication Tools
  • Usage
• Auto Evaluation Toolkit
  • Preparation
  • Usage
  • Configuration
• GPT EVAL: Evaluate your model with OpenAI API
  • Quick Start
• Evaluation Results Recorder
  • Usage
  • Configuration
• Format Conversion Tools
• Multimodal Tools
  • Convert Absolute Path to Relative Path
  • Dataset Format Conversion
• Post Tuning Tools
  • Usage
• Label Studio Service Utility
  • Features
  • Usage
  • Connection File: label_studio_localhost_connection.json
  • Enabling Legacy Token Authentication
  • Label Studio Version
  • Example
  • Troubleshooting
• Metrics for video generation
  • Usage
  • Introduction of Metrics
• VBench metrics
• Postprocess tools
  • Usage
• Preprocess Tools
  • Usage

thirdparty

• LLM Ecosystems
  • Installation
• Third-party Model Library
  • EasyAnimate