DOCS

Tutorial

• DJ-Cookbook
  • Curated Resources
  • Coding with Data-Juicer (DJ)
  • Use Cases & Data Recipes
  • Interactive Examples
• Installation Guide
  • Prerequisites
  • Basic DJ Installation
  • Scenario-based Installation
  • Common Installation Patterns
  • Installation for Specific OPs
  • Installation Using Docker
  • Notes & Troubleshooting
• Quick Start
  • Dataset Configuration
  • Data Processing
  • Distributed Data Processing
  • Data Analysis
  • Data Visualization
  • Build Up Config Files
  • Sandbox
  • Preprocess Raw Data (Optional)
  • For Docker Users

docs

• Operator Schemas 算子提要
  • Overview 概览
  • aggregator
  • deduplicator
  • filter
  • formatter
  • grouper
  • mapper
  • selector
  • Contributing 贡献
• Dataset Configuration Guide
  • Supported Dataset Formats
  • Other features
• “Bad” Data Exhibition
  • Table of Contents
  • Involved OPs
  • Multimodal Datasets
  • Text-only Datasets
• DJ-SORA
  • Motivation
  • Roadmap
• DataJuicer-Agent
• DJ_service
  • API Service
  • MCP Server
• How-to Guide for Developers
  • 1. Build Your Own OPs Quickly
  • 2. Build Your Own Data Recipes and Configs
  • 3. Dependency Management
  • 4. Contribution to the Open-source Community
• Distributed Data Processing in Data-Juicer
  • Overview
  • Implementation and Optimizations
  • Performance Results
  • Quick Start
• Data Recipe Gallery
  • 1. Data-Juicer Minimal Example Recipe
  • 2. Reproduce Open Source Text Datasets
  • 3. Improved Open Source Pre-training Text Datasets
  • 4. Improved Open Source Post-tuning Text Dataset
  • 5. Synthetic Contrastive Learning Image-text datasets
  • 6. Improved Open Source Image-text datasets
  • 7. Basic Example Recipes for Video Data
  • 8. Synthesize Human-centric Video Benchmarks
  • 9. Improve Existing Open Source Video Datasets
• Sandbox
  • User Guide
  • Developer Guide
  • Q&A
• Awesome Data-Model Co-Development of MLLMs
  • News
  • Candidate Co-Development Tags
  • Paper List
  • Contribution to This Survey
  • “Section - Mentioned Papers” Retrieval List

demos

• Demos
  • Usage
  • Available Demos
• Note for dataset path

tools

• Distributed Fuzzy Deduplication Tools
  • Usage
• Auto Evaluation Toolkit
  • Preparation
  • Usage
  • Configuration
• GPT EVAL: Evaluate your model with OpenAI API
  • Quick Start
• Evaluation Results Recorder
  • Usage
  • Configuration
• Format Conversion Tools
• Multimodal Tools
  • Convert Absolute Path to Relative Path
  • Dataset Format Conversion
• Post Tuning Tools
  • Usage
• Hyper-parameter Optimization for Data Recipe
  • Auto-HPO based on 3-Sigma principles
  • Auto-HPO with WandB
• Label Studio Service Utility
  • Features
  • Usage
  • Connection File: label_studio_localhost_connection.json
  • Enabling Legacy Token Authentication
  • Label Studio Version
  • Example
  • Troubleshooting
• Metrics for video generation
  • Usage
  • Introduction of Metrics
• VBench metrics
• Postprocess tools
  • Usage
• Preprocess Tools
  • Usage
• Data Scoring
  • Data Scoring Capabilities
  • Quality Classifier Toolkit (GPT-3 Reproduced)

thirdparty

• LLM Ecosystems
  • Installation
• Third-party Model Library
  • EasyAnimate