DOCS

Tutorial

• DJ-Cookbook
  • Curated Resources
  • Coding with Data-Juicer (DJ)
  • Use Cases & Data Recipes
  • Interactive Examples
• Installation Guide
  • Prerequisites
  • Basic DJ Installation
  • Scenario-based Installation
  • Common Installation Patterns
  • Installation From Source
  • Installation for Specific OPs
  • Installation Using Docker
  • Notes & Troubleshooting
• Quick Start
  • Dataset Configuration
  • Data Processing
  • Distributed Data Processing
  • Data Analysis
  • Data Visualization
  • Build Up Config Files
  • Sandbox
  • Preprocess Raw Data (Optional)
  • For Docker Users

docs

• Operator Schemas 算子提要
  • Overview 概览
  • aggregator
  • deduplicator
  • filter
  • formatter
  • grouper
  • mapper
  • pipeline
  • selector
  • Contributing 贡献
• Dataset Configuration Guide
  • Supported Dataset Formats
  • Other features
• “Bad” Data Exhibition
  • Table of Contents
  • Involved OPs
  • Multimodal Datasets
  • Text-only Datasets
• Cache Management
  • Overview
  • Configuration
  • Cache Directory Structure
  • Cache Compression
  • Cache Control API
  • Cache vs Checkpoint
  • Disabling Cache and Temporary Directory
  • Performance Considerations
  • Troubleshooting
• DJ-SORA
  • Motivation
  • Roadmap
• DJ_service
  • API Service
  • MCP Server
• How-to Guide for Developers
  • 1. Build Your Own OPs Quickly
  • 2. Build Your Own Data Recipes and Configs
  • 3. Dependency Management
  • 4. Contribution to the Open-source Community
• Distributed Data Processing in Data-Juicer
  • Overview
  • Implementation and Optimizations
  • Performance Results
  • Quick Start
• Dataset Export
  • Overview
  • Configuration
  • Supported Formats
  • Shard Export
  • Parallel Export
  • S3 Export
  • Stats and Hash Management
  • WebDataset Export (Ray Mode)
  • API Reference
  • Troubleshooting
• Job Management
  • Processing Snapshot
  • Resource-Aware Partitioning
  • Logging
  • API Reference
  • Troubleshooting
• Partitioned Processing with Checkpointing
  • Overview
  • Directory Structure
  • Configuration
  • Usage
  • Auto-Configuration
  • Job Management Utilities
  • Event Types
  • Performance Considerations
  • Troubleshooting
• Data Tracing
  • Overview
  • Configuration
  • Output Structure
  • Traced Operator Types
  • Sample Collection Behavior
  • trace_keys
  • API Reference
  • Performance Considerations
  • Troubleshooting
• Awesome Data-Model Co-Development of MLLMs
  • News
  • Candidate Co-Development Tags
  • Paper List
  • Contribution to This Survey
  • “Section - Mentioned Papers” Retrieval List

demos

• Demos
  • Usage
  • Available Demos
• Agent 交互数据：Bad case 洞察
  • Overview
  • Agent 对话形态：多段 assistant 与 tool
  • 数据血缘字段（normalize 自动写入）
  • Upstream 信号覆盖（第 9 步尽量吃满 pipeline）
  • 第 10 步：agent_insight_llm_mapper（auto-analyst）
  • 为何不强依赖「用户不满意」单一视角
  • 深挖思路（按 model / pt）
  • Pipeline 之后的分析脚本
  • jq 快速筛选
  • 相关算子
• Bad case 自助报告（简化入口）
  • 一行命令
  • 与 smoke / full 的关系
  • 直接调 Python（可选）
  • 仍是「进阶」的内容（刻意不收进报告页）
• Agent 流水线里 LLM 算子：加速与超参
  • 1. 并行（多进程发 API）
  • 2. 减少「每个算子」的工作量
  • 3. 减少 token（更快、更便宜）
  • 4. 迭代开发时的推荐组合
  • 5. 与本仓库脚本的衔接
• Bad case 流水线：一键运行与端到端指南
  • 只想看报告（最少步骤）
  • 前置条件
  • 一键命令（推荐）
  • 端到端两条路径
  • 脚本目录说明
  • LLM 算子慢、想加速？
  • 导出里的 meta 键名
  • 常见问题
• Agent quality & bad-case docs
• Agent 质检 / bad-case 文档索引
  • 维护指南
• Agent 流水线最小可运行配置（便于逐项调试）
  • 环境与依赖（参考 docs/DeveloperGuide.md）
  • 运行方式（在仓库根目录）
  • 配置说明摘要
  • 建议调试顺序
  • 运行环境注意
• Agent pipeline 后分析脚本
  • Python 脚本一览
  • 手动分步示例
  • 与端到端流程的对应关系
• Note for dataset path

tools

• Distributed Fuzzy Deduplication Tools
  • Usage
• Auto Evaluation Toolkit
  • Preparation
  • Usage
  • Configuration
• GPT EVAL: Evaluate your model with OpenAI API
  • Quick Start
• Evaluation Results Recorder
  • Usage
  • Configuration
• Format Conversion Tools
• Multimodal Tools
  • Convert Absolute Path to Relative Path
  • Dataset Format Conversion
• Post Tuning Tools
  • Usage
• Label Studio Service Utility
  • Features
  • Usage
  • Connection File: label_studio_localhost_connection.json
  • Enabling Legacy Token Authentication
  • Label Studio Version
  • Example
  • Troubleshooting
• Metrics for video generation
  • Usage
  • Introduction of Metrics
• VBench metrics
• Postprocess tools
  • Usage
• Preprocess Tools
  • Usage

thirdparty

• LLM Ecosystems
  • Installation
• Third-party Model Library
  • EasyAnimate