Doc API

Sandbox Hub Agents

English 简体中文

main

Home

Getting Started

Installation Guide
Quickstart
DJ-Cookbook

Guides

Processing Data
Data Analysis
Web Playground
Dataset Configuration
Dataset Export
Cache Management
Data Tracing

Lookup Materials

Global Configuration Reference
Operator Schemas 算子提要

Distributed Processing

Distributed Data Processing in Data-Juicer
Partitioned Processing with Checkpointing
Job Management

Extension & Development

Operator Plugins
DJ Service
DJ-SORA
Juicer: Natural-Language Data Refinement Model
How-to Guide for Developers

Resources

Awesome Data-Model Co-Development of MLLMs
“Bad” Data Exhibition

operators

Aggregator
Deduplicator
Filter
Mapper
Formatter
Grouper
Selector
Pipeline

demos

Demos
Agent Quality & Bad-Case Pipeline
VLA Visualization Demo
Elastic Multi-Node Sharding on Shared Storage
Note for dataset path
HumanVBench Operators Demo

tools

Distributed Fuzzy Deduplication Tools
Auto Evaluation Toolkit
GPT EVAL: Evaluate your model with OpenAI API
Evaluation Results Recorder
Format Conversion Tools
Multimodal Tools
Post Tuning Tools
Label Studio Service Utility
Metrics for video generation
VBench metrics
Postprocess tools
Preprocess Tools

thirdparty

LLM Ecosystems
HumanVBench Models Setup
Third-party Model Library

Related projects

Sandbox Hub Agents

Pipeline¶

llm_ray_vllm_engine_pipeline
ray_repartition_pipeline
vlm_ray_vllm_engine_pipeline

On this page

Pipeline

Previous topk_specified_field_selector Next llm_ray_vllm_engine_pipeline

© 2024, Data-Juicer Team — Built with Sphinx & Data-Juicer Theme

↵ to select ↑↓ to navigate esc to close