Data-Juicer Agents: Towards Agentic Data Processing#

A Suite of Agents for Agentic Data Processing. Built on Data-Juicer (DJ) and AgentScope.

简体中文 | English

🏗️ Overview Doc • ⚡️ Quick Start Doc • >_ CLI Doc • 🔧 Tools Doc • 🎯 Roadmap

News#

🚀 [2026-03-11] Major refactor and upgrade of data_juicer_agents completed.
- The project architecture and CLI/session capabilities were comprehensively redesigned for better maintainability and extensibility.
- 🏗️ Overview | ⚡️ Quick Start | >_ CLI Doc | 🔧 Tools | 🎯 Roadmap
- Try processing data by chatting with the agent!
🚀[2026-01-15] Q&A Copilot has been deployed on the official Doc Site | DingTalk | Discord of Data-Juicer. Feel free to ask Juicer anything related to the Data-Juicer ecosystem!
- 📃 Deploy-ready codes | 🎬 More demos | 🎯 Roadmap.

Roadmap#

The long-term vision of DJ-Agents is to enable a development-free data processing lifecycle, allowing developers to focus on what to do rather than how to do it.

To achieve this vision, we are tackling two fundamental challenges:

Agents: How to design and build powerful agents specialized in data processing
Services & Tools: How to package these agents into ready-to-use, out-of-the-box products

We continuously iterate on both directions, and the roadmap may evolve accordingly as our understanding and capabilities improve.

Agents#

~~Data-Juicer Data Processing Agent (DJ Process Agent) & Data-Juicer Code Development Agent (DJ Dev Agent)~~
We have stopped building scenario-specific data processing agents, and instead are building data processing tools for general-purpose agents. From there:
- Hard-orchestrate these tools into capabilities, exposed as the djx CLI
- Soft-orchestrate them through prompts, packaged as skills
- Rely on agent self-orchestration to support conversational data processing

Services & Tools#

Q&A Copilot: a Q&A assistant for the Data-Juicer ecosystem
- [2026-01-15]: already deployed on the official Doc Site of Data-Juicer | DingTalk | Discord
InteRecipe: interactive data recipe construction through natural language
- [2026-03-11]: the current ./interactive_recipe only shows workflow-based examples. The dj-agents CLI entry is already built and supports interactive data-recipe construction through natural language in the TUI. We are developing a frontend tool (studio) on top of this foundation as the next upgrade.

Priority Items#

DJ Skills: use prompt-based soft orchestration to package tools into skills for general-purpose agents.
InteRecipe Studio: support interactive data recipe construction through natural language, with multi-dimensional data and result views.
Plan Tool: extend support for fuller Data-Juicer capability coverage, DJ Hub recipe matching, and more.
Dev Tool: stabilization testing and optimization

Long-term Directions#

Continue building tools and skills for broader data-processing scenarios, enabling wider and more flexible applications.
- RAG
- Embodied Intelligence
- Data Lakehouse architectures

Context Management#

Session agents keep full tool results in the ReAct memory, which grows the prompt cost every turn and can overflow small-context models. DJ Agents now bounds the model context with three layers, all enabled by default:

Tool-result compaction: deterministic, rule-based shrinking of large tool payloads before they enter memory (full payloads are still emitted through runtime events, so UIs and logs are unaffected).
Memory compression: when the visible history exceeds the trigger budget, older messages are summarized and only the recent messages stay intact.
Formatter hard budget: a final per-request truncation guarantees a single prompt never exceeds the configured window.

Configure via environment variables:

Variable	Default	Meaning
`DJA_CONTEXT_WINDOW_TOKENS`	`40000`	Model context window size in tokens
`DJA_CONTEXT_TRIGGER_RATIO`	`0.65`	Window fraction that triggers compression
`DJA_CONTEXT_FORMATTER_RATIO`	`0.85`	Window fraction for the formatter budget
`DJA_CONTEXT_KEEP_RECENT`	`10`	Recent messages kept intact on compression
`DJA_CONTEXT_COMPRESSION_ENABLED`	`true`	Toggle memory compression
`DJA_TOOL_RESULT_COMPACTION_ENABLED`	`true`	Toggle tool-result compaction
`DJA_CONTEXT_CHAR_PER_TOKEN`	`3`	Conservative chars-per-token estimate

For small-context deployments (30k–50k windows), lower the window and ratios, e.g. DJA_CONTEXT_WINDOW_TOKENS=30000 DJA_CONTEXT_TRIGGER_RATIO=0.55 DJA_CONTEXT_FORMATTER_RATIO=0.80 DJA_CONTEXT_KEEP_RECENT=8.

To measure how much a specific tool payload saves, use:

djx debug token-usage <tool_name> --input-file payload.json --provider char
djx debug token-usage <tool_name> --input-file payload.json --provider qwen

--provider char runs fully offline with the local AgentScope char counter; --provider qwen reports real usage.prompt_tokens from an OpenAI-compatible endpoint.

Common Issues#

Q: How to get DashScope API key? A: Visit DashScope official website to register an account and apply for an API key.

DOCS