Data-Juicer Agents: Towards Agentic Data Processing#

A Suite of Agents for Agentic Data Processing. Built on Data-Juicer (DJ) and AgentScope.

简体中文 | English

🏗️ Overview Doc • ⚡️ Quick Start Doc • >_ CLI Doc • 🔧 Tools Doc • 🎯 Roadmap

News#

Roadmap#

The long-term vision of DJ-Agents is to enable a development-free data processing lifecycle, allowing developers to focus on what to do rather than how to do it.

To achieve this vision, we are tackling two fundamental challenges:

  • Agents: How to design and build powerful agents specialized in data processing

  • Services & Tools: How to package these agents into ready-to-use, out-of-the-box products

We continuously iterate on both directions, and the roadmap may evolve accordingly as our understanding and capabilities improve.

Agents#

  • Data-Juicer Data Processing Agent (DJ Process Agent) & Data-Juicer Code Development Agent (DJ Dev Agent)

  • We have stopped building scenario-specific data processing agents, and instead are building data processing tools for general-purpose agents. From there:

    • Hard-orchestrate these tools into capabilities, exposed as the djx CLI

    • Soft-orchestrate them through prompts, packaged as skills

    • Rely on agent self-orchestration to support conversational data processing

Services & Tools#

  • Q&A Copilot: a Q&A assistant for the Data-Juicer ecosystem

  • InteRecipe: interactive data recipe construction through natural language

    • [2026-03-11]: the current ./interactive_recipe only shows workflow-based examples. The dj-agents CLI entry is already built and supports interactive data-recipe construction through natural language in the TUI. We are developing a frontend tool (studio) on top of this foundation as the next upgrade.

Priority Items#

  • DJ Skills: use prompt-based soft orchestration to package tools into skills for general-purpose agents.

  • InteRecipe Studio: support interactive data recipe construction through natural language, with multi-dimensional data and result views.

  • Plan Tool: extend support for fuller Data-Juicer capability coverage, DJ Hub recipe matching, and more.

  • Dev Tool: stabilization testing and optimization

Long-term Directions#

  • Continue building tools and skills for broader data-processing scenarios, enabling wider and more flexible applications.

    • RAG

    • Embodied Intelligence

    • Data Lakehouse architectures

Context Management#

Session agents keep full tool results in the ReAct memory, which grows the prompt cost every turn and can overflow small-context models. DJ Agents now bounds the model context with three layers, all enabled by default:

  1. Tool-result compaction: deterministic, rule-based shrinking of large tool payloads before they enter memory (full payloads are still emitted through runtime events, so UIs and logs are unaffected).

  2. Memory compression: when the visible history exceeds the trigger budget, older messages are summarized and only the recent messages stay intact.

  3. Formatter hard budget: a final per-request truncation guarantees a single prompt never exceeds the configured window.

Configure via environment variables:

Variable

Default

Meaning

DJA_CONTEXT_WINDOW_TOKENS

40000

Model context window size in tokens

DJA_CONTEXT_TRIGGER_RATIO

0.65

Window fraction that triggers compression

DJA_CONTEXT_FORMATTER_RATIO

0.85

Window fraction for the formatter budget

DJA_CONTEXT_KEEP_RECENT

10

Recent messages kept intact on compression

DJA_CONTEXT_COMPRESSION_ENABLED

true

Toggle memory compression

DJA_TOOL_RESULT_COMPACTION_ENABLED

true

Toggle tool-result compaction

DJA_CONTEXT_CHAR_PER_TOKEN

3

Conservative chars-per-token estimate

For small-context deployments (30k–50k windows), lower the window and ratios, e.g. DJA_CONTEXT_WINDOW_TOKENS=30000 DJA_CONTEXT_TRIGGER_RATIO=0.55 DJA_CONTEXT_FORMATTER_RATIO=0.80 DJA_CONTEXT_KEEP_RECENT=8.

To measure how much a specific tool payload saves, use:

djx debug token-usage <tool_name> --input-file payload.json --provider char
djx debug token-usage <tool_name> --input-file payload.json --provider qwen

--provider char runs fully offline with the local AgentScope char counter; --provider qwen reports real usage.prompt_tokens from an OpenAI-compatible endpoint.

Common Issues#

Q: How to get DashScope API key? A: Visit DashScope official website to register an account and apply for an API key.