Demos#

This folder contains both interactive demos for experiencing Data-Juicer and functional examples that demonstrate specific features and workflows.

Sample datasets used by the demos and examples are available in the data subdirectory.

Interactive Demos#

Run an interactive demo with its app.py:

cd <subdir_of_demos>
streamlit run app.py

  • Overview scan (overview_scan)

    • This demo introduces the basic concepts and functions of Data-Juicer, such as features, configuration, operators, and so on.

  • Data process loop (data_process_loop)

    • This demo analyzes and processes a dataset, providing a comparison of statistical information before and after the processing.

  • Data visualization diversity (data_visualization_diversity)

    • This demo analyzes the verb-noun structure of the CFT dataset and plots its diversity in sunburst format.

  • Data visualization op effect (data_visualization_op_effect)

    • This demo analyzes the statistics of dataset, and displays the effect of each Filter op by setting different thresholds.

  • Data visualization statistics (data_visualization_statistics)

    • This demo analyzes the dataset and obtain up to 13 statistics.

  • Process CFT Chinese data (process_cft_zh_data)

    • This demos analyzes and processes part of Chinese dataset in Alpaca-CoT to show how to process IFT or CFT data for LLM fine-tuning.

  • Process SCI data (process_sci_data)

    • This demos analyzes and processes part of arXiv dataset to show how to process scientific literature data for LLM pre-training.

  • Process code data (process_code_data)

    • This demos analyzes and processes part of Stack-Exchange dataset to show how to process code data for LLM pre-training.

  • Text quality classifier (tool_quality_classifier)

    • This demo provides 3 text quality classifier to score the dataset.

  • Dataset splitting by language (tool_dataset_splitting_by_language)

    • This demo splits a dataset to different sub-datasets by language.

  • Data mixture (data_mixture)

    • This demo selects and mixes samples from multiple datasets and exports them into a new dataset.

Functional Examples#

Follow the instructions in the corresponding subdirectory to run these examples.

  • Partition and checkpoint (partition_and_checkpoint)

    • This demo showcases distributed processing with partitioning, checkpointing, and event logging. It demonstrates the new job management features including resource-aware partitioning, comprehensive event logging, and the processing snapshot utility for monitoring job progress.

  • Elastic sharding (elastic_sharding)

    • This demo pre-splits a JSONL dataset, launches any number of DLC workers with one command, dynamically claims shards through a shared POSIX filesystem, and processes each claimed shard with node-local Ray. It includes CPU-only and mixed CPU/GPU recipes.