API Reference

• data_juicer.core
  • Adapter
  • Analyzer
  • NestedDataset
  • ExecutorBase
  • ExecutorFactory
  • DefaultExecutor
  • Exporter
  • RayExporter
  • Monitor
  • Tracer
• data_juicer.ops
  • load_ops()
  • Filter
  • Mapper
  • Deduplicator
  • Selector
  • Grouper
  • Aggregator
• data_juicer.ops.filter
  • AlphanumericFilter
  • AudioDurationFilter
  • AudioNMFSNRFilter
  • AudioSizeFilter
  • AverageLineLengthFilter
  • CharacterRepetitionFilter
  • FlaggedWordFilter
  • ImageAestheticsFilter
  • ImageAspectRatioFilter
  • ImageFaceCountFilter
  • ImageFaceRatioFilter
  • ImageNSFWFilter
  • ImagePairSimilarityFilter
  • ImageShapeFilter
  • ImageSizeFilter
  • ImageTextMatchingFilter
  • ImageTextSimilarityFilter
  • ImageWatermarkFilter
  • LanguageIDScoreFilter
  • InContextInfluenceFilter
  • InstructionFollowingDifficultyFilter
  • LLMAnalysisFilter
  • LLMQualityScoreFilter
  • LLMPerplexityFilter
  • LLMDifficultyScoreFilter
  • LLMTaskRelevanceFilter
  • MaximumLineLengthFilter
  • PerplexityFilter
  • PhraseGroundingRecallFilter
  • SpecialCharactersFilter
  • SpecifiedFieldFilter
  • SpecifiedNumericFieldFilter
  • StopWordsFilter
  • SuffixFilter
  • TextActionFilter
  • TextEmbdSimilarityFilter
  • TextEntityDependencyFilter
  • TextLengthFilter
  • TextPairSimilarityFilter
  • TokenNumFilter
  • VideoAestheticsFilter
  • VideoAspectRatioFilter
  • VideoDurationFilter
  • VideoFramesTextSimilarityFilter
  • VideoMotionScoreFilter
  • VideoMotionScoreRaftFilter
  • VideoNSFWFilter
  • VideoOcrAreaRatioFilter
  • VideoResolutionFilter
  • VideoTaggingFromFramesFilter
  • VideoWatermarkFilter
  • WordRepetitionFilter
  • WordsNumFilter
  • GeneralFieldFilter
• data_juicer.ops.mapper
  • AudioAddGaussianNoiseMapper
  • AudioFFmpegWrappedMapper
  • CalibrateQAMapper
  • CalibrateQueryMapper
  • CalibrateResponseMapper
  • ChineseConvertMapper
  • CleanCopyrightMapper
  • CleanEmailMapper
  • CleanHtmlMapper
  • CleanIpMapper
  • CleanLinksMapper
  • DetectCharacterAttributesMapper
  • DetectCharacterLocationsMapper
  • DetectMainCharacterMapper
  • DialogIntentDetectionMapper
  • DialogSentimentDetectionMapper
  • DialogSentimentIntensityMapper
  • DialogTopicDetectionMapper
  • Difference_Area_Generator_Mapper
  • Difference_Caption_Generator_Mapper
  • DownloadFileMapper
  • ExpandMacroMapper
  • ExtractEntityAttributeMapper
  • ExtractEntityRelationMapper
  • ExtractEventMapper
  • ExtractKeywordMapper
  • ExtractNicknameMapper
  • ExtractSupportTextMapper
  • ExtractTablesFromHtmlMapper
  • FixUnicodeMapper
  • GenerateQAFromExamplesMapper
  • GenerateQAFromTextMapper
  • HumanPreferenceAnnotationMapper
  • ImageBlurMapper
  • ImageCaptioningFromGPT4VMapper
  • ImageCaptioningMapper
  • ImageDetectionYoloMapper
  • ImageDiffusionMapper
  • ImageFaceBlurMapper
  • ImageRemoveBackgroundMapper
  • ImageSegmentMapper
  • ImageTaggingMapper
  • MllmMapper
  • NlpaugEnMapper
  • NlpcdaZhMapper
  • OptimizePromptMapper
  • OptimizeQAMapper
  • OptimizeQueryMapper
  • OptimizeResponseMapper
  • PairPreferenceMapper
  • PunctuationNormalizationMapper
  • PythonFileMapper
  • PythonLambdaMapper
  • QuerySentimentDetectionMapper
  • QueryIntentDetectionMapper
  • QueryTopicDetectionMapper
  • RelationIdentityMapper
  • RemoveBibliographyMapper
  • RemoveCommentsMapper
  • RemoveHeaderMapper
  • RemoveLongWordsMapper
  • RemoveNonChineseCharacterlMapper
  • RemoveRepeatSentencesMapper
  • RemoveSpecificCharsMapper
  • RemoveTableTextMapper
  • RemoveWordsWithIncorrectSubstringsMapper
  • ReplaceContentMapper
  • SDXLPrompt2PromptMapper
  • SentenceAugmentationMapper
  • SentenceSplitMapper
  • TextChunkMapper
  • VggtMapper
  • VideoCaptioningFromAudioMapper
  • VideoCaptioningFromFramesMapper
  • VideoCaptioningFromSummarizerMapper
  • VideoCaptioningFromVideoMapper
  • VideoExtractFramesMapper
  • VideoFFmpegWrappedMapper
  • VideoFaceBlurMapper
  • VideoRemoveWatermarkMapper
  • VideoResizeAspectRatioMapper
  • VideoResizeResolutionMapper
  • VideoSplitByDurationMapper
  • VideoSplitByKeyFrameMapper
  • VideoSplitBySceneMapper
  • VideoTaggingFromAudioMapper
  • VideoTaggingFromFramesMapper
  • WhitespaceNormalizationMapper
• data_juicer.ops.deduplicator
  • DocumentDeduplicator
  • DocumentMinhashDeduplicator
  • DocumentSimhashDeduplicator
  • ImageDeduplicator
  • RayBasicDeduplicator
  • RayDocumentDeduplicator
  • RayImageDeduplicator
  • RayVideoDeduplicator
  • RayBTSMinhashDeduplicator
  • VideoDeduplicator
• data_juicer.ops.selector
  • FrequencySpecifiedFieldSelector
  • RandomSelector
  • RangeSpecifiedFieldSelector
  • TagsSpecifiedFieldSelector
  • TopkSpecifiedFieldSelector
• data_juicer.ops.common
  • get_sentences_from_document()
  • get_words_from_document()
  • merge_on_whitespace_tab_newline()
  • split_on_newline_tab_whitespace()
  • split_on_whitespace()
  • strip()
  • words_augmentation()
  • words_refinement()
  • split_text_by_punctuation()
• data_juicer.analysis
  • ColumnWiseAnalysis
  • CorrelationAnalysis
  • DiversityAnalysis
  • OverallAnalysis
• data_juicer.config
  • init_configs()
  • get_init_configs()
  • export_config()
  • merge_config()
  • prepare_side_configs()
  • get_default_cfg()
  • prepare_cfgs_for_export()
  • update_op_attr()
• data_juicer.format
  • JsonFormatter
  • LocalFormatter
  • RemoteFormatter
  • TextFormatter
  • ParquetFormatter
  • CsvFormatter
  • TsvFormatter
  • EmptyFormatter
  • RayEmptyFormatter