API#
- data_juicer.core
- data_juicer.ops
- data_juicer.ops.filter
AlphanumericFilterAudioDurationFilterAudioNMFSNRFilterAudioSizeFilterAverageLineLengthFilterCharacterRepetitionFilterFlaggedWordFilterImageAestheticsFilterImageAspectRatioFilterImageFaceCountFilterImageFaceRatioFilterImageNSFWFilterImagePairSimilarityFilterImageShapeFilterImageSizeFilterImageSubplotFilterImageTextMatchingFilterImageTextSimilarityFilterImageWatermarkFilterLanguageIDScoreFilterInContextInfluenceFilterInstructionFollowingDifficultyFilterLLMAnalysisFilterLLMQualityScoreFilterLLMPerplexityFilterLLMDifficultyScoreFilterLLMTaskRelevanceFilterMaximumLineLengthFilterPerplexityFilterPhraseGroundingRecallFilterSpecialCharactersFilterSpecifiedFieldFilterSpecifiedNumericFieldFilterLLMConditionFilterStopWordsFilterSuffixFilterTextActionFilterTextEmbdSimilarityFilterTextEntityDependencyFilterTextLengthFilterTextPairSimilarityFilterTokenNumFilterVideoAestheticsFilterVideoAspectRatioFilterVideoDurationFilterVideoFramesTextSimilarityFilterVideoMotionScoreFilterVideoMotionScorePtlflowFilterVideoMotionScoreRaftFilterVideoNSFWFilterVideoOcrAreaRatioFilterVideoResolutionFilterVideoTaggingFromFramesFilterVideoWatermarkFilterWordRepetitionFilterWordsNumFilterGeneralFieldFilter
- data_juicer.ops.mapper
AgentBadCaseSignalMapperAgentDialogNormalizeMapperAgentInsightLLMMapperAgentSkillInsightMapperAgentToolTypeMapperAgentToolRelevanceMapperAgentTraceCoherenceMapperAudioAddGaussianNoiseMapperAudioFFmpegWrappedMapperCalibrateQAMapperCalibrateQueryMapperCalibrateResponseMapperChineseConvertMapperCleanCopyrightMapperCleanEmailMapperCleanHtmlMapperCleanIpMapperCleanLinksMapperDetectCharacterAttributesMapperDetectCharacterLocationsMapperDetectMainCharacterMapperDialogIntentDetectionMapperDialogSentimentDetectionMapperDialogSentimentIntensityMapperDialogTopicDetectionMapperDialogClarificationQualityMapperDialogCoreferenceMapperDialogErrorRecoveryMapperDialogMemoryConsistencyMapperDialogNonRepetitionMapperDialogProactivityMapperDialogTopicShiftMapperDifference_Area_Generator_MapperDifference_Caption_Generator_MapperDownloadFileMapperExpandMacroMapperExtractEntityAttributeMapperExtractEntityRelationMapperExtractEventMapperExtractKeywordMapperExtractNicknameMapperExtractSupportTextMapperExtractTablesFromHtmlMapperFixUnicodeMapperGenerateQAFromExamplesMapperGenerateQAFromTextMapperHumanPreferenceAnnotationMapperImageBlurMapperImageCaptioningMapperImageDetectionYoloMapperImageDiffusionMapperImageMMPoseMapperImageFaceBlurMapperImageRemoveBackgroundMapperImageSAM3DBodyMapperImageSegmentMapperImageTaggingMapperImageTaggingVLMMapperLatexFigureContextExtractorMapperLatexMergeTexMapperMllmMapperNlpaugEnMapperNlpcdaZhMapperOptimizePromptMapperOptimizeQAMapperOptimizeQueryMapperOptimizeResponseMapperPairPreferenceMapperPiiLlmSuspectMapperPiiRedactionMapperPunctuationNormalizationMapperPythonFileMapperPythonLambdaMapperQuerySentimentDetectionMapperQueryIntentDetectionMapperQueryTopicDetectionMapperRelationIdentityMapperRemoveBibliographyMapperRemoveCommentsMapperRemoveHeaderMapperRemoveLongWordsMapperRemoveNonChineseCharacterlMapperRemoveRepeatSentencesMapperRemoveSpecificCharsMapperRemoveTableTextMapperRemoveWordsWithIncorrectSubstringsMapperReplaceContentMapperS3DownloadFileMapperLLMExtractMapperS3UploadFileMapperSDXLPrompt2PromptMapperSentenceAugmentationMapperSentenceSplitMapperTextChunkMapperTextTaggingByPromptMapperToolSuccessTaggerMapperUsageCounterMapperVggtMapperVideoCameraCalibrationStaticDeepcalibMapperVideoCameraCalibrationStaticMogeMapperVideoCaptioningFromAudioMapperVideoCaptioningFromFramesMapperVideoCaptioningFromSummarizerMapperVideoCaptioningFromVideoMapperVideoCaptioningFromVLMMapperVideoDepthEstimationMapperVideoExtractFramesMapperVideoFFmpegWrappedMapperVideoHandReconstructionHaworMapperVideoHandReconstructionMapperVideoFaceBlurMapperVideoObjectSegmentingMapperVideoRemoveWatermarkMapperVideoResizeAspectRatioMapperVideoResizeResolutionMapperVideoSplitByDurationMapperVideoSplitByKeyFrameMapperVideoSplitBySceneMapperVideoTaggingFromAudioMapperVideoTaggingFromFramesMapperVideoUndistortMapperVideoWholeBodyPoseEstimationMapperWhitespaceNormalizationMapper
- data_juicer.ops.deduplicator
DocumentDeduplicatorDocumentLineDeduplicatorDocumentMinhashDeduplicatorDocumentMinhashDeduplicatorWithUidDocumentSimhashDeduplicatorImageDeduplicatorRayBasicDeduplicatorRayDocumentDeduplicatorRayImageDeduplicatorRayVideoDeduplicatorRayBTSMinhashDeduplicatorRayBTSMinhashDeduplicatorWithUidRayBTSMinhashCppDeduplicatorVideoDeduplicator
- data_juicer.ops.selector
- data_juicer.ops.common
- data_juicer.analysis
- data_juicer.config
- data_juicer.format