data_juicer_sandbox.hooks module

class data_juicer_sandbox.hooks.BaseHook(job_cfg, watcher, *args, **kwargs)

Bases: object

__init__(job_cfg, watcher, *args, **kwargs)

run(context_infos: ContextInfos)

hook(**kwargs)

specify_dj_and_extra_configs(allow_fail=False)

class data_juicer_sandbox.hooks.ProbeViaAnalyzerHook(job_cfg, watcher, *args, **kwargs)

Bases: BaseHook

The hook to probe dataset via Data-Juicer Analyzer.

Input:

• A data-juicer config.

Output:

• the path to export the analyzed dataset.

__init__(job_cfg, watcher, *args, **kwargs)

Initialize the hook for probing the data via Analyzer

Parameters:

• job_cfg – the job configs

• watcher – for watching the result

hook(**kwargs)

class data_juicer_sandbox.hooks.ProbeViaModelInferHook(job_cfg, watcher, *args, **kwargs)

Bases: BaseHook

__init__(job_cfg, watcher, *args, **kwargs)

Initialize the hook for probing the data via Model Infer

Parameters:

• job_cfg – the job configs

• watcher – for watching the result

hook(**kwargs)

class data_juicer_sandbox.hooks.GeneralProbeHook(job_cfg, watcher, *args, **kwargs)

Bases: BaseHook

__init__(job_cfg, watcher, *args, **kwargs)

hook(**kwargs)

class data_juicer_sandbox.hooks.RefineRecipeViaKSigmaHook(job_cfg, watcher, *args, **kwargs)

Bases: BaseHook

__init__(job_cfg, watcher, *args, **kwargs)

Initialize the hook for refining the recipe via K Sigma

Parameters:

• job_cfg – the job configs

• watcher – for watching the result

hook(**kwargs)

class data_juicer_sandbox.hooks.RefineRecipeViaModelFeedbackHook(job_cfg, watcher, *args, **kwargs)

Bases: BaseHook

__init__(job_cfg, watcher, *args, **kwargs)

Initialize the hook for refining the recipe via Model Feedback

Parameters:

• job_cfg – the job configs

• watcher – for watching the result

hook(**kwargs)

class data_juicer_sandbox.hooks.ProcessDataHook(job_cfg, watcher, *args, **kwargs)

Bases: BaseHook

__init__(job_cfg, watcher, *args, **kwargs)

Initialize the hook for processing the data via Data-Juicer

Parameters:

• job_cfg – the job configs

• watcher – for watching the result

hook(**kwargs)

class data_juicer_sandbox.hooks.DataPoolManipulationHook(job_cfg, watcher, *args, **kwargs)

Bases: BaseHook

Hook for data pool manipulation, including construction, combination, ranking, etc.

__init__(job_cfg, watcher, *args, **kwargs)

hook(**kwargs)

class data_juicer_sandbox.hooks.GeneralDataExecutorHook(job_cfg, watcher, *args, **kwargs)

Bases: BaseHook

__init__(job_cfg, watcher, *args, **kwargs)

hook(**kwargs)

class data_juicer_sandbox.hooks.TrainModelHook(job_cfg, watcher, *args, **kwargs)

Bases: BaseHook

__init__(job_cfg, watcher, *args, **kwargs)

Initialize the hook for model training

Parameters:

• job_cfg – the job configs

• watcher – for watching the result

hook(**kwargs)

class data_juicer_sandbox.hooks.InferModelHook(job_cfg, watcher, *args, **kwargs)

Bases: BaseHook

__init__(job_cfg, watcher, *args, **kwargs)

Initialize the hook for model training

Parameters:

• job_cfg – the job configs

• watcher – for watching the result

hook(**kwargs)

class data_juicer_sandbox.hooks.EvaluateDataHook(job_cfg, watcher, *args, **kwargs)

Bases: BaseHook

__init__(job_cfg, watcher, *args, **kwargs)

Initialize the hook for data evaluation

Parameters:

• job_cfg – the job configs

• watcher – for watching the result

hook(**kwargs)

class data_juicer_sandbox.hooks.EvaluateModelHook(job_cfg, watcher, *args, **kwargs)

Bases: BaseHook

__init__(job_cfg, watcher, *args, **kwargs)

Initialize the hook for model evaluation

Parameters:

• job_cfg – the job configs

• watcher – for watching the result

hook(**kwargs)

data_juicer_sandbox.hooks.register_hook(job_cfg, watcher)