data_juicer.core.executor.base module

class data_juicer.core.executor.base.ExecutorBase(cfg: Namespace | None = None)

Bases: ABC

abstractmethod __init__(cfg: Namespace | None = None)

abstractmethod run(load_data_np: Annotated[int, Gt(gt=0)] | None = None, skip_return=False)

Abstract method for ExecutorBase.run

Parameters:

load_data_np – deprecated; number of workers when loading the dataset. Still applied by the default executor and the Analyzer, which is where np and load_dataset_kwargs.num_proc replace it. The Ray executors have never applied it; their read parallelism comes from override_num_blocks and the cluster.