data_juicer.ops.fused_batch_executor module

Shared batch-local execution helpers for sequential fused operators.

class data_juicer.ops.fused_batch_executor.SequentialBatchExecutionPolicy(copy_input: bool = False, shared_context: bool = False, use_op_wrappers: bool = True, validate: bool = True, ensure_fields: bool = True)

Bases: object

Behavior owned by an outer fused operator, not by the shared loop.

copy_input: bool = False

shared_context: bool = False

use_op_wrappers: bool = True

validate: bool = True

ensure_fields: bool = True

__init__(copy_input: bool = False, shared_context: bool = False, use_op_wrappers: bool = True, validate: bool = True, ensure_fields: bool = True) → None

data_juicer.ops.fused_batch_executor.get_batch_size(samples: Any) → int

data_juicer.ops.fused_batch_executor.execute_sequential_batch(samples: Any, ops: Iterable[Any], *, rank=None, owner_name: str = 'fused op', policy: SequentialBatchExecutionPolicy = SequentialBatchExecutionPolicy(copy_input=False, shared_context=False, use_op_wrappers=True, validate=True, ensure_fields=True), cleanup_columns: Iterable[str] | None = None, on_op_complete: Callable[[Any, float], None] | None = None) → Any

Execute mapper/filter ops in order inside one batch stage.

Construction, resource planning, and dataset-level preparation remain the responsibility of the outer fused operator. This helper owns only the shared batch execution semantics.