data_juicer.core.executor.dag_execution_mixin module#

DAG Execution Mixin for Data-Juicer Executors

This mixin provides DAG execution planning and monitoring that can be integrated into existing executors to provide intelligent pipeline analysis and execution tracking.

class data_juicer.core.executor.dag_execution_mixin.DAGExecutionMixin[source]#

Bases: object

Mixin that provides DAG-based execution planning and monitoring.

This mixin can be integrated into any executor to provide: - DAG execution planning - Execution monitoring tied to DAG nodes - Event logging with DAG context

__init__()[source]#

Initialize the DAG execution mixin.

get_dag_execution_status() → Dict[str, Any][source]#

Get DAG execution status.

visualize_dag_execution_plan() → str[source]#

Get visualization of the DAG execution plan.

get_dag_execution_plan_path() → str[source]#

Get the path to the saved DAG execution plan.

reconstruct_dag_state_from_events(job_id: str) → Dict[str, Any] | None[source]#

Reconstruct DAG execution state from event logs.

This method has been decomposed into smaller, focused methods for better maintainability and testability.

Parameters:

job_id – The job ID to analyze

Returns:

Dictionary containing reconstructed DAG state and resumption information

resume_dag_execution(job_id: str, dataset, ops: List) → bool[source]#

Resume DAG execution from the last known state.

Parameters:

  • job_id – The job ID to resume

  • dataset – The dataset to process

  • ops – List of operations to execute

Returns:

True if resumption was successful, False otherwise