data_juicer.ops.pipeline.ray_vllm_pipeline module

class data_juicer.ops.pipeline.ray_vllm_pipeline.RayVLLMEnginePipeline(accelerator_type: str | None = None, *args, **kwargs)

Bases: Pipeline

Pipeline for Ray vLLM engine.

__init__(accelerator_type: str | None = None, *args, **kwargs)

Initialization method. :param accelerator_type: The type of accelerator to use (e.g., “V100”, “A100”).

Default to None, meaning that only the CPU will be used.

Parameters:

• args – extra args

• kwargs – extra args

run(dataset: Dataset, *, exporter=None, tracer=None, reduce=True) → Dataset