data_juicer.utils.hdfs_utils module

HDFS utilities for Data-Juicer.

Provides unified HDFS filesystem creation for the default executor and Ray executor (PyArrow backend). PyArrow's HadoopFileSystem relies on libhdfs and a working Hadoop/JVM environment on every node, so make sure the following environment variables are configured on all nodes before use:

export HADOOP_HOME=/path/to/hadoop export JAVA_HOME=/path/to/java export CLASSPATH=$($HADOOP_HOME/bin/hadoop classpath --glob) export ARROW_LIBHDFS_DIR=$HADOOP_HOME/lib/native # depends on env

data_juicer.utils.hdfs_utils.parse_hdfs_path(path: str) → Tuple[str | None, int | None]

Parse the host and port from an HDFS URI.

e.g. 'hdfs://namenode:8020/user/data' -> ('namenode', 8020)

'hdfs://namenode/user/data' -> ('namenode', None) 'hdfs:///user/data' -> (None, None) # rely on default fs

参数:

path -- HDFS path to parse.

返回:

A tuple of (host, port). host/port may be None, in which case PyArrow will fall back to the default filesystem configured via Hadoop conf (fs.defaultFS).

data_juicer.utils.hdfs_utils.strip_hdfs_scheme(path: str) → str

Strip the hdfs:// scheme/authority from a path, returning the bare filesystem path that PyArrow HadoopFileSystem expects.

e.g. 'hdfs://namenode:8020/user/data/file.jsonl' -> '/user/data/file.jsonl'

参数:

path -- HDFS path to strip.

返回:

// scheme.

返回类型:

Bare filesystem path without the hdfs

data_juicer.utils.hdfs_utils.create_pyarrow_hdfs_filesystem(ds_config: Dict | None = None) → pyarrow.fs.HadoopFileSystem

Create a PyArrow HadoopFileSystem for reading/writing HDFS.

Configuration priority for host/port: 1. Explicit fields in ds_config ('hdfs_host', 'hdfs_port'). 2. Parsed from the 'path' field in ds_config. 3. Default ('default'), letting PyArrow use Hadoop conf (fs.defaultFS).

Optional ds_config fields:

• hdfs_host: namenode host, or 'default'.

• hdfs_port: namenode port (int).

• hdfs_user: user name for HDFS access.

• hdfs_kerb_ticket: path to the Kerberos ticket cache.

• hdfs_extra_conf: dict of extra Hadoop configurations.

参数:

ds_config -- Dataset/export configuration dictionary.

返回:

A configured pyarrow.fs.HadoopFileSystem instance.

data_juicer.utils.hdfs_utils.validate_hdfs_path(path: str) → None

Validate that a path is a valid HDFS path.

参数:

path -- Path to validate.

抛出:

ValueError -- If path doesn't start with 'hdfs://'.