data_juicer.utils.hdfs_utils module#

HDFS utilities for Data-Juicer.

Provides unified HDFS filesystem creation for the default executor and Ray executor (PyArrow backend). PyArrow’s HadoopFileSystem relies on libhdfs and a working Hadoop/JVM environment on every node, so make sure the following environment variables are configured on all nodes before use:

export HADOOP_HOME=/path/to/hadoop export JAVA_HOME=/path/to/java export CLASSPATH=$($HADOOP_HOME/bin/hadoop classpath –glob) export ARROW_LIBHDFS_DIR=$HADOOP_HOME/lib/native # depends on env

data_juicer.utils.hdfs_utils.parse_hdfs_path(path: str) → Tuple[str | None, int | None][source]#

Parse the host and port from an HDFS URI.

e.g. ‘hdfs://namenode:8020/user/data’ -> (‘namenode’, 8020)

‘hdfs://namenode/user/data’ -> (‘namenode’, None) ‘hdfs:///user/data’ -> (None, None) # rely on default fs

Parameters:

path – HDFS path to parse.

Returns:

A tuple of (host, port). host/port may be None, in which case PyArrow will fall back to the default filesystem configured via Hadoop conf (fs.defaultFS).

data_juicer.utils.hdfs_utils.strip_hdfs_scheme(path: str) → str[source]#

Strip the hdfs:// scheme/authority from a path, returning the bare filesystem path that PyArrow HadoopFileSystem expects.

e.g. ‘hdfs://namenode:8020/user/data/file.jsonl’ -> ‘/user/data/file.jsonl’

Parameters:

path – HDFS path to strip.

Returns:

// scheme.

Return type:

Bare filesystem path without the hdfs

data_juicer.utils.hdfs_utils.create_pyarrow_hdfs_filesystem(ds_config: Dict | None = None) → pyarrow.fs.HadoopFileSystem[source]#

Create a PyArrow HadoopFileSystem for reading/writing HDFS.

Configuration priority for host/port: 1. Explicit fields in ds_config (‘hdfs_host’, ‘hdfs_port’). 2. Parsed from the ‘path’ field in ds_config. 3. Default (‘default’), letting PyArrow use Hadoop conf (fs.defaultFS).

Optional ds_config fields:

  • hdfs_host: namenode host, or ‘default’.

  • hdfs_port: namenode port (int).

  • hdfs_user: user name for HDFS access.

  • hdfs_kerb_ticket: path to the Kerberos ticket cache.

  • hdfs_extra_conf: dict of extra Hadoop configurations.

Parameters:

ds_config – Dataset/export configuration dictionary.

Returns:

A configured pyarrow.fs.HadoopFileSystem instance.

data_juicer.utils.hdfs_utils.validate_hdfs_path(path: str) → None[source]#

Validate that a path is a valid HDFS path.

Parameters:

path – Path to validate.

Raises:

ValueError – If path doesn’t start with ‘hdfs://’.