data_juicer.utils.unittest_utils module

data_juicer.utils.unittest_utils.TEST_TAG(*tags)

Tags for test case. Currently, standalone, ray are supported.

data_juicer.utils.unittest_utils.set_clear_model_flag(flag)

data_juicer.utils.unittest_utils.set_from_fork_flag(flag)

class data_juicer.utils.unittest_utils.DataJuicerTestCaseBase(methodName='runTest')

Bases: TestCase

classmethod setUpClass()

Hook method for setting up class fixture before running tests in the class.

classmethod tearDownClass(hf_model_name=None) → None

Hook method for deconstructing the class fixture after running all tests in the class.

setUp()

Hook method for setting up the test fixture before exercising it.

tearDown() → None

Hook method for deconstructing the test fixture after testing it.

generate_dataset(data) → DJDataset

Generate dataset for a specific executor.

Parameters:

• type (str, optional) – “standalone” or “ray”.

• "standalone". (Defaults to)

run_single_op(dataset: DJDataset, op, column_names)

Run operator in the specific executor.

assertDatasetEqual(first, second)

assertListOfDictEqual(first: List[Dict], second: List[Dict], ignore_order=True)

Assert two list of dicts are equal

data_juicer.utils.unittest_utils.get_diff_files(prefix_filter=['data_juicer/', 'tests/'])

Get git diff files in target dirs except the __init__.py files

data_juicer.utils.unittest_utils.find_corresponding_test_file(file_path)

data_juicer.utils.unittest_utils.get_partial_test_cases()