data_juicer.tools.quality_classifier.eval module

data_juicer.tools.quality_classifier.eval.main(positive_datasets=None, negative_datasets=None, model='my_quality_model', tokenizer=None, text_key='text')

Evaluate a trained quality classifier using specific positive/negative datasets :param positive_datasets: the paths to the positive datasets. It could be a

string for a single dataset, e.g. 'pos.parquet', or a list of strings for multiple datasets, e.g. '["pos1.parquet", "pos2.parquet"]'

参数:

• negative_datasets -- the paths to the negative datasets. It could be a string for a single dataset, e.g. 'neg.parquet', or a list of strings for multiple datasets, e.g. '["neg1.parquet", "neg2.parquet"]'

• model -- quality classifier name to apply. It's "my_quality_model" in default. You can use one of ["gpt3", "chinese", "code"] we provided, or you can set it to the path to your own model trained using the train.py tool

• tokenizer -- what tokenizer to use to tokenize texts. It's None in default, which means using the standard Tokenizer of PySpark. You can use one of ["zh.sp.model", "code.sp.model"] we provided, or you can set it to the path to your own sentencepiece model

• text_key -- the field key name to hold texts to be classified. It's "text" in default

返回: