data_juicer.analysis

class data_juicer.analysis.ColumnWiseAnalysis(dataset, output_path, overall_result=None, save_stats_in_one_file=True)

Bases: object

Apply analysis on each column of stats respectively.

__init__(dataset, output_path, overall_result=None, save_stats_in_one_file=True)

Initialization method

Parameters:

• dataset – the dataset to be analyzed

• output_path – path to store the analysis results

• overall_result – optional precomputed overall stats result

• save_stats_in_one_file – whether save all analysis figures of all stats into one image file

analyze(show_percentiles=False, show=False, skip_export=False)

Apply analysis and draw the analysis figure for stats.

Parameters:

• show_percentiles – whether to show the percentile line in each sub-figure. If it’s true, there will be several red lines to indicate the quantiles of the stats distributions

• show – whether to show in a single window after drawing

• skip_export – whether save the results into disk

Returns:

draw_box(ax, data, save_path, percentiles=None, show=False)

Draw the box plot for the data.

Parameters:

• ax – the axes to draw

• data – data to draw

• save_path – the path to save the box figure

• percentiles – the overall analysis result of the data including percentile information

• show – whether to show in a single window after drawing

Returns:

draw_hist(ax, data, save_path, percentiles=None, show=False)

Draw the histogram for the data.

Parameters:

• ax – the axes to draw

• data – data to draw

• save_path – the path to save the histogram figure

• percentiles – the overall analysis result of the data including percentile information

• show – whether to show in a single window after drawing

Returns:

draw_wordcloud(ax, data, save_path, show=False)

class data_juicer.analysis.CorrelationAnalysis(dataset, output_path)

Bases: object

Analyze the correlations among different stats. Only for numerical stats.

__init__(dataset, output_path)

Initialization method.

Parameters:

• dataset – the dataset to be analyzed

• output_path – path to store the analysis results

analyze(method='pearson', show=False, skip_export=False)

class data_juicer.analysis.DiversityAnalysis(dataset, output_path, lang_or_model='en')

Bases: object

Apply diversity analysis for each sample and get an overall analysis result.

__init__(dataset, output_path, lang_or_model='en')

Initialization method :param dataset: the dataset to be analyzed :param output_path: path to store the analysis results :param lang_or_model: the diversity model or a specific language used to load the diversity model.

analyze(lang_or_model=None, column_name='text', postproc_func=<function get_diversity>, **postproc_kwarg)

Apply diversity analysis on the whole dataset.

Parameters:

• lang_or_model – the diversity model or a specific language used to load the diversity model

• column_name – the name of column to be analyzed

• postproc_func – function to analyze diversity. In default, it’s function get_diversity

• postproc_kwarg – arguments of the postproc_func

Returns:

compute(lang_or_model=None, column_name='text')

Apply lexical tree analysis on each sample.

Parameters:

• lang_or_model – the diversity model or a specific language used to load the diversity model

• column_name – the name of column to be analyzed

Returns:

the analysis result.

class data_juicer.analysis.OverallAnalysis(dataset, output_path)

Bases: object

Apply analysis on the overall stats, including mean, std, quantiles, etc.

__init__(dataset, output_path)

Initialization method.

Parameters:

• dataset – the dataset to be analyzed

• output_path – path to store the analysis results.

analyze(percentiles=[], num_proc=1, skip_export=False)

Apply overall analysis on the whole dataset based on the describe method of pandas.

Parameters:

• percentiles – percentiles to analyze

• num_proc – number of processes to analyze the dataset

• skip_export – whether export the results to disk

Returns:

the overall analysis result.

refine_single_column(col)