data_juicer.ops.filter.image_subplot_filter module

class data_juicer.ops.filter.image_subplot_filter.ImageSubplotFilter(*args, **kwargs)

Bases: Filter

Filter to detect and remove samples with images containing subplots.

This filter uses Hough Line Transform to detect straight lines in images, which is particularly effective for detecting grid-like subplot layouts with perfectly straight edges.

The algorithm works by: 1. Converting images to grayscale and applying edge detection 2. Using Hough Line Transform to detect straight lines 3. Classifying lines as horizontal or vertical based on angle 4. Counting lines that meet length and angle requirements 5. Calculating confidence based on line counts and distribution

__init__(min_horizontal_lines: int = 3, min_vertical_lines: int = 3, min_confidence: float = 0.5, any_or_all: str = 'any', canny_threshold1: int = 70, canny_threshold2: int = 190, hough_threshold: int = 110, min_line_length: int = 110, max_line_gap: int = 18, angle_tolerance: float = 4.0, *args, **kwargs)

Initialization method.

Parameters:

• min_horizontal_lines – Minimum number of horizontal lines to consider an image as containing subplots.

• min_vertical_lines – Minimum number of vertical lines to consider an image as containing subplots.

• min_confidence – Minimum confidence score for filtering. Images with subplot confidence above this threshold will be considered as containing subplots.

• any_or_all – Strategy for multi-image samples. ‘any’ filters the sample if any image contains subplots. ‘all’ filters the sample only if all images contain subplots.

• canny_threshold1 – First threshold for Canny edge detector.

• canny_threshold2 – Second threshold for Canny edge detector.

• hough_threshold – Accumulator threshold for Hough transform.

• min_line_length – Minimum line length to be detected.

• max_line_gap – Maximum gap between line segments to be treated as a single line.

• angle_tolerance – Tolerance in degrees for classifying lines as horizontal/vertical.

• args – Extra args.

• kwargs – Extra args.

compute_stats_single(sample, context=False)

Detect subplots in images and compute statistics.

Parameters:

• sample – Single sample.

• context – Whether to store context information.

Returns:

Sample with computed statistics.

process_single(sample)

Process single sample to determine if it should be filtered.

Parameters:

sample – Single sample.

Returns:

True for keeping, False for filtering.