dataquality.utils.semantic_segmentation package#

Submodules#

dataquality.utils.semantic_segmentation.constants module#

dataquality.utils.semantic_segmentation.errors module#

resize_maps(mislabeled_pixel_map, height, width)#

Resize the lm/dep map to the correct size (size of the mask) for interpolate need to be in format bs, c, h, w and right now we only have bs, h, w this results in mislabeled_pixel_map.shape = (1, 1, h, w)

Return type:

ndarray

background_accuracy(img_mask, polygon_mask)#

Calculates the amount of background predicted on a polygon calculated as (number of background pixels) / (number of pixels in polygon)

Parameters:

  • img_mask (ndarray) – mask of image with class (pred or gold) shape = (height, width)

  • polygon_mask (ndarray) – mask of image with only the polygon (pred or gold) shape = (height, width)

Return type:

float

returns: accuracy of the predictions

add_background_error_to_polygon(mask, polygon_img, polygon, polygon_type)#

Adds background error to a polygon

Parameters:

  • mask (np.ndarray) – mask of the image either gold or pred

  • polygon_img (np.ndarray) – img with polygon drawn on it

  • polygon (Polygon) – actual polygon object

  • polygon_type (str) – polygon type either gold or pred

Return type:

None

add_dep_to_polygon(dep_map, polygon_img, polygon)#

Adds dep of the polygon to a specific polygon

Parameters:

  • dep_map (np.ndarray) – dep heatmap with correct values

  • polygon_img (np.ndarray) – polygon drawn onto an image

  • polygon (Polygon) – polygon object

Return type:

None

add_area_to_polygon(polygon_img, polygon)#

Adds area of the polygon to a specific polygon

Parameters:

  • polygon_img (np.ndarray) – polygon drawn onto an image

  • polygon (Polygon) – polygon object

Return type:

None

calculate_classification_error(candidate_mask, comparison_mask, correct_class, number_classes)#

Calculates the accuracy of one ground truth polygon accuracy = (number of correct pixels) / (number of pixels in polygon) as well as the class with the most incorrect pixels in that respective polyon and the proportion of pixels in the polygon that are that class

Parameters:

  • candidate_mask (ndarray) – mask we are using to calulcate the accuracy ie. if we are computing accuracy for gold polygons this is the pred mask and vice versa

  • comparison_mask (ndarray) – mask we are using to compare the candidate mask to ie. if we are computing accuracy for gold polygons this is the gold mask and vice versa

  • correct_class (int) – the correct class of the polygon

  • number_classes (int) – number of classes

Return type:

ClassificationErrorData

returns: pixel accuracy of the predictions

add_class_errors_to_polygon(mask, polygon_img, polygon, polygon_type, number_classes)#

Adds class error to a polygon

Parameters:

  • mask (np.ndarray) – mask of the image either gold or pred

  • polygon_img (np.ndarray) – img with polygon drawn on it

  • polygon (Polygon) – actual polygon object

  • polygon_type (str) – polygon type either gold or pred

  • number_classes (int) – number of classes in the mask

Return type:

None

calculate_lm_pct(mislabeled_pixel_map, polygon_img)#

Calculates the percentage of mislabeled pixels in a polygon

Parameters:

  • mislabeled_pixel_map (torch.Tensor) – map of bs, h, w of mislabled pixels with value 1 if LM, 0 otherwise

  • polygon_img (np.ndarray) – np array of the polygon drawn onto an image

Returns:

percentage of mislabeled pixels in a polygon

Return type:

float

add_lm_to_polygon(mislabeled_pixels, polygon_img, polygon)#

Adds lm of the polygon to a specific polygon

Parameters:

  • lm_map (np.ndarray) – lm heatmap with correct values

  • polygon_img (np.ndarray) – polygon drawn onto an image

  • polygon (Polygon) – polygon object

Return type:

None

add_all_errors_and_metrics_for_image(mask, polygons, number_classes, polygon_type, dep_heatmap, mislabeled_pixels)#

Calculates and attaches the error types to each polygon

Parameters:

  • mask (np.ndarray) – mask of the image

  • polygons (List[Polygon]) – list of all polygons for an image

  • number_classes (int) – number of classes in the mask

  • polygon_type (PolygonType) – whether the polygons are gold or pred

  • dep_heatmap (np.ndarray) – heatmap of the depth

  • mislabled_pixels (np.ndarray) – map of h, w of mislabled pixels

Return type:

None

add_errors_and_metrics_to_polygons_batch(masks, polygons, number_classes, polygon_type, dep_heatmaps, mislabeled_pixels, height, width)#

Calculates and attaches the error types to each polygon for a whole batch

Parameters:

  • mask (np.ndarray) – masks of the images

  • polygons (List[Polygon]) – list of all polygons for all images

  • number_classes (int) – number of classes in the mask

  • polygon_type (PolygonType) – whether the polygons are gold or pred

  • dep_heatmaps (torch.Tensor) – dep heatmaps for each image

  • mislabeled_pixels (torch.Tensor) – map of bs, h, w of mislabled pixels

  • heights (List[int]) – height of each image

  • widths (List[int]) – width of each image

Return type:

None

dataquality.utils.semantic_segmentation.lm module#

calculate_lm_for_batch(self_confidence, confident_count, class_counts, gold, number_classes, probs)#

Caluclate likely mislabeled for a batch and return in image form

Parameters:

  • self_confidence (torch.Tensor) – confidence of each sample at gold (bs, h, w)

  • confident_count (torch.Tensor) – matrix of (classes, classes) of counts of confident predictions for each class

  • class_counts (torch.Tensor) – number of samples for each class (classes,)

  • gold (torch.Tensor) – ground truth labels for each sample (bs, h, w)

  • number_of_samples (int) – number_clases

  • probs (torch.Tensor) – probabilities for each sample (bs, h, w, classes)

Returns:

bs, h, w matrix of 1s and 0s where 1s are likely mislabeled

Return type:

torch.Tensor

get_mislabeled_by_noise(confident_joint, probs, gold)#

Gets the most likely mislabeled samples per given GT and actual GT pair using

the confidence_joint. See the CL paper for more information, but using the confidence joint we can get an estimated probability for the percentage of samples given the i GT label but actually belonging to the j GT label.

We extrapolate from the probability a number of samples that are most likely to be mislabeled for each cell in the CJ matrix and choose that many of the highest margin samples

(highest margin between predicted to be class j but given class i).

Parameters:

  • confident_joint (torch.Tensor) – confidence joint matrix

  • probs (torch.Tensor) – bs, h, w, classes matrix of probabilities

  • gold (torch.Tensor) – bs h, w matrix of ground truth labels

Returns:

bs, h, w matrix of 1s and 0s where 1s are likely mislabeled

Return type:

torch.Tensor

calculate_mislabled_by_noise(confident_joint, probs, given_gold_idx, num_samples)#

Calculated mislabled for a given gold index

Parameters:

  • confident_joint (torch.Tensor) – confidence joint matrix

  • probs (torch.Tensor) – matrix of probabilities for each sample where the gold label is given_gold_idx

  • given_gold_idx (torch.Tensor) – the gold label we are looking at

  • num_samples (int) – number of samples in probs

Returns:

list of sample ids that are most likely mislabeled

Return type:

List[int]

get_mislabeled_by_class_confidence(self_confidence, confidence_joint, gold, num_classes)#

Gets the most likely mislabeled samples per class using the confidence_joint

See the CL paper for more information, but using the confidence joint we can get an estimated probability for the percentage of a class that’s likely mislabeled. We extrapolate from the probaiblity a number of samples that are most likely to be mislabeled for each class and choose that many of the lowest self-confidence samples.

(1) get self confidence for all samples, (2) sort ascending across all, (3) get number_of_samples_in_this_class_likely_mislabeled per class and (4) take the top number_of_samples_in_this_class_likely_mislabeled per class with the lowest self confidence

Parameters:

  • self_confidence (Tensor) – self confidence for all samples (bs, h, w)

  • confidence_joint (Tensor) – confidence joint for all samples (num_classes, num_classes)

  • gold (Tensor) – ground truth labels for all samples (bs, h, w)

Returns:

mislabeled samples (bs, h, w)

Return type:

torch.Tensor

calculate_self_confidence(probs, gold)#

Gets the self confidence for each sample meaning the confidence in a prediction of its given GT label

Parameters:

  • probs (torch.Tensor) – probability mask for each sample

  • gold (torch.Tensor) – ground truth label for each sample

Returns:

self confidence for each sample in original dimensions

Return type:

torch.Tensor

fill_confident_counts(probs, gold, given_class, per_class_threshold, confident_counts)#
Helper function that fills the confident counts

based on how many values are above the threshold for a given class and are labeled a given class

ex. if label is a but above threshold for b increment confident_counts[a, b] by 1

Read the confidence count as: of all of those with above the threshold confidence in class a this is how many were labeled class b

Parameters:

  • probs (torch.Tensor) – probability mask for a give class

  • gold (torch.Tensor) – label mask for each sample

  • given_class (int) – the class we are looking at

  • per_class_threshold (np.ndarray) – per class threshold

  • confident_counts (np.ndarray) – matrix of confident counts

Returns:

confident counts matrix

Return type:

np.ndarray

normalize_confident_counts(confident_counts, class_counts)#

Normalize the confident_counts -> confidence joint.

Going from a count matrix to a valid probability matrix by making sure that the rows sum to the class distribution. In case this batch doesnt have all classes represented, fill with 0

class_counts is the count of number of samples per unique class in the data

Return type:

Tensor

calculate_self_confidence_threshold(probs, gold)#

Calculates the self confidence threshold for each class

Parameters:

  • probs (torch.Tensor) – bs, h, w, c probability mask

  • gold (torch.Tensor) – bs, h, w label mask

Returns:

(classes, ) self confidence threshold for each class

Return type:

torch.Tensor

dataquality.utils.semantic_segmentation.metrics module#

calculate_and_upload_dep(probs, gold_masks, image_ids, local_folder_path)#

Calculates the Data Error Potential (DEP) for each image in the batch

Uploads the heatmap to Minio as a png. Returns the dep_heatmaps

Return type:

Tensor

colorize_dep_heatmap(image, dep_mean)#

Recolors a grayscale image to a color image based on our dep mapping

Return type:

Image

calculate_dep_heatmaps(probs, gold_masks)#

Calculates the Data Error Potential (DEP) for each image in the batch

Parameters:

  • probs (Tensor) – np array of floats, size = (bs, height, width, n_classes)

  • gold_masks (Tensor) – np array of gold masks as ints, size = (bs, height, width)

Return type:

Tensor

Returns:

(bs, height, width)

write_dep_to_disk(dep_heatmaps, image_ids, local_folder_path)#

Writes dep to disk as a png locally

Parameters:

  • dep_heatmaps (torch.Tensor) – bs x height x width dep heatmaps

  • image_ids (List[int]) – image id for each image in the batch

  • local_folder_path (str) – folder path to store the dep heatmaps

Return type:

None

upload_dep_heatmaps(dep_heatmaps, image_ids, obj_prefix)#

Uploads dep heatmap to Minio for each image in the batch

Parameters:

dep_heatmaps (Tensor) – DEP heatmap for each image in the batch shape = (bs, height, width)

Return type:

None

dep_heatmap_to_img(dep_heatmap)#

Converts DEP heatmap to PIL Image We cast the heatmap to a 1-channel PIL Image object in grey scale and store it as a PNG file in Minio in order to compress the file size as much as possible.

To keep DEP heatmaps small, the maximum heatmap size is 64x64 pixels.

Parameters:

dep_heatmap (ndarray) – DEP heatmap for each image in the batch shape = (height, width)

Return type:

Image

Returns:

PIL Image object

compute_metric(pred_mask, gold_mask, metric_type, num_labels)#

Computes the SemSeg metric data for a single image

See ‘calculate_batch_metric’ for more details on the metrics

Parameters:

  • pred_mask (np.ndarray) – argmax of the probability predictions

  • gold_mask (np.ndarray) – ground truth mask

  • metric_type (SemSegMetricType) – type of metric to compute

  • num_labels (int) – number of classes

Returns:

metric data for the image

Return type:

SemSegMetricData

calculate_batch_metric(pred_masks, gold_masks, metric_type, num_labels)#

Function to calcuate semseg metrics for each image in a batch

SemSeg metrics can be one of:

  • Mean IoU

  • Boundary IoU

  • Dice coefficient

Dice score is the intersection over the total number of pixels per class. We take the ‘macro’ average where we weight each class’s dice score by the amount of pixels in that class, and then after computing each class’s dice score we average them together.

IoU is simply the intersection over union, where boundary IoU is the IoU computed on the boundary masks.

Parameters:

  • pred_masks (torch.Tensor) – argmax of predicted probabilities

  • gold_masks (torch.Tensor) – ground truth masks

  • metric_type (SemSegMetricType) – type of metric to compute

  • num_labels (int) – number of classes

Returns:

A metric data object for each image in the batch

Return type:

List[SemSegMetricData]

calculate_polygon_area(polygon, height, width)#

Calculates the area for a polygon :type polygon: Polygon :param polygon: polygon to calculate area for :type polygon: Polygon :type height: int :param height: :type height: int :type width: int :param width: :type width: int

Returns:

area of the polygon

Return type:

int

add_area_to_polygons(polygons, height, width)#

Adds the area of each polygon in an image to the obj

Parameters:

  • polygons (List[Polygon]) – list of each images polygons

  • height (int) –

  • width (int) –

Return type:

None

add_area_to_polygons_batch(polygon_batch, heights, widths)#

Calculates the area for every polygon in a btach :type polygon_batch: List[List[Polygon]] :param polygon_batch: list of each images polygons :type polygon_batch: List[List[Polygon]] :param size: shape to draw the polygons onto :type size: Tuple[int, int]

Return type:

None

dataquality.utils.semantic_segmentation.polygons module#

find_polygons_batch(pred_masks, gold_masks)#

Creates polygons for a given batch

NOTE: Background pixels do not get polygons

Parameters:

masks – Tensor of ground truth or predicted masks torch.Tensor of shape (batch_size, height, width)

Return type:

Tuple[List, List]

Returns (Tuple):

List of pred polygons for each image in the batch List of ground truth polygons for each image in the batch

build_polygons_image(mask, polygon_type)#

Returns a list of Polygons for the mask of a single image

Parameters:

mask (ndarray) – numpy array of shape (height, width) either gold or pred

Returns:

A list of polygons for the image

Return type:

List

A polygon is a list of CV2 contours, where each contour is a list of pixel coordinates that make up the boundary of a shape.

build_polygons_label(contours, hierarchy, label_idx, polygon_type)#

Builds the polygons given contours of a single label for one image

Parameters:

  • contours (Tuple[ndarray]) – a tuple of numpy arrays where each array is a contour

  • hierarchy (ndarray) – a numpy array of shape (num_contours, 4) where each row is a contour

  • label_idx (int) – the label index for all polygons in the given contours

Return type:

List[Polygon]

Returns:

a list of polygons where each polygon is a list of contours

draw_polygon(polygon, shape)#

Draws one polygon onto a blank image, assigning the polygon a label

drawContours takes in order an image, a list of contours, the index of the contour to draw (or -1 if drawing all), the color of the contour, and the thickness of the line to draw or -1 if you want to fill it in

Parameters:

  • polygon (Polygon) – Polygon object

  • shape (Tuple[int]) – Dimensions of returned image

Returns:

image with single polygon drawn on it

Return type:

np.ndarray

write_polygon_contours_to_disk(polygon, prefix, image_id)#

Writes polygons to disk in json format

Parameters:

  • polygon (Polygon) – A Polygon object

  • prefix (str) – prefix of the object name in storage “{proj_run_path}/{split_name_path}/contours”

Return type:

None

dataquality.utils.semantic_segmentation.utils module#

mask_to_boundary(mask, dilation_ratio=0.02)#

Convert binary mask to boundary mask. The boundary mask is a mask tracing the edges of the object in the mask. Therefore, the inside of the mask is now hollow which means our IoU calculation will only be on the ‘boundary’ of each polygon. The dilation ratio controls how large the tracing is around the polygon edges. The smaller the dilation ratio the smaller the boundary mask will be and thus likely decrease Boundary IoU.

This function creates an eroded mask which essentially shrinks all the polygons and then subtracts the eroded mask from the original mask to get the boundary mask.

Parameters:

  • uint8) (mask (numpy array,) – binary mask

  • (float) (dilation_ratio) – ratio to calculate dilation dilation = dilation_ratio * image_diagonal

Return type:

ndarray

Returns:

boundary mask (numpy array)

Module contents#