Image Segmentation

Image segmentation is a computer vision task that involves partitioning an image into multiple segments or regions, where each segment corresponds to a specific object or part of an object. Unlike object detection, which outputs bounding boxes around objects, image segmentation aims to assign a class label to every pixel in the image, thereby providing a much more granular understanding of the image content and the exact shape of objects.

The goal is to simplify or change the representation of an image into something that is more meaningful and easier to analyze.

• Types of Image Segmentation

  ▶
  • Semantic Segmentation: Assigns a class label (e.g., "car", "person", "sky", "road") to each pixel in the image. All instances of the same object class share the same label (e.g., all cars are colored red in the segmentation map). It does not distinguish between different instances of the same class.
  • Instance Segmentation: Goes a step further than semantic segmentation by not only labeling each pixel with a class but also distinguishing between different instances of the same class. For example, if there are three cars in an image, instance segmentation would label each car pixel as "car" and also assign a unique instance ID to each of the three cars (e.g., car-1, car-2, car-3 will have different colors in the segmentation map).
  • Panoptic Segmentation: A combination of semantic and instance segmentation. It assigns a class label to every pixel in the image and, for objects that are instances ("things" like cars, people), it also assigns a unique instance ID. For amorphous regions ("stuff" like sky, road, grass), it only assigns semantic labels. Every pixel in the image is assigned exactly one semantic label and, if applicable, one instance ID.

• Fully Convolutional Networks (FCNs)

  ▶

  FCNs were a groundbreaking development for semantic segmentation. They adapt classification CNNs (like VGG, ResNet) by replacing fully connected layers with convolutional layers, allowing them to output a spatial segmentation map (heatmap) instead of a single class label.

  Key features:

  • End-to-end pixel-to-pixel prediction: Can take an image of arbitrary size and produce an output of corresponding spatial dimensions.
  • Upsampling/Deconvolution: To recover the spatial resolution lost during downsampling in the encoder part of the network, FCNs use upsampling layers (e.g., bilinear interpolation) or deconvolution layers (transposed convolutions) in the decoder part.
  • Skip Connections: Combine coarse, semantic information from deeper layers with fine-grained, spatial information from shallower layers to produce more precise segmentation boundaries. This was a key innovation.

• U-Net Architecture

  ▶

  U-Net is a very popular FCN architecture, initially designed for biomedical image segmentation but now widely used in various domains. It features a symmetric U-shaped encoder-decoder structure.

  Key features:

  • Encoder (Contracting Path): Consists of repeated blocks of convolutions and max pooling to capture context and extract features at different scales.
  • Decoder (Expanding Path): Symmetrically expands the feature maps using up-convolutions (transposed convolutions) to gradually recover spatial resolution.
  • Skip Connections: Extensive use of skip connections that concatenate feature maps from the encoder path with the corresponding feature maps in the decoder path. This allows the decoder to leverage high-resolution features from the encoder, leading to more precise localization and better segmentation of details.

  U-Net and its variants (e.g., U-Net++, Attention U-Net) are known for their good performance, especially with limited training data.

• DeepLab Family (ASPP, CRFs)

  ▶

  The DeepLab family of models (DeepLabv1, v2, v3, v3+) introduced several important techniques for semantic segmentation:

  • Atrous (Dilated) Convolutions: Allow for a larger receptive field to capture multi-scale context without increasing the number of parameters or significantly reducing spatial resolution. This helps in segmenting objects of different sizes.
  • Atrous Spatial Pyramid Pooling (ASPP): Probes an incoming convolutional feature layer with filters at multiple sampling rates and effective fields-of-view, thus capturing objects as well as image context at multiple scales. The resulting features are then fused.
  • Conditional Random Fields (CRFs): Used as a post-processing step (especially in earlier versions) to refine segmentation maps by encouraging smoothness and adherence to image boundaries. Fully connected CRFs can capture long-range dependencies. Later versions aimed to incorporate this refinement into the network itself.

• Mask R-CNN (for Instance Segmentation)

  ▶

  Mask R-CNN is a widely used framework for instance segmentation. It extends the Faster R-CNN object detection model by adding a parallel branch for predicting segmentation masks for each Region of Interest (RoI).

  Key components:

  1. Backbone Network: Extracts features (e.g., ResNet, FPN).
  2. Region Proposal Network (RPN): Proposes candidate object bounding boxes.
  3. RoIAlign: A layer that accurately pools features for each RoI, addressing misalignment issues present in RoIPool. This is crucial for precise mask prediction.
  4. Parallel Heads: For each RoI, Mask R-CNN has three heads:
     • One for classifying the object.
     • One for regressing the bounding box coordinates.
     • One for predicting a binary segmentation mask (pixel-wise) within the RoI.

  Mask R-CNN is known for its high accuracy in instance segmentation tasks.

• Transformers for Segmentation (e.g., SegFormer, Mask2Former)

  ▶

  More recently, Transformer-based architectures have shown strong performance in image segmentation tasks, challenging traditional CNN-based approaches.

  • SegFormer: A simple and efficient Transformer-based framework that uses a hierarchical Transformer encoder to output multi-scale features and a lightweight MLP decoder to combine these features for segmentation. It avoids complex decoders and positional encodings used in some earlier Vision Transformers.
  • Mask2Former / Mask DINO: Unified frameworks that can handle semantic, instance, and panoptic segmentation using a mask classification paradigm. They typically employ a Transformer encoder and a Transformer decoder that processes a set of learnable queries (object queries or segment queries) to predict masks and class labels. These models often achieve state-of-the-art results across all three segmentation types.

• Loss Functions for Segmentation

  ▶

  Common loss functions used for training segmentation models include:

  • Pixel-wise Cross-Entropy Loss: Treats each pixel as an independent classification problem. Commonly used for semantic segmentation.
  • Dice Loss (Sørensen-Dice Coefficient Loss): Directly optimizes the Dice coefficient, which is a measure of overlap (similar to IoU). It is often more robust to class imbalance than cross-entropy. $$Dice = \frac{2 |X \cap Y|}{|X| + |Y|}$$, Loss = 1 - Dice.
  • Jaccard/IoU Loss: Directly optimizes the Intersection over Union. Loss = 1 - IoU.
  • Focal Loss: An adaptation of cross-entropy loss that down-weights the contribution of well-classified examples, allowing the model to focus on hard examples. Useful for handling class imbalance.
  • Combined Losses: Often, a combination of losses (e.g., Cross-Entropy + Dice Loss) is used to leverage the benefits of different formulations.

• Evaluation Metrics

  ▶

  Metrics for evaluating image segmentation models include:

  • Pixel Accuracy (PA): The percentage of pixels in the image that are correctly classified. $$PA = \frac{\sum_i TP_i}{\sum_i (TP_i + FP_i)}$$
  • Mean Pixel Accuracy (mPA): The average of pixel accuracies computed per class.
  • Intersection over Union (IoU) / Jaccard Index: Calculated per class as $$IoU_c = \frac{TP_c}{TP_c + FP_c + FN_c}$$.
  • Mean Intersection over Union (mIoU): The average of IoU values across all classes. This is the most common and important metric for semantic segmentation.
  • Dice Coefficient (F1 Score): $$Dice_c = \frac{2 TP_c}{2 TP_c + FP_c + FN_c}$$. Closely related to IoU.
  • For instance segmentation, metrics are often adapted from object detection, such as Average Precision (AP) based on mask IoU (e.g., AP at IoU=0.50, AP at IoU=0.75, and average AP over IoU thresholds).
  • For panoptic segmentation, Panoptic Quality (PQ) is used, which combines segmentation quality (SQ - average IoU of matched segments) and recognition quality (RQ - essentially an F1 score for detected segments).