Semantic and Instance Segmentation of Aerial Drone Imagery

A custom CNN model inspired by ResNet-18 that can perform Semantics on footage obtained from drones.

Github Link to the project - Link

Accurate Semantic and Instance segmentation is critical for autonomous drone navigation, especially when maneuvering through obstacles like racing windows. Deep Learning models such as a simple U-Net can be trained for executing segmentation tasks. Further, models like Mask R-CNNs can be used for further tasks like Object detection and Instance Segmentation.

Using UNet and Connected Component Analysis, Semantic and Instance Segmentation were performed on drone racing windows.

Project Overview

This project explores Semantic and Instance segmentation for drone perception tasks. It follows three key stages:

  1. Dataset Generation:
    • Use Blender to create images with various lighting, backgrounds, and occlusions.
    • Generate segmentation masks for training.
  2. Semantic Segmentation:
    • Implement U-Net with a MobileNet encoder for object segmentation.
  3. Instance Segmentation:
    • Apply connected component analysis to distinguish multiple objects.

Dataset Generation

Since manually collecting data is impractical, we generated images using Blender with domain randomization to create realistic training data.

  • Different Object Orientations
  • Lighting & Background Variations
  • Occlusion Handling

Each generated image includes a corresponding ground truth segmentation mask for training.

Image of a scene generated along with its Ground truth segmentation mask, both of which are generated through Blender.

Data Augmentation

To improve generalization, data augmentation techniques were applied:

Different Augmentations done to the generated data including Brightness shift, Camera Angle tilt, Color Jitter, etc.

Augmentations were implemented using PyTorch’s torchvision.transforms.

Semantic Segmentation Model

We trained a U-Net-based model with a MobileNet encoder for semantic segmentation.

Architecture Overview

  • Encoder: Uses a ResNet-like structure with convolutional layers and skip connections.
  • Decoder: Upsamples features using transposed convolutions.
  • Final Layer: Produces a binary segmentation mask.
U-Net architecture used for semantic segmentation.

Loss Function

The Binary Cross-Entropy (BCE) Loss was used for pixel-wise classification:

\[L_{BCE} = - \frac{1}{N} \sum_{i=1}^{N} [ y_i \log(p_i) + (1 - y_i) \log(1 - p_i) ]\]

where:

  • (y_i) is the ground truth label (0 or 1).
  • (p_i) is the predicted probability.

Instance Segmentation

Instance segmentation aims to distinguish multiple overlapping objects. We used the Connected Components Algorithm to assign unique labels to each object.

Algorithm Steps

  1. Input: Binary segmentation mask.
  2. Identify Connected Regions: Assign unique labels to each cluster.
  3. Use 4-connectivity or 8-connectivity to group pixels into objects.
Results of Semantic and Instance Segmentation.

Experiments & Results

Training Hyperparameters

| Hyperparameter | Value | |—————|——| | Epochs | 100 | | Batch Size | 32 | | Total Images | 40,000 | | Learning Rate | 1e-4 | | Optimizer | ADAM |

Loss curve showing model convergence during training.

Failure Cases

  1. Tilted Windows
    • The model struggled to segment windows when heavily tilted, likely due to a lack of diverse training data.
  2. Small Windows on Dark Backgrounds
    • Objects with low contrast were harder to segment accurately.
  3. Windows close to each other
    • Watershed segmentation fails when there is a significant overlap between the windows and it identifies it as a single entity.
Different Limiting cases of our approach.