How to Annotate Images for Semantic Segmentation Using Deep Learning

Annotators use polygon tools or brush strokes to draw boundaries. This method has proved its worth in producing the most accurate masks. The method perfectly meets the requirements of small, high-stakes datasets for medical imaging, legal forensics, and projects where creating accurate boundaries is the primary challenge. It takes 30–60 minutes to annotate every complex image, and it is unable to scale up to production volumes.

Annotators have to confirm and correct the boundary suggestions generated by edge detection and Superpixel algorithms. Using this annotation method cuts labeling time by 40–60% with minimal accuracy loss when masking natural images like outdoor scenes, industrial inspection footage, etc., that have strong contrast boundaries. Though faster this method is judgment intensive. Also, annotators using this method require a different annotation skill set when it comes to reviewing.

In this approach, a pre-trained annotation model generates candidate masks. Now the annotators job is to review, accept, reject, or refine. This is the new default norm for production projects above 1,000 images. Leveraging the SAM2 integration helps in reducing the annotation time 8-12 minutes per image. It also eliminates fully manual work by 75–80%. Quality is nowhere compromised as the entire project is executed under supervision of a trained annotator.

Virtual environments are used to automatically generate pixel masks which are used for game engines, 3D simulators, and procedural rendering pipelines. With synthetic data annotation costs nearly zero, and the scalability is unlimited. The only challenge of working with synthetic data is the gap between simulation to real domain performance. Models trained using synthetic data perform differently when used for annotating real-world images.

Domain adaption techniques are applied to address this issue. This approach is significantly useful in bootstrapping pipelines or augmenting real labeled data. It does not come in handy as a standalone strategy.

Hybrid, the model-assisted annotation, is the best solution for most of the production pipelines. It is so because the approach has manual correction for edge cases like occluded objects, low-contrast boundaries, rare classes and many more. This is the approach the 6-step workflow below is built around.

Your training datasets should have common-case scenarios, as well as edge cases that the model will encounter when deployed for production.

Class distribution is the first critical thing to take care of. The class imbalance in real-world datasets, a situation where background pixels are more than foreground object pixels, is a prominent challenge. As we all know, segmentation model trained using imbalanced training dataset will more likely learn to predict the dominant class. It will make the model achieve misleadingly high pixel accuracy, but the model will fail on rare classes. Fixing this issue will need you to do over-sampling. It means collecting more images of rare classes, augmenting existing annotations, and applying weighted loss functions during training – if required.

Pixel-mask alignment is the second critical thing to take care of. The ground truth will get corrupted due to the spatial offset between an image and its corresponding segmentation mask. Using SLIC or Felzenszwalb, as edge detection and Superpixel segmentation techniques during preprocessing helps you verify and refine boundary alignment well before actual image annotation begins.

Using a model pre-trained on ImageNet is a smart move. You cannot train your models using random initialization on typical dataset sizes will do no good. It will only add training time without improving final performance.

First let’s check out the three architectures that dominate the production of semantic segmentation. Each one of them suits a different domain altogether:

• FCN: Fast inference speed, ideal for real-time, low-latency requirements.
• U-Net: Skip connections preserve, high boundary accuracy, standard for medical.
• DeepLab v3+: No downsampling resolution, multi-scale contextual features, best complex objects.

Selecting the LOSS function will directly impact model performance while using imbalanced datasets. Cross-entropy, known for treating all pixels equally, suffices the need when using balanced datasets. In case of datasets where background pixels dominate, Dice loss directly optimizes mask overlap. For medical datasets, Dice loss outshines cross-entropy on rare-structure classes.

For all classes, mIoU should be used to measure the average overlap between predicted masks and ground truth. So, track mIoU on a held-out validation set till the time training continues. Remember, a model that has achieved 85% overall pixel accuracy, may end up scoring less than 0.50 mIoU on rare classes. Use per-class IoU to identify such failures.

Use a bounding box prompt to generate a pixel mask. This is the fastest, and high quality result providing starting point for most of the annotation projects.

Prompt-based mask generation was introduced by Meta’s Segment Anything Model – SAM 2023. You give the model a prompt, a bounding box, or a rough explanation about an object; and it will provide you with a precise segmentation mask without task-specific fine-tuning. But SAM 2024 is a smarter version, as it extends the above arrangement to videos, while maintaining mask consistency across all frames. It is effectively applied in annotating autonomous driving footage, surgical videos, and drone imagery.

SAM generates class-agnostic masks, and that is one distinction that you should consider when designing a workflow. It segments objects accurately, however, does not name them. Annotators have to generate the mask in one step and assign the class label in second step; in case they are using SAM for semantic segmentation annotation. CVAT and Roboflow, both, implement this two-step workflow natively.

SAM2 assisted annotation is known to reach 94% of fully manual mIoU in 18% of the annotation time, in one of the independent benchmark testing. SAM2 has become the favorite model-assisted approach for production pipelines due to near-equivalent quality at a fraction of the labor.

Here are the three annotation modes that you can apply depending on your project requirements:

To ascertain model uncertainty, one can use Monte Carlo dropout or ensemble disagreement. Remember, regions with high uncertainty correlate strongly with ambiguous boundaries, occlusions, and rare-class instances. These are the regions that majorly impact the final mIoU.

Unlabeled images are typically 15-20% of the entire dataset. Using active learning to identify and annotate these subset images can majorly contribute to model improvement. It will help you achieve 90%+ of the mIoU of training on the complete dataset.

For those who don’t know, the active learning cycle is an iterative process. It is very simple.

• Start by training an initial model using a small, labeled speed set.
• Apply the model to the unlabeled pool.
• Rank unlabeled images by informativeness with help of an acquisition function.
• Now, annotate the top ranked images.
• Add them to the training set, retrain, and repeat.

Here are the three image sampling strategies with the key characteristics and benefits of each of them:

Now, let’s define the ratio of active learning value proposition. So, for example, you have an autonomous driving dataset of 50,000 images. Of which, uncertainty sampling selected 8,000 images to annotate. The resulting model attained 91% of mIoU of one trained using the entire dataset. It will also result in reduction of annotation budget by 84% and a max. of 9% quality trade-off.

A visually correct looking mask has the capacity to fail your model. Yes, you read it right. Deploy a metric-driven QA process which has pass/fail benchmarks defined before you start annotating images, and of course not after the first batch is complete.

The primary consistency metric for pixel-level annotation is Inter-annotator agreement – IAA. Use it to measure the average IoU between masks produced by individual annotators for the same image. IAA score less than 85% on boundary pixels is an indicator that there is a problem with your annotation guidelines. Mind it, that’s not an annotator skill issue. In order to fix it, revise the guidelines instead of retraining the image annotators.

Automated QA checks run before any masks enter the training pipeline:

Either the mIoU on the validation set drops below the project threshold, or any class has an F1 score below 0.80. These are the two conditions that trigger human review, and both of them are capable of identifying errors in systematic annotation that missed the automated check.

Imagine your QA process discovers a failure after your team has done annotating more than 20,000 images. It will cost you more to correct it across all images if compared to building and implementing a robust QA process before you start annotating images. This is the cost reality that most of the AI teams underestimate. Be proactive in defining IAA thresholds, deploy automated check criteria, and keep a close watch on triggers in the annotation guideline document. To be on the safer side, you can run a 200-image pilot batch through the full QA stack before you scale it up for all the images.

Final training will optimize your images for generalization, but that does not guarantee completeness. There more chances that a model that has memorized training masks will fail miserably on production data.

Your AI teams can use hyperparameter tuning to target three different variables:

• Learning rate schedule: cosine annealing typically outperforms fixed learning rates for segmentation tasks.
• Batch size – larger batches stabilize gradient estimates; smaller batches generalize better on limited data.
• Augmentation policy – random cropping, horizontal flipping, color jitter, and CutMix are standard for semantic segmentation.

In addition, here are two architectural additions that will help your AI teams to improve final model quality when it comes to boundary-critical image annotation tasks:

The final deployment evaluation checklist has three critical attributes:

• mIoU on the held-out test set
• Per-class IoU for rare and safety-critical classes
• Inference speed in milliseconds per frame against the production latency requirement.

The third attribute is where you get trapped in. For real-time autonomous driving, a segmentation model with 92% mIoU running at 200ms per frame will not help. Because here 30ms is the practical upper bound. Remember, latency is a constraint of first order. Evaluate it before you deploy – not afterwards.