Instance Segmentation at Scale: YOLO, Mask R-CNN & SAM on HoloLens 2 Data

Mixed reality devices like the Microsoft HoloLens 2 open a fascinating frontier: they can perceive the physical world through depth sensors and RGB cameras, yet they need to understand that world — not just see it. For our research at LIARA lab (Laboratoire d'Intelligence Ambiante pour la Reconnaissance d'Activités, UQAC), the key question was: can we automatically segment every object in a room, reconstruct it in 3D, and overlay digital information on top in real time?

The first critical step is instance segmentation — distinguishing not just what is in the scene (semantic segmentation) but which individual instance of each class each pixel belongs to. This is harder, computationally heavier, and far more useful for AR overlays where each physical object needs its own digital twin.

We evaluated three leading architectures: YOLOv8-seg (the fastest), Mask R-CNN with a ResNet-50-FPN backbone (the established baseline), and SAM (Segment Anything Model by Meta, the most recent zero/few-shot approach). Each has a radically different design philosophy, and the results — spoiler: SAM does not automatically win — were genuinely surprising.

No public dataset captured with a HoloLens 2 existed at the scale we needed. We collected 1,400+ RGB frames across 8 indoor scenes (office, kitchen, corridor, lab benches…), then annotated 200+ object instances using polygon masks in CVAT. The annotation pipeline was the most labour-intensive part — roughly 3 weeks of part-time work for two people.

The dataset exhibits three challenges not present in standard benchmarks like COCO: (1) near-infrared sensor noise from the depth camera bleeding into colour channels, (2) motion blur from users moving their heads naturally while wearing the device, and (3) extreme occlusion since the device captures from a first-person perspective where hands and arms frequently occlude objects.

The train/val/test split was 70/15/15 stratified by scene, not by frame — meaning the test set contains entirely unseen environments. This makes the benchmark more realistic and harder than random splits would produce.

Before comparing numbers, it is worth understanding why the three architectures are built so differently — because the design choices directly explain the benchmark results.

YOLOv8-seg: single-stage speed

YOLOv8 (Ultralytics, 2023) extends the YOLO detection family with a lightweight segmentation head. A CSPDarknet backbone extracts features, a Path Aggregation Network (PANNet) fuses multi-scale representations, and a prototype-based mask head predicts instance masks in a single forward pass. The model has ~11M parameters in the -n variant and can run at 80+ FPS on a mid-range GPU.

1from ultralytics import YOLO
2
3# Load YOLOv8n-seg pretrained on COCO
4model = YOLO("yolov8n-seg.pt")
5
6results = model.train(
7    data="hololens2.yaml",
8    epochs=100,
9    imgsz=640,
10    batch=16,
11    lr0=1e-3,
12    lrf=1e-2,
13    mosaic=0.8,
14    mixup=0.1,
15    copy_paste=0.3,
16    degrees=10.0,
17    translate=0.1,
18    scale=0.5,
19    cls=0.5,
20    device="cuda:0",
21    workers=8,
22    project="runs/segment",
23    name="yolov8n_holo",
24)

Mask R-CNN: two-stage robustness

Mask R-CNN (He et al., 2017) remains the gold standard for instance segmentation in constrained settings. It adds a small FCN mask branch in parallel to the box-regression head of Faster R-CNN. The Region Proposal Network generates candidate boxes, RoIAlign extracts fixed-size feature maps, and the mask head generates a 28×28 binary mask per proposal. This two-stage approach sacrifices speed but gains significant robustness on small and occluded objects.

SAM: zero-shot foundation model

Segment Anything Model (Kirillov et al., Meta AI, 2023) is a prompt-based foundation model trained on 1.1 billion masks. SAM does not learn class-specific detectors — instead it accepts a point, a box, or a text prompt and returns a mask. For our use case, we used the automatic mask generator mode. SAM ViT-H has 636M parameters and is orders of magnitude larger than the other two models.

Training on a small, domain-shifted dataset demands careful regularisation. We applied a multi-stage augmentation pipeline designed specifically for the HoloLens 2 failure modes.

For Mask R-CNN, we used a step LR schedule (warm-up for 1k iterations, decay at epoch 60 and 80), SGD with momentum=0.9, weight_decay=1e-4, and learning rate 0.005. Training for 100 epochs on a single NVIDIA T4 took approximately 4 hours.

All three models were evaluated on the held-out test split using mask AP at IoU=0.50:0.95 (COCO standard), mask AP50, inference time, and GPU memory.

The YOLOv8 superiority on mAP@50 is explained by its aggressive augmentation pipeline (mosaic, copy-paste) which dramatically improves recall on occluded objects. Mask R-CNN's RoIAlign gives it noticeably better mask boundary precision on thin objects like pens and cables.

The segmentation model is one component in a larger pipeline. The full system flow is: HoloLens 2 RGB stream → YOLO segmentation → depth-aligned bounding volume extraction → OccupancyNet 3D reconstruction → mesh export → HoloLens MRTK overlay.

• ✓Annotate more, train later. We underestimated annotation time. Going from 1,400 to 3,000 frames would likely have pushed mAP above 85%.
• ✓YOLOv8 copy-paste is transformative for scenes with heavy occlusion.
• ✓SAM is not a free lunch. It shines as a post-processing refinement tool rather than as a standalone detector.
• ✓Mixed precision training is non-optional on small GPU budgets — it halved memory usage with no accuracy penalty.
• ✓Validate on unseen scenes from day one. We initially validated on random frame splits and saw optimistically inflated numbers.

“The most dangerous metric is validation accuracy on in-distribution data. Always hold out entire scenes, sessions, or domains as your test set.”

Instance segmentation on custom, domain-shifted data is a solvable problem — but it requires attention to every link in the chain. YOLOv8 is the right choice for real-time AR applications today; Mask R-CNN remains relevant when boundary precision matters more than speed; SAM's role is evolving toward fine-tuned specialisation.

Next steps for this research: (1) expand the dataset to 5,000+ annotated frames, (2) explore SAM 2 fine-tuning on our domain, (3) benchmark YOLOv10 and RT-DETR as drop-in replacements, and (4) push for on-device inference on the HoloLens 2's integrated NPU.