From 2D Masks to 3D Meshes: Object Reconstruction with OccupancyNet on HoloLens 2

Imagine wearing a HoloLens 2 and walking into any room. You want the device to not only know where walls and floors are but to understand each individual object: its precise 3D shape, orientation, and surface, so that you can attach digital annotations, simulate physical interactions, or replace it with a virtual counterpart.

This is the 3D object reconstruction problem in AR, and it is surprisingly hard. A HoloLens 2 sees each object from a limited viewing angle, with significant depth noise, and often only partially. Classical approaches — volumetric fusion, photogrammetry — require dense, calibrated multi-view capture incompatible with a freely-moving headset.

In this post I walk through OccupancyNet (Mescheder et al., CVPR 2019), explain why it fits our constraints, show the full reconstruction pipeline, and share honest results including the cases where it fails.

To appreciate why occupancy networks are interesting, it helps to understand what came before them.

The key advantage of OccupancyNet over voxel grids is that it does not discretise space upfront: you can extract a mesh at any resolution by running Marching Cubes at whatever grid density you want.

Why not NeRF?

NeRF requires dense multi-view capture (100+ calibrated images) and takes hours to train per scene. Our use case provides at most 5–10 partial views of a single object from a moving headset. OccupancyNet with a learned shape prior can reconstruct from a single partial point cloud by drawing on the latent space it learned during training on ShapeNet.

OccupancyNet defines a binary classifier f_θ(p, z) → [0,1] where p is a 3D query point and z is a latent code encoding the shape. Training minimises binary cross-entropy over randomly sampled surface and free-space points.

Encoder: PointNet for partial observations

The input is a partial point cloud from HoloLens 2 depth data. A PointNet encoder applies a shared MLP to each point independently, then max-pools over all points to produce a permutation-invariant global feature vector z ∈ ℝ^256.

1import torch
2import torch.nn as nn
3
4class PointNetEncoder(nn.Module):
5    """Encode a partial point cloud into a global shape latent code."""
6    def __init__(self, latent_dim: int = 256):
7        super().__init__()
8        self.mlp = nn.Sequential(
9            nn.Linear(3, 64),   nn.ReLU(),
10            nn.Linear(64, 128), nn.ReLU(),
11            nn.Linear(128, latent_dim),
12        )
13
14    def forward(self, pts: torch.Tensor) -> torch.Tensor:
15        features = self.mlp(pts)        # (B, N, latent_dim)
16        z, _ = features.max(dim=1)      # global max-pool
17        return z

Decoder: conditioned occupancy MLP

The decoder takes query point p ∈ ℝ³ and shape code z ∈ ℝ^256, outputting occupancy probability. We use Conditional Batch Normalisation (CBN) to inject the shape code at each layer — this improves reconstruction quality on thin structures.

Mesh extraction with Marching Cubes

Once trained, we extract meshes by evaluating the occupancy function on a dense 3D grid and running Marching Cubes at the 0.5 iso-surface. Resolution is a trade-off: 32³ is fast (0.05s) but blocky; 128³ gives smooth surfaces but takes ~2s per object.

We pre-trained OccupancyNet on ShapeNet Core55 — 51,300 3D models across 55 categories. Pre-training provides a rich shape prior: the encoder learns a latent space where similar shapes cluster, and the decoder learns smooth implicit surfaces.

Fine-tuning on our custom 214-object dataset improved Chamfer Distance by 34% over zero-shot inference. The biggest gains were on non-ShapeNet categories like 'lab equipment' and 'wiring panels'.

We evaluated on 42 held-out objects using Chamfer Distance (CD), F-Score at τ=0.01, and volumetric IoU.

Where the model fails

• →Very thin structures (cables, pens, chair legs under 5mm): Marching Cubes at 128³ cannot resolve sub-voxel details.
• →Symmetric objects with extreme partial occlusion: when less than 15% of the surface is observed, the latent code is ambiguous.
• →Transparent objects (glass, bottles): the depth sensor returns no depth returns for transparent surfaces.
• →Objects outside ShapeNet categories: near-total failure on fume hoods and wall-mounted projector screens.

The output of OccupancyNet is a standard .obj mesh. Deploying it into a running HoloLens 2 mixed reality session requires several engineering steps beyond the ML pipeline.

• ✓OccupancyNet is the right tool for shape completion from sparse partial observations.
• ✓Pre-training on ShapeNet is essential. Training from scratch on 200 objects produces degenerate results.
• ✓128³ resolution is the practical sweet spot: good surface quality, 2s per object, 8GB VRAM.
• ✓Transparent and very thin objects are unresolved problems requiring different sensor modalities.
• ✓The engineering around the ML model — coordinate alignment, mesh post-processing, MRTK streaming — takes more time than the model itself.