Vision Transformers Explained: From ViT to DINOv2 — A Practitioner's Guide

For a decade, the Convolutional Neural Network (CNN) was the undisputed king of computer vision. Convolutions are brilliant because they encode inductive biases perfectly suited for images: translation invariance (a cat is a cat anywhere) and local neighbourhood connectivity (pixels close to each other are related).

But in 2020, Dosovitskiy et al. published 'An Image is Worth 16x16 Words'. They stripped away these inductive biases entirely, treating an image exactly like a sequence of text tokens, and trained a standard Transformer encoder on it. It scaled better, learned more general representations, and outperformed ResNets on ImageNet.

We will cover four milestones in the ViT evolution: the original ViT (Dosovitskiy et al., 2020), DeiT (Touvron et al., 2021) which made ViTs data-efficient, Swin Transformer (Liu et al., 2021) which introduced hierarchical structure, and DINOv2 (Oquab et al., 2024) — Meta's self-supervised powerhouse that may be the most useful foundation model for CV today.

The core insight of ViT is brutally simple: split an image into fixed-size patches, flatten them into vectors, and feed them to a standard transformer encoder — the same architecture used for text.

The key difference from CNNs: a ViT has global receptive field from the very first layer. A CNN builds receptive field gradually through stacked convolutions. This is why ViTs excel on tasks requiring long-range spatial reasoning (e.g., understanding that a cat's tail 200px away belongs to the same instance as its head).

The ViT ecosystem has exploded. Here are the four models every CV practitioner should know, ranked by practical impact.

1. ViT (2020): the proof of concept

Dosovitskiy et al. proved that a pure transformer — with zero convolutions — could match EfficientNet on ImageNet when pre-trained on enough data. Impact: enormous for research. Practical use: limited unless you have Google-scale data.

2. DeiT (2021): data-efficient training

Touvron et al. (Facebook AI) showed that with the right training recipe — aggressive augmentation, regularisation, and a knowledge distillation token — ViTs can be trained competitively on ImageNet-1K alone. DeiT-B achieves 81.8% top-1 accuracy on ImageNet without external data.

3. Swin Transformer (2021): hierarchical + efficient

Liu et al. (Microsoft Research) replaced global self-attention with shifted window (Swin) attention — computing attention within local windows and shifting them to enable cross-window information flow. This produces hierarchical feature maps, making Swin a drop-in replacement for ResNet in detection and segmentation frameworks like Mask R-CNN and UPerNet.

4. DINOv2 (2024): self-supervised everything

Oquab et al. (Meta AI) trained ViT models on 142M curated images using a combination of self-supervised objectives (self-distillation + masked image modelling). The result: DINOv2 produces features that are useful for everything — classification, segmentation, depth estimation, retrieval — without any task-specific fine-tuning. Frozen DINOv2 features + a linear probe often beat fully fine-tuned supervised models.

Despite the hype, CNNs are not dead. Here is an honest decision framework based on practical experience.

On edge devices (mobile phones, embedded systems, IoT), CNNs still dominate. MobileNetV3 and EfficientNet-Lite run at 30+ FPS on a smartphone CPU. ViTs require dedicated NPU/GPU acceleration and even then lag behind in latency-per-accuracy on small models.

Here is the recipe I use for fine-tuning DINOv2 on custom datasets. It works for classification, segmentation, and retrieval tasks with minimal modification.

1import torch
2import torch.nn as nn
3from torchvision import transforms
4
5# Load DINOv2 ViT-Base from torch.hub
6backbone = torch.hub.load('facebookresearch/dinov2', 'dinov2_vitb14')
7
8# Freeze the backbone — train only the head
9for param in backbone.parameters():
10    param.requires_grad = False
11
12# Add classification head
13class DINOv2Classifier(nn.Module):
14    def __init__(self, backbone, num_classes: int):
15        super().__init__()
16        self.backbone = backbone
17        self.head = nn.Sequential(
18            nn.LayerNorm(768),
19            nn.Linear(768, 256),
20            nn.GELU(),
21            nn.Dropout(0.1),
22            nn.Linear(256, num_classes),
23        )
24
25    def forward(self, x):
26        with torch.no_grad():
27            features = self.backbone(x)  # [CLS] token
28        return self.head(features)
29
30model = DINOv2Classifier(backbone, num_classes=10)
31
32# Training config
33optimizer = torch.optim.AdamW(
34    model.head.parameters(), lr=1e-3, weight_decay=0.05
35)
36scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(
37    optimizer, T_max=30
38)

• ✓Freeze the backbone for small datasets (< 10K images). Only train the head.
• ✓Use AdamW with weight_decay=0.05 and lr=1e-3 for the head.
• ✓Cosine annealing schedule for 30 epochs is usually sufficient.
• ✓Minimal augmentation needed — DINOv2 features are already robust to transformations.
• ✓Input resolution: DINOv2 ViT-B/14 expects 518×518 by default. You can use 224×224 with interpolated position embeddings for 2× speed-up with ~1% accuracy loss.

I have used ViTs in several projects. Here are honest benchmarks from production-adjacent work — not cherry-picked paper results.

The pattern is clear: DINOv2 frozen features are extraordinarily good for classification and retrieval. For dense prediction (segmentation, depth), you need a proper decoder head, but the backbone quality still gives you a significant advantage over training from scratch.

“Foundation models like DINOv2 represent a genuine paradigm shift. The question is no longer 'how do I train a good backbone?' but 'how do I build the best head for my task?'”

Vision Transformers are no longer experimental — they are the default choice for most computer vision tasks in 2025. But knowing which ViT to use and how to deploy it efficiently is where the real skill lies.

• ✓DINOv2 is the best starting point for most CV tasks. Frozen features + lightweight head = fast iteration.
• ✓Swin Transformer is the best drop-in replacement for ResNet in detection/segmentation frameworks.
• ✓CNNs still win on edge devices, tiny datasets, and extreme latency requirements.
• ✓Don't train ViTs from scratch unless you have 10M+ images. Use pre-trained weights and fine-tune.
• ✓Self-supervised pre-training (DINO, MAE, I-JEPA) is the future — it removes the data labelling bottleneck.