DINO → DINOv2 → DINOv3

An interactive field guide

How can a model teach itself without class labels, and what changes across three generations?

Before this article: ViT · Semantic representations

What was already possible

This article covers the self-supervised visual DINO family. The object detector also named DINO is a different method family; its detection queries and objectives are outside this series.

What is learned without labels?

An image has no supplied class name in DINO pretraining. The model sees different crops of the same image and learns outputs that agree across those views. The backbone f produces image and patch features. A separate head h maps the image feature to K prototype scores. These scores define a prediction distribution, not a human class vocabulary and not a VQ codebook. No nearest-code lookup replaces the backbone feature, and no pixel decoder is trained by the DINO objective.

$$g_\theta(x)=h_\theta(f_\theta(x)),\qquad p^{(k)}=\frac{e^{g^{(k)}/\tau}}{\sum_j e^{g^{(j)}/\tau}}$$

What changes, and why

Read from left to right. Select a module to connect its role, tensor shape, and calculation. Snippets are teaching pseudocode. Sources provide the paper method or audited implementation.

1. What is learned without labels?view: H × W × 3 → feature: D → logits: K2. Two networks, different views, different updates2 teacher global views; 2 + L student views3. Centering and sharpening solve different failure modesteacher scores: B × K; center: K4. EMA is a parameter update, not a gradienttwo parameter sets with matching shapes

What is learned without labels?

An image has no supplied class name in DINO pretraining. The model sees different crops of the same image and learns outputs that agree across those views. The backbone f produces image and patch features. A separate head h maps the image feature to K prototype scores. These scores define a prediction distribution, not a human class vocabulary and not a VQ codebook. No nearest-code lookup replaces the backbone feature, and no pixel decoder is trained by the DINO objective.

# Teaching pseudocode; not a complete training implementation.
feature = backbone(view).cls
logits = projection_head(feature)
p = softmax(logits / temperature)

Training state: Training and inference roles are separated below.

$$g_\theta(x)=h_\theta(f_\theta(x)),\qquad p^{(k)}=\frac{e^{g^{(k)}/\tau}}{\sum_j e^{g^{(j)}/\tau}}$$

Step-by-step explanation: What is learned without labels?

Two networks, different views, different updates

The original DINO teacher receives two global crops. The student receives the global crops and smaller local crops. Their backbone and head architecture match, but their parameter values differ. For each teacher view, compare against student views other than that same view. Backpropagation updates the student only. The teacher starts from the student and follows an exponential moving average (EMA); it is not a separately pretrained expert supplied at the start.

# Teaching pseudocode; not a complete training implementation.
teacher = [stop_grad(teacher_net(v)) for v in global_views]
student = [student_net(v) for v in global_views + local_views]
loss = cross_view_ce(teacher, student, exclude_same_view=True)

Training state: Training and inference roles are separated below.

$$\mathcal L=-\sum_{v\in V_g}\sum_{u\in V,\,u\ne v}\sum_k p_t^{(k)}(v)\log p_s^{(k)}(u)$$

Step-by-step explanation: Two networks, different views, different updates

Centering and sharpening solve different failure modes

A constant output for every image can satisfy view agreement without useful features. Original DINO subtracts a running mean vector c from teacher logits, then applies a low teacher temperature. Centering limits domination by a few prototype dimensions; sharpening resists a uniform output. The center has one value per prototype dimension. Subtracting the same scalar from all logits would cancel inside softmax and do nothing. These operations and the momentum teacher form a tested recipe; changing one toy distribution does not establish non-collapse during training.

# Teaching pseudocode; not a complete training implementation.
target = stop_grad(softmax((teacher_logits - center) / tau_t))
loss = cross_entropy(target, student_logits / tau_s)
center = mc * center + (1 - mc) * batch_mean(teacher_logits)

Training state: Training and inference roles are separated below.

$$p_t=\operatorname{softmax}\!\left(\frac{g_t-c}{\tau_t}\right),\quad c\leftarrow m_c c+(1-m_c)\,\operatorname{mean}_{batch}g_t$$

Step-by-step explanation: Centering and sharpening solve different failure modes

EMA is a parameter update, not a gradient

Let θt be the old teacher and θs the student after its optimizer step. With momentum m=0.9, an old teacher parameter of 0.8 and new student parameter of 0.2 produce 0.74. At m=1 the teacher does not move; at m=0 it copies the student. The loss target is detached before student backpropagation. Teacher EMA and the center EMA are separate state updates with separate coefficients. Downstream inference normally keeps one trained backbone and discards the self-distillation head.

# Teaching pseudocode; not a complete training implementation.
student = optimizer_step(student, loss)
# Separate no-gradient state update:
teacher = momentum * teacher + (1 - momentum) * student

Training state: Training and inference roles are separated below.

$$\theta_t\leftarrow m\theta_t+(1-m)\theta_s,\qquad \nabla_{\theta_t}\mathcal L=0$$

Step-by-step explanation: EMA is a parameter update, not a gradient

Inspect the pinned source

$$\mathcal L=-\sum_{v\in V_g}\sum_{u\in V,\,u\ne v}\sum_k p_t^{(k)}(v)\log p_s^{(k)}(u)$$

Two networks, different views, different updates

The original DINO teacher receives two global crops. The student receives the global crops and smaller local crops. Their backbone and head architecture match, but their parameter values differ. For each teacher view, compare against student views other than that same view. Backpropagation updates the student only. The teacher starts from the student and follows an exponential moving average (EMA); it is not a separately pretrained expert supplied at the start.

$$\mathcal L=-\sum_{v\in V_g}\sum_{u\in V,\,u\ne v}\sum_k p_t^{(k)}(v)\log p_s^{(k)}(u)$$

DINO: cross-view targets, student gradients and an EMA teacher

DINO: cross-view targets, student gradients and an EMA teacher Original teaching diagram; exact configurations are explained in the article. · Paper sourceScroll on narrow screens; select the figure for full size.

Work through the mechanism

Read the architecture and the equations together. Each version changes a specific part of the learning system.

Centering and sharpening solve different failure modes

A constant output for every image can satisfy view agreement without useful features. Original DINO subtracts a running mean vector c from teacher logits, then applies a low teacher temperature. Centering limits domination by a few prototype dimensions; sharpening resists a uniform output. The center has one value per prototype dimension. Subtracting the same scalar from all logits would cancel inside softmax and do nothing. These operations and the momentum teacher form a tested recipe; changing one toy distribution does not establish non-collapse during training.

$$p_t=\operatorname{softmax}\!\left(\frac{g_t-c}{\tau_t}\right),\quad c\leftarrow m_c c+(1-m_c)\,\operatorname{mean}_{batch}g_t$$

DINOv2 adds local targets and a scaling recipe

DINOv2 combines the image-level DINO objective with an iBOT patch objective. The student sees masked patches; an unmasked teacher supplies targets at the matching positions of the same crop. Image and patch losses use separate heads. Sinkhorn–Knopp balances teacher assignments; KoLeo encourages normalized image features to spread within a batch. Curated data, implementation efficiency and a final high-resolution phase also matter. It is inaccurate to explain DINOv2 as DINO with only a larger ViT.

$$\mathcal L_{patch}=-\sum_{i\in M}\sum_k p_{t,i}^{(k)}\log p_{s,i}^{(k)},\quad \mathcal L_{KoLeo}=-\frac1B\sum_i\log\min_{j\ne i}\|\bar f_i-\bar f_j\|_2$$

DINOv3 protects relations between dense features

DINOv3 reports that long training can improve global features while degrading local consistency. Gram anchoring uses an earlier teacher with better dense features as a reference. Normalize patch rows of X and compare XXᵀ: an N×N matrix of patch-to-patch similarities. This constrains relations without forcing every channel to equal the teacher. If Q is orthogonal, (XQ)(XQ)ᵀ=XXᵀ, so channel rotations can preserve the loss. The Gram teacher is separate from the usual EMA target and is periodically refreshed during refinement.

$$X_s,X_g\in\mathbb R^{N\times D},\qquad\mathcal L_{Gram}=\|X_sX_s^T-X_gX_g^T\|_F^2$$

The published implementation computes mean squared error and offers negative-similarity clipping options. Those settings change normalization or the compared matrices; inspect the selected configuration before reproducing the equation numerically.

DINO generations: global targets, masked patches and patch relations

DINO generations: global targets, masked patches and patch relations Original teaching diagram; exact configurations are explained in the article. · Paper sourceScroll on narrow screens; select the figure for full size.

DINOv3 Gram anchoring: preserve patch relations, not individual channels

DINOv3 Gram anchoring: preserve patch relations, not individual channels Original teaching diagram; exact configurations are explained in the article. · Paper sourceScroll on narrow screens; select the figure for full size.

Temperature, centering and teacher updates

Fixed teaching vectors. We compute probabilities and losses, not training stability or model quality.

Temperature τ

0.20

Subtract per-dimension center [0.4, −0.1, 0.2]Teacher EMA momentum m

0.90

Teacher logits: [1, 0.3, −0.5]. Student logits: [0.1, 0.8, −0.2], with temperature 0.4. Old teacher parameter: 0.8. New student parameter: 0.2.

Prototype dimension	Teacher probability	Student probability
0	0.7303	0.1384
1	0.2686	0.7963
2	0.0011	0.0654

Teacher entropy:

0.5901

· CE:

1.5085

· EMA:

0.7400

Reset

Training is not sampling

EMA is a parameter update, not a gradient

Let θt be the old teacher and θs the student after its optimizer step. With momentum m=0.9, an old teacher parameter of 0.8 and new student parameter of 0.2 produce 0.74. At m=1 the teacher does not move; at m=0 it copies the student. The loss target is detached before student backpropagation. Teacher EMA and the center EMA are separate state updates with separate coefficients. Downstream inference normally keeps one trained backbone and discards the self-distillation head.

$$\theta_t\leftarrow m\theta_t+(1-m)\theta_s,\qquad \nabla_{\theta_t}\mathcal L=0$$

Read version changes by role

DINOv3 also changes scale, positional encoding and training. Its large teacher uses RoPE and registers; smaller released students are distilled from a fixed large teacher. The paper does not apply Gram anchoring to that small-model distillation phase. DINOv2 also distinguishes large-model self-distillation from distillation into smaller backbones. Registers supply internal workspace, not a smaller image-token payload. A later text-aligned DINO branch adds training beyond image-only pretraining; it should not be silently treated as ordinary DINO output.

What this does not establish

From a semantic backbone to a reconstruction system

A CLS vector summarizes an image. Patch vectors retain an indexed spatial grid; registers must be excluded before reshaping that grid. RAE trains an additional image decoder on semantic features. A strong segmentation or linear-probe result does not prove lossless reconstruction. Compare the chosen checkpoint, patch size, resolution, feature width, normalization, freeze policy and decoder together. Our numerical lab illustrates target probabilities and EMA arithmetic; it neither trains DINO nor measures feature quality.

Check your understanding

Are these features discrete reconstruction codes?

No. The backbone emits continuous features. Prediction prototypes or text IDs do not make them a VQ codebook. Reconstruction requires a separately specified decoder and training objective.

Sources and next steps

• DINO series · 2104.14294v2
• Method / pinned implementation source
• RAE
• Vision Transformers Need Registers

Return to the learning map

Original teaching diagrams and snippets are identified separately from attributed paper figures. Paper reports, implementation observations, and analytical examples are distinguished. No research model runs in this page.