Semantic features: useful for which task?

An interactive field guide

A representation can keep an object identity while dropping its texture. That can help classification and hurt exact reconstruction.

Before this article: Autoencoders · ViT · Evaluation

What was already possible

Reconstruction asks which pixels were present. Recognition asks which properties matter for a task. A feature that ignores lighting or small texture changes can be useful for recognition. The same invariance removes information a decoder may need. There is no single scalar called semantic quality that answers every task.

What changes, and why

Contrastive image-text learning brings paired image and text embeddings together relative to mismatched pairs. Self-supervised visual learning can instead use views or patch targets without paired text. Both can produce useful visual features, but visual usefulness alone does not imply a shared text space.

Read from left to right. Select a module to connect its role, tensor shape, and calculation. Snippets are teaching pseudocode. Sources provide the paper method or audited implementation.

1. Frozen or trainable encoderimage → [B,N,D]2. Task readoutpooled [B,D] → labels or similarity3. Reconstruction readout[B,N,D] → [B,3,H,W]

Frozen or trainable encoder

Record what was pretrained and which weights change.

features = encoder(image)
if probe_mode: features = stop_gradient(features)

Task readout

A probe and a text-similarity head need different training data.

probe_logits = linear(pool(features))
text_scores = normalize(pool(features)) @ normalize(text_features).T

Reconstruction readout

A decoder tests retained image information, under its own capacity and training.

reconstruction = image_decoder(features)

Inspect the pinned source

$$\ell_i=-\log\frac{\exp(s(v_i,t_i)/T)}{\sum_j\exp(s(v_i,t_j)/T)}$$

Work through the mechanism

The demo has an invented semantic factor fixed at 1 and a texture factor. Lowering the retained detail weight makes the representation insensitive to texture. That may help a hypothetical label depending only on the first factor, but it increases reconstruction error for the second. Real features do not come with these cleanly separated axes.

A linear probe freezes the encoder and trains a linear classifier on labeled examples. Zero-shot classification instead compares images with text-encoded class prompts and requires a compatible text encoder. VQA trains or evaluates a larger system that must read the image and answer a question. These tests are not interchangeable.

Change a parameter, inspect the computed values, then compare this small example with the model described above.

Analytical teaching model · controls change the example, not a trained research model.

Retained detail weight

0.5

Input texture strength

0.7

Reset

Two invented factors illustrate invariance versus detail. Neither axis is an empirical semantic score.

Computed values for the current example

Quantity	Value
Input (semantic, detail)	1, 0.700
Retained representation	1, 0.350
Lost detail squared	0.122

Training is not sampling

In RAE, semantics mainly comes from the pretrained frozen encoder. In MAETok, masking and auxiliary targets shape a trainable latent space. UniTok and VILA-U combine reconstruction with image-text alignment. Compare pretraining data, frozen modules, and downstream training separately before attributing a gain to the tokenizer.

What this does not establish

A high probe score does not establish text alignment. Low reconstruction error does not establish recognition. A downstream multimodal model can also compensate for a weaker tokenizer through more data or training. Report each stage and its evaluation protocol.

Check your understanding

Can a DINO-style visual encoder perform text zero-shot classification by itself?

A visual feature extractor alone does not supply the required aligned text encoder. A linear probe uses labeled training data and is a different test.

Sources and next steps

• Method / pinned implementation source
• RAE
• MAETok
• UniTok
• VILA-U
• AIM
• EOSTok
• RAE-T2I

Return to the learning map

Diagrams and snippets are teaching reconstructions. Paper reports, implementation observations, and analytical examples are distinguished. No research model runs in this page.