ViT: an image becomes a sequence

An interactive field guide

A patch is a group of pixels. A token is the vector used to represent it. ViT makes this interface explicit.

Before this article: Attention and autoregression · Autoencoders

What was already possible

CNNs already learned visual features with shared local filters. Transformers already mixed sequence elements through attention. The question was whether a mostly standard Transformer could learn useful image features at scale.

What changes, and why

ViT divides an image into non-overlapping patches. It flattens each patch, projects it to width D, adds a learned position vector, and applies a bidirectional Transformer encoder. A separate class token feeds the classification head. The new contribution concerns image recognition and scaling, not image reconstruction or autoregression.

Read from left to right. Select a module to connect its role, tensor shape, and calculation. Snippets are teaching pseudocode. Sources provide the paper method or audited implementation.

1. Patch projection[B,H,W,3] → [B,N,D]2. Position and attention[B,N+1,D]3. Readout[B,D] → [B,classes]

Patch projection

One matrix is shared across patches. Spatial mixing comes later.

patches = patchify(image, P)
z = patches @ W + bias

Position and attention

Add learned absolute positions, then use bidirectional attention.

z = concat(cls_token, z) + position
z = transformer(z, causal=False)

Readout

The original head classifies the class token; it does not unpatchify an image.

logits = classifier(z[:, 0])

Inspect the pinned source

$$N=HW/P^2,\quad z_i=\operatorname{flatten}(x_i)W+e_i$$

Work through the mechanism

For a 224 × 224 RGB image and 16 × 16 patches, N=196. Each raw patch has 768 values. A learned matrix maps these to D features. Attention then mixes information across 197 positions, including the class token. N counts positions; D counts features at each position.

The small grid below uses smaller images. Double the patch side while holding image size fixed. The patch count becomes one quarter. Dense attention scores have N × N entries, so their count becomes one sixteenth. This is a score-matrix estimate, not an end-to-end speed measurement.

Change a parameter, inspect the computed values, then compare this small example with the model described above.

Analytical teaching model · controls change the example, not a trained research model.

Image side H (pixels)

32

Patch scale k (P = 4 × 2ᵏ)

1

Reset

Patch projection changes feature width. It does not, by itself, reduce the number of patches.

Computed values for the current example

Quantity	Value
N = (H / P)²	16
Raw values / patch	192

Training is not sampling

Original ViT uses supervised classification. It does not predict the next image token. During higher-resolution fine-tuning, patch size stays fixed and the learned patch-position table is interpolated on its two-dimensional grid. The stored table is a 1D sequence; the interpolation still uses spatial structure.

What this does not establish

To reconstruct pixels, add a bottleneck, output queries, a pixel head, and a reconstruction objective. TiTok and RAE make different choices for these parts. Merely replacing a CNN encoder with ViT does not specify a complete autoencoder.

Check your understanding

Is a continuous patch vector automatically an autoregressive token?

No. Continuity describes the value space. Autoregression describes a conditional factorization and its training or sampling order.

Sources and next steps

• ViT · 2010.11929v2
• Method / pinned implementation source
• Semantic representations
• RoPE-ViT
• TiTok
• DiT
• AIM

Return to the learning map

Diagrams and snippets are teaching reconstructions. Paper reports, implementation observations, and analytical examples are distinguished. No research model runs in this page.