VAR: predict the next scale, not the next patch

An interactive field guide

What if a generator first decided the rough image, then filled in finer spatial detail in parallel? VAR changes the order and the target of autoregressive prediction.

Before this article: VQGAN · VQ-VAE-2 · Attention and autoregression

What was already possible

VQGAN models a flattened grid one code at a time. VQ-VAE-2 already uses coarse and fine grids, but its priors still generate positions autoregressively within each grid. VAR uses a multi-scale residual tokenizer and trains a Transformer to predict all positions of the next scale together.

What changes, and why

Start with an encoder feature map and a zero reconstruction. At each scale, resize the remaining residual, quantize it with a shared dictionary, upsample and transform the selected vectors, and add them to the reconstruction. The next residual is what remains unexplained. Shared code IDs do not mean that each scale has the same spatial meaning.

Read from left to right. Select a module to connect its role, tensor shape, and calculation. Snippets are teaching pseudocode. Sources provide the paper method or audited implementation.

1. Encode features[B,3,256,256] → [B,32,16,16]2. Residual quantizationscale k: [B,s_k,s_k] IDs3. Next-scale Transformercoarse features → next-scale logits4. Decode the accumulated map[B,32,16,16] → image

Encode features

A convolutional encoder creates the full-resolution latent feature grid.

f = encoder(x)
residual = f
f_hat = zeros_like(f)

Residual quantization

Each scale represents the residual after preceding scales.

q = lookup(quantize(resize(residual, s_k)))
contribution = phi_k(upsample(q))
f_hat += contribution
residual -= contribution

Next-scale Transformer

Teacher-forced inputs must contain only information from earlier target scales.

next_input = resize(f_hat_coarser, s_k)
logits_k = transformer(next_input, level=k)

Decode the accumulated map

The final image decoder reads the sum of scale contributions.

image = decoder(f_hat)

Inspect the pinned source

$$p(r_1,\ldots,r_K)=\prod_{k=1}^{K}p(r_k\mid r_{<k})$$

Work through the mechanism

The small demo counts grids with sides 1,2,3,4. The published default uses sides 1,2,3,4,5,6,8,10,13,16: their squared sizes sum to 680 code positions. Ten scale predictions are not ten code tokens. Nor is token count alone a latency measure.

Change a parameter, inspect the computed values, then compare this small example with the model described above.

Analytical teaching model · controls change the example, not a trained research model.

Included scales

2

Code levels per scale

4

Reset

This teaching schedule uses sides 1, 2, 3, 4. VAR’s published schedule is longer.

Computed values for the current example

Quantity	Value
Toy cumulative tokens	5
Independent-level nominal bits	10.000

Training is not sampling

At scale k, inputs are formed from accumulated coarser reconstructions resized to the target grid. A block-causal mask permits access through that scale’s inputs. It does not expose the true target codes at that scale. The model can therefore use within-scale attention without reading its answer.

Training uses ground-truth coarse information; sampling builds it from earlier generated scales. The public train.py trains stage two and loads a tokenizer. It does not supply the complete original stage-one training recipe. The code uses learned position and level embeddings rather than RoPE.

Reported evidence, not a local reproduction

Setting	Metric	Value	Interpretation
ImageNet 256²; d30-re; paper Table 1	Generation FID ↓	1.73	Includes rejection sampling; not tokenizer rFID.
Default tokenizer schedule	Code positions	680	Sum of squared scale sides; 10 scales.

What this does not establish

The best reported generation score is not a reconstruction score. VAR’s comparisons also change parameter count and sampling choices. Parallel prediction has a different dependency structure from a fully sequential code model; that is a design choice, not a violation of the probability chain rule.

Check your understanding

Why is same-scale attention not target leakage here?

Its inputs come from coarser reconstructions. True same-scale target IDs are labels, not the visible input values.

Sources and next steps

• VAR · 2404.02905v2
• Method / pinned implementation source

Return to the learning map

Diagrams and snippets are teaching reconstructions. Paper reports, implementation observations, and analytical examples are distinguished. No research model runs in this page.