MAR: autoregression without a codebook

An interactive field guide

Autoregression specifies what is conditioned on. It does not require integer-valued tokens.

Before this article: Diffusion and flow matching · Attention and autoregression · Evaluation

What was already possible

A categorical head predicts probabilities over discrete IDs. Direct L2 regression of a continuous next token predicts a conditional mean under a fixed-variance Gaussian model. MAR uses a small conditional diffusion model to represent a richer distribution over each continuous token.

What changes, and why

First, a Transformer reads known tokens and produces a context feature for an unknown position. Then a diffusion MLP samples that token conditioned on the feature. The paper studies causal AR and masked AR. Default MAR fills groups in a random order, allowing known tokens to interact bidirectionally without exposing unknown target values.

Read from left to right. Select a module to connect its role, tensor shape, and calculation. Snippets are teaching pseudocode. Sources provide the paper method or audited implementation.

1. Known-token contextvisible [B,Nknown,C] → context features2. Conditional diffusion headcontext + noisy token + t → token noise3. Grouped samplingknown set → larger known set → full grid

Known-token context

Unknown values never enter the encoder.

visible = remove_masked(latents,mask)
context = masked_transformer(visible,positions)

Conditional diffusion head

The MLP is reused through the inner denoising loop.

prediction = diffusion_mlp(noisy_target,t,context)
loss = mse(prediction,target_noise)

Grouped sampling

Recompute outer context after inserting newly sampled tokens.

for group in generation_order:
    context = transformer(known)
    known[group] = token_sampler(context[group])
image = tokenizer.decode(known)

Inspect the pinned source

$$p(x)=\prod_g p_\theta(x_{G_g}\mid x_{G_{<g}})$$

Work through the mechanism

The default KL-16 tokenizer produces 16 × 16 × 16 continuous latents for 256² images. Masked positions are removed before the encoder, then restored as queries in the decoder. Spatial positions remain attached to their original grid even when generation order is random.

The demo separates outer generation groups from inner denoising steps. For 16 tokens, group size 4 and 4 inner steps yield 4 outer groups and 16 group-step iterations. Those iterations run a small head, not necessarily the full Transformer. The toy count therefore is not a latency measurement.

Change a parameter, inspect the computed values, then compare this small example with the model described above.

Analytical teaching model · controls change the example, not a trained research model.

Tokens generated per group

4

Inner denoising steps

4

Reset

Numbers show generation groups, not token values. Network size and caching still determine runtime.

Computed values for the current example

Quantity	Value
Outer groups	4
Inner steps per group	4
Group × step evaluations	16

Training is not sampling

Diffusion Loss trains the head to denoise target tokens. It is not direct L2 prediction of a clean next token. In the pinned defaults, the outer MAR sampler uses 64 rounds and the head uses 100 denoising steps. Context features can be reused during a token-head trajectory. A separate image decoder converts the completed latent grid to RGB.

Reported evidence, not a local reproduction

Setting	Metric	Value	Interpretation
ImageNet-256, 400 epochs, continuous, no CFG	FID ↓ raster causal / random causal	19.23 / 13.07	Table 1, order changes
Same table, random bidirectional: single / grouped	FID ↓	3.43 / 3.50	Attention and grouping are separate axes

What this does not establish

Group prediction does not model all within-group dependencies as a fully sequential factorization would. Bidirectional context changes also prevent blindly reusing strict causal KV-cache assumptions. The comparisons below vary order and attention; their entire gain cannot be attributed only to removing quantization.

Check your understanding

Is a squared denoising loss the same as direct squared next-token error?

No. The denoiser learns predictions across noise levels, which define a sampled distribution. Direct regression produces one point estimate from context.

Sources and next steps

• MAR · 2406.11838v3
• Method / pinned implementation source
• Fluid
• CausalFusion
• NextStep-1

Return to the learning map

Diagrams and snippets are teaching reconstructions. Paper reports, implementation observations, and analytical examples are distinguished. No research model runs in this page.