VQ-VAE-2: separate global structure and local detail

An interactive field guide

A single latent grid must describe both layout and texture. VQ-VAE-2 learns a hierarchy, then gives each level its own conditional prior.

Before this article: VQ-VAE · Attention and autoregression

What was already possible

VQ-VAE already supplies discrete codes and a learned prior. Increasing image resolution makes one large prior expensive. VQ-VAE-2 retains feed-forward encoding and decoding, but distributes information over multiple spatial scales.

What changes, and why

For the 256×256 configuration, top codes form a 32×32 grid and bottom codes a 64×64 grid. Bottom encoding uses top information as well as image features. The final decoder uses both. The top prior models global correlations; the bottom prior conditions on the sampled top grid.

Read from left to right. Select a module to connect its role, tensor shape, and calculation. Snippets are teaching pseudocode. Sources provide the paper method or audited implementation.

1. Top encoder and quantizer[B,32,32,D]2. Condition bottom encoding[B,64,64,D]3. Two priors, one image decoderp(top) × p(bottom | top)

Top encoder and quantizer

A second downsampling stage creates a coarse latent map.

h = bottom_encoder(x)
z_top = quantize_top(top_encoder(h))

Condition bottom encoding

Combine image features with decoded top information before bottom quantization.

top_features = top_decoder(z_top)
z_bottom = quantize_bottom(combine(h, top_features))

Two priors, one image decoder

Both levels matter at image decoding. The priors are trained after the tokenizer.

top = sample(top_prior)
bottom = sample(bottom_prior, condition=top)
image = decode(top, bottom)

Inspect the pinned source

$$p(z_{\mathrm{top}},z_{\mathrm{bottom}})=p(z_{\mathrm{top}})p(z_{\mathrm{bottom}}\mid z_{\mathrm{top}})$$

Work through the mechanism

Change the two teaching codes independently. One changes the broad signal phase; the other changes its local variation. This illustrates complementary controls, not a guarantee that real latent maps cleanly separate shape and texture. Both code levels are learned jointly during reconstruction.

Change a parameter, inspect the computed values, then compare this small example with the model described above.

Analytical teaching model · controls change the example, not a trained research model.

Top code

1

Bottom code

2

Reset

Grey: target signal. Green: decoded teaching signal. No trained image model is used.

Computed values for the current example

Quantity	Value
Toy signal MSE	0.027
Signal samples	48

Training is not sampling

Generation samples top codes first, then bottom codes conditioned on them, then decodes both in one feed-forward image pass. Within each grid, the priors are still autoregressive. Hierarchical grids therefore do not imply VAR-style parallel next-scale prediction.

What this does not establish

The paper also uses classifier-based rejection sampling to trade coverage for sample quality. That is an extra sampling mechanism, not a free improvement to the tokenizer. A deeper hierarchy is also different from residual quantization at one spatial position.

Check your understanding

Can the bottom prior generate first and still use the published factorization?

No. The bottom prior needs the top codes as context. Reversing the order requires a different model of the conditional distribution.

Sources and next steps

• Method / pinned implementation source
• VAR

Return to the learning map

Diagrams and snippets are teaching reconstructions. Paper reports, implementation observations, and analytical examples are distinguished. No research model runs in this page.