VQGAN: a tokenizer that preserves perceptual detail

An interactive field guide

Pixel accuracy alone can make reconstructions smooth. VQGAN combines a discrete bottleneck with perceptual and adversarial training, then models its codes with a Transformer.

Before this article: VQ-VAE · Attention and autoregression · Evaluation

What was already possible

VQ-VAE already learns a dictionary and a decoder. PixelCNN-style priors model the code sequence. GANs already encourage realistic image statistics, and pretrained feature networks already provide perceptual distances. VQGAN combines these tools into a compact image tokenizer plus an autoregressive Transformer prior.

What changes, and why

The first-stage convolutional encoder reduces image size. Nearest-code quantization retains a discrete interface. The decoder is trained with pixel and LPIPS losses, while a patch discriminator distinguishes real from reconstructed image regions. An adaptive weight can balance reconstruction and adversarial gradients at the decoder’s last layer.

Read from left to right. Select a module to connect its role, tensor shape, and calculation. Snippets are teaching pseudocode. Sources provide the paper method or audited implementation.

1. Convolutional tokenizerimage → [B,h,w,D] → IDs2. Image decoder and lossesreconstruction: [B,3,H,W]3. Patch discriminatorimage → patch logits4. Autoregressive priorIDs: [B,N] → logits: [B,N,K]

Convolutional tokenizer

The bottleneck follows VQ-VAE’s nearest-code structure.

z = encoder(x)
z_q, ids, vq_loss = quantizer(z)

Image decoder and losses

Pixel and perceptual terms compare the input with its reconstruction.

x_hat = decoder(z_q)
rec = pixel_loss(x,x_hat) + lpips(x,x_hat)

Patch discriminator

Generator and discriminator losses use different gradient paths.

g_adv = -mean(D(x_hat))
L_G = rec + adaptive_weight * g_adv + vq_loss

Autoregressive prior

The prior models code occurrence, not pixel reconstruction.

logits = transformer(ids[:, :-1])
L_AR = cross_entropy(logits, ids[:, 1:])

Inspect the pinned source

$$L_G=L_{\mathrm{pixel}}+L_{\mathrm{perceptual}}+\lambda L_{\mathrm{adv}}+L_{\mathrm{VQ}}$$

Work through the mechanism

Vary the detail amplitude in the signal demo. Matching fine structure lowers paired error only when its phase and strength match the target. A plausible extra oscillation is not necessarily the original detail. The example exposes why realism and exact reconstruction require separate evidence; it does not simulate a GAN or LPIPS network.

Change a parameter, inspect the computed values, then compare this small example with the model described above.

Analytical teaching model · controls change the example, not a trained research model.

Fine-detail amplitude

0.5

Target detail amplitude

0.2

Reset

Grey: target signal. Green: decoded teaching signal. No trained image model is used.

Computed values for the current example

Quantity	Value
Toy signal MSE	0.045
Signal samples	48

Training is not sampling

The discriminator and generator have different updates. Stop gradients through fake images for the discriminator step; freeze discriminator parameters for the generator step while allowing gradients to its input. LPIPS features are fixed during this use. The Transformer prior is trained after the tokenizer, predicting shifted code IDs with cross-entropy.

The inspected taming implementation retains a legacy quantizer option that places β on the old term. Its adaptive-weight epsilon also differs from the paper. Read the exact version before copying a training recipe: a repository default is not automatically the paper configuration.

What this does not establish

Adversarial training adds an optimizer, a schedule, and possible instability. A perceptually convincing reconstruction can change small letters or textures. The paper’s full image-generation quality includes the second-stage Transformer and its sampling settings, not just the tokenizer.

Check your understanding

Does a Transformer generate the first-stage reconstruction?

No. The tokenizer uses convolutional encoding and decoding. The Transformer models the discrete codes in the second stage.

Sources and next steps

• VQGAN · 2012.09841v3
• Method / pinned implementation source
• VAR
• TiTok
• Fluid

Return to the learning map

Diagrams and snippets are teaching reconstructions. Paper reports, implementation observations, and analytical examples are distinguished. No research model runs in this page.