TiTok: a short sequence can describe an image
An interactive field guide
Flattening a grid keeps one token per location. TiTok instead learns a small set of global latent slots.
Before this article: ViT · VQGAN · Evaluation
What was already possible
Section titled “What was already possible”VQGAN compresses images into a smaller spatial grid and quantizes each position. The grid gives a useful local structure, but its length is tied to resolution and downsampling. TiTok asks whether the bottleneck must retain this one-to-one spatial layout.
What changes, and why
Section titled “What changes, and why”The encoder reads image patches together with K learned slots. Self-attention lets each slot gather information from the whole image. Only slot outputs pass through vector quantization. The decoder combines these quantized slots with position-bearing mask queries to recover a dense output. Slots have identities, but no fixed pixel neighborhoods.
Read from left to right. Select a module to connect its role, tensor shape, and calculation. Snippets are teaching pseudocode. Sources provide the paper method or audited implementation.
Read patches and slots
Keep only the learned slot outputs.
tokens = encoder(concat(patches + patch_pos, slots + slot_pos))
z = tokens[:, -K:]Quantize
Each slot selects a codebook vector.
ids, zq = vector_quantizer(z)Decode through queries
The original main recipe predicts proxy codes before pixel decoding.
dense = decoder(concat(zq, mask_queries + output_pos))
proxy_logits = proxy_head(dense)
image = pixel_decoder(proxy_lookup(proxy_logits))Work through the mechanism
Section titled “Work through the mechanism”Follow a 256² image: patch features enter the encoder, 32 latent vectors leave the bottleneck, and many output queries restore the spatial grid. More output queries do not mean more encoded information; they ask the same compact representation different spatial questions.
The demo retains global basis coefficients of a signal. Reduce their count and inspect the lost frequency. This illustrates a global bottleneck, but does not reproduce learned attention or TiTok quality. The paper compares different trained K settings, not this analytic threshold.
Change a parameter, inspect the computed values, then compare this small example with the model described above.
Analytical teaching model · controls change the example, not a trained research model.
Grey: target signal. Green: decoded teaching signal. No trained image model is used.
| Quantity | Value |
|---|---|
| Toy signal MSE | 0.000 |
| Signal samples | 48 |
Training is not sampling
Section titled “Training is not sampling”The original main recipe first predicts proxy codes from an existing MaskGIT-VQGAN. Its pixel decoder produces the image. A second stage freezes the encoder and quantizer while tuning the decoding path. A separate MaskGIT generates TiTok codes. Tokenizer reconstruction and code generation therefore have different objectives and costs.
Reported evidence, not a local reproduction
Section titled “Reported evidence, not a local reproduction”| Setting | Metric | Value | Interpretation |
|---|---|---|---|
| ImageNet 256², L-32, K=32 | rFID ↓ | 2.21 | Validation reconstruction after decoder tuning |
| Same L-32 tokenizer, MaskGIT-ViT, 8 steps | gFID ↓ | 2.77 | Generated images, a separate model stage |
What this does not establish
Section titled “What this does not establish”A Transformer tokenizer is not necessarily a CNN-free deployed decoder. The original recipe uses a proxy model and pixel decoder. The 32-token result must not be combined with the best generation result of the 128-token model. Shorter sampling sequences also do not establish cheaper tokenizer training.
Check your understanding
Section titled “Check your understanding”Can every slot be assigned a fixed square in the image?
No. Its information is learned globally. Dense output queries, rather than latent slots, provide the reconstruction grid.
Sources and next steps
Section titled “Sources and next steps”Diagrams and snippets are teaching reconstructions. Paper reports, implementation observations, and analytical examples are distinguished. No research model runs in this page.