VILA-U: one visual tower for reading and writing

An interactive field guide

VILA-U uses discrete visual codes in a multimodal next-token system, with an extra prediction depth inside each visual position.

Before this article: UniTok · Semantic representations · Attention and autoregression

What was already possible

A multimodal language model can read continuous visual features, while a separate image generator uses another representation. VILA-U instead builds a reconstructable, text-aligned discrete visual tower. The challenge is keeping useful pretrained alignment while learning image recovery.

What changes, and why

Initialize the visual and text components from a pretrained CLIP-like model. Freeze the text tower and train the visual side with reconstruction and contrastive objectives. Residual quantization approximates a high-dimensional feature as a sum of code vectors chosen in sequence. A main LLM handles spatial positions and text; a depth Transformer handles codes within a visual position.

Read from left to right. Select a module to connect its role, tensor shape, and calculation. Snippets are teaching pseudocode. Sources provide the paper method or audited implementation.

1. Aligned visual towerimage → dense semantic features2. Residual quantizationfeature → [depth] IDs per position3. Main and depth predictionposition context → residual-code sequence

Aligned visual tower

Keep the text tower fixed while adapting the visual tower.

visual = image_encoder(image)
text = frozen_text_encoder(caption)

Residual quantization

Each stage codes what earlier stages did not explain.

residual = visual
for level in levels:
    code = nearest(residual,codebook[level])
    residual -= code

Main and depth prediction

Image output still needs a decoder after discrete sampling.

context = main_llm(sequence)
visual_codes = depth_transformer.sample(context)
image = decoder(lookup_and_sum(visual_codes))

Inspect the pinned source

$$r_0=z,\quad q_m=Q_m(r_{m-1}),\quad r_m=r_{m-1}-q_m,\quad \hat z=\sum_m q_m$$

Work through the mechanism

In the scalar demo, choose a target feature and a number of residual stages. Each stage picks from the same tiny teaching codebook and subtracts that choice from the remaining error. The reconstructed feature is the sum of selected codes. Unlike UniTok, later stages see the residual, not a different channel slice. Real VILA-U uses learned high-dimensional vectors.

Change a parameter, inspect the computed values, then compare this small example with the model described above.

Analytical teaching model · controls change the example, not a trained research model.

Target scalar feature

1.7

Residual stages

2

Reset

Each stage codes the remaining residual using the same illustrative scalar codebook {−1,−0.5,0,0.5,1}.

Computed values for the current example

Quantity	Value
Stage 1 code	1.000
Stage 2 code	0.500
Reconstructed feature	1.500
Remaining error	0.200

Training is not sampling

After visual-tower training, adapt the multimodal LLM to image/text sequences. Image understanding consumes visual codes before text answers. Image generation predicts visual codes before pixel decoding. Both use next-token learning, but the main sequence and residual-code depth are distinct dependency axes.

What this does not establish

The paper reports difficulties with training its joint visual tower from scratch. That does not refute UniTok under different quantizers, data, and optimization. This article is based on the paper method; official code was not audited. A successful tokenizer alone is not the full VQA or generation system.

Check your understanding

What does the depth Transformer predict?

It predicts residual codes within one visual position. The main LLM models dependencies between visual positions and text tokens.

Sources and next steps

• VILA-U · 2409.04429v3
• Method / pinned implementation source

Return to the learning map

Diagrams and snippets are teaching reconstructions. Paper reports, implementation observations, and analytical examples are distinguished. No research model runs in this page.