Visual tokenizers in multimodal models

An interactive field guide

Follow pixels into a language model, and separate that interface from an image codec.

Before this article: ViT · Semantic representations · Vision Transformers Need Registers

What was already possible

Three meanings of an image token

In a VQ autoencoder, an image token is often a codebook index. In a ViT, it is usually a continuous feature vector. In an LLM input, an image placeholder ID marks where a visual vector will be inserted. The placeholder is not a visual vocabulary entry that encodes the image by itself.

A visual tokenizer can mean the complete input path: resize, split pixels, encode features, reduce positions, and align widths. A reconstruction tokenizer also needs a pixel decoder and a suitable objective. A model that answers questions about an image does not thereby provide that decoder.

What changes, and why

Read from left to right. Select a module to connect its role, tensor shape, and calculation. Snippets are teaching pseudocode. Sources provide the paper method or audited implementation.

1. Trace the interface between five stagesSee the stage equations below2. Count positions, dimensions and valid entries separatelySee the stage equations below

Trace the interface between five stages

First record the resized image size, not only the source file size. Then record the patch grid, encoder width, compression rule and LLM width. Spatial grouping changes the number of positions. Projection changes feature width. Neither operation is automatically vector quantization.

image = resize_with_model_rules(image)
patches = patchify(image)
features = visual_path(patches)
visual_embeddings = connector(features)
llm_input[image_positions] = visual_embeddings

Training state: Static source inspection; training state is discussed below.

$$N_{\rm patch}=\frac{H^\prime W^\prime}{p^2},\quad N_{\rm visual}=\frac{N_{\rm patch}}{m^2},\quad V\in\mathbb R^{N_{\rm visual}\times D_{\rm LLM}}$$

Step-by-step explanation: Trace the interface between five stages

Count positions, dimensions and valid entries separately

A 256×4096 continuous representation contains 1,048,576 scalars. Calling it “256 tokens” does not give a bitrate. Storage precision also differs from learned information content. Compare the effective image area, vector width and downstream task before drawing conclusions about efficiency.

valid_visual = sum(valid_image_positions)
# Do not count padded image slots as content.
# Do not call valid_visual the complete context length.

Training state: Static source inspection; training state is discussed below.

$$N_{\rm context}=N_{\rm visual}+N_{\rm text}+N_{\rm delimiters}+N_{\rm timestamp\ text}$$

Step-by-step explanation: Count positions, dimensions and valid entries separately

Inspect the pinned source

$$\text{visual token count}\ne\text{codebook size}\ne\text{complete context length}$$

Trace the interface between five stages

Teaching diagram · From pixels to language-model inputs Read each named row from left to right. Rows show distinct pathways or comparison cases. · Method sourceSelect the image to inspect it at full size.

First record the resized image size, not only the source file size. Then record the patch grid, encoder width, compression rule and LLM width. Spatial grouping changes the number of positions. Projection changes feature width. Neither operation is automatically vector quantization.

For a prepared H′×W′ image, square patch side p and spatial merge side m, the encoder grid has H′W′/p² positions and the LLM receives H′W′/(pm)² visual vectors. For Gemma 4 12B, use p=48 and m=1 at the model interface. Its earlier 16-pixel processor units are regrouped raw pixels.

$$N_{\rm patch}=\frac{H^\prime W^\prime}{p^2},\quad N_{\rm visual}=\frac{N_{\rm patch}}{m^2},\quad V\in\mathbb R^{N_{\rm visual}\times D_{\rm LLM}}$$

image = resize_with_model_rules(image)
patches = patchify(image)
features = visual_path(patches)
visual_embeddings = connector(features)
llm_input[image_positions] = visual_embeddings

Similar token counts can hide different computations

Teaching diagram · Three different uses of extra or fewer tokens Read each named row from left to right. Rows show distinct pathways or comparison cases. · Method sourceSelect the image to inspect it at full size.

Qwen3-VL and Qwen3.5 use learned mergers: four neighboring feature vectors are concatenated and passed through an MLP. Gemma 4 encoder variants average groups of nine encoded spatial positions. Gemma 4 12B directly embeds larger pixel blocks. These operations differ in what is mixed before positions are reduced.

Qwen3-VL adds intermediate visual features to three early LLM layers. In the inspected Qwen3.5 implementation, only the final merged visual output enters the LLM input path. Gemma 4 must be split into small-encoder, large-encoder and encoder-free variants. The table below uses specific checkpoint revisions, not family-wide guesses.

Work through the mechanism

Count positions, dimensions and valid entries separately

A 256×4096 continuous representation contains 1,048,576 scalars. Calling it “256 tokens” does not give a bitrate. Storage precision also differs from learned information content. Compare the effective image area, vector width and downstream task before drawing conclusions about efficiency.

A padded batch tensor may contain empty slots. Language-model context can also include image delimiters, newlines, timestamps and ordinary text. The calculator counts valid visual positions only. Prepared-video mode assumes frame sampling and resizing have already happened; it does not estimate a complete serving request.

$$N_{\rm context}=N_{\rm visual}+N_{\rm text}+N_{\rm delimiters}+N_{\rm timestamp\ text}$$

valid_visual = sum(valid_image_positions)
# Do not count padded image slots as content.
# Do not call valid_visual the complete context length.

Trace pixels to visual tokens

Shape arithmetic, without model inference. Image mode follows the pinned resize rules. Here a Qwen budget sets the pixel-area limit corresponding to a maximum visual-token count; it is not a billing estimate.

Height (pixels)Width (pixels)Sampled frames (video mode)

Stage / quantity	Result
Resized height	512
Resized width	512
Temporal groups	1
Repeated frames	1
Model patch-grid rows	32
Model patch-grid columns	32
Processor spatial units × time groups	1024
Model patch positions	1024
Effective visual tokens	256
LLM visual feature shape	[256, 4096]
Continuous scalar values	1048576

2×16×16 patches → ViT → 2×2 concat + MLP → LLM

27 vision blocks · width 1152 → 4096

Input: visual embeddings occupy image positions; text embeddings remain.

Pinned configuration for this variant

Reset

Effective tokens exclude padding slots, image boundaries, newlines, timestamp text and other text. Video mode does not sample frames or allocate video budgets: frame sides must be multiples of 32 for Qwen or 48 for Gemma. Repeating a still image satisfies the temporal patch interface; it adds no motion information.

Inspected checkpoint configurations

Checked: 2026-11-03. Each row links to a pinned version. Matching dimensions do not imply matching weights. The 12B patch column denotes the pixel block received by the model.

Variant	Model patch	Vision blocks / width	Spatial operation	LLM width	DeepStack zero-based blocks
Qwen3-VL-2B-Instruct	16×16	24 / 1024	2×2 concat + MLP	2048	5, 11, 17
Qwen3-VL-8B-Instruct	16×16	27 / 1152	2×2 concat + MLP	4096	8, 16, 24
Qwen3-VL-235B-A22B-Instruct	16×16	27 / 1152	2×2 concat + MLP	4096	8, 16, 24
Qwen3.5-2B	16×16	24 / 1024	2×2 concat + MLP	2048	—
Qwen3.5-9B	16×16	27 / 1152	2×2 concat + MLP	4096	—
Qwen3.5-397B-A17B	16×16	27 / 1152	2×2 concat + MLP	4096	—
gemma-4-E2B-it	16×16	16 / 768	3×3 average	1536	—
gemma-4-E4B-it	16×16	16 / 768	3×3 average	2560	—
gemma-4-12B-it	48×48	0 / 3840	Pixel projection	3840	—
gemma-4-31B-it	16×16	27 / 1152	3×3 average	5376	—
gemma-4-26B-A4B-it	16×16	27 / 1152	3×3 average	2816	—

“—” means no DeepStack injection in the inspected path, not an absence of visual information. All branches pass continuous vectors; text vocabulary sizes are not treated as visual codebooks.

Training and inference

Why the pretraining objective matters

CLIP uses contrastive image–text matching. SigLIP uses a sigmoid-based pairwise objective; SigLIP-2 adds a broader training recipe. DINO-style methods use self-distillation across image views. These objectives can yield useful semantic features, but they do not explicitly require exact recovery of every pixel.

Multimodal text training can update a connector, an encoder and an LLM under different schedules. A frozen encoder still computes features, and its input gradients may still exist. An early-fusion training claim does not mean that the architecture lacks an encoder. Weight quantization for deployment does not make continuous visual features into VQ codes.

What this does not establish

What can transfer to a reconstruction project

A useful UAR experiment would freeze one specified encoder and compare a decoder trained on features before and after the merger. Hold image data and decoder capacity fixed, and measure reconstruction, semantic utility and cost separately. This is a proposed experiment, not a result reported by the three families.

This survey is a snapshot checked on 2026-11-03. It focuses on the named families. It does not rank incomparable benchmark numbers, infer missing visual pretraining recipes, or replace the current VAR-first roadmap.

Check your understanding

Do equal visual-token counts imply equal compression or quality?

No. Feature width, input resolution, representation type, training and downstream use can differ.

Sources and next steps

• Method / pinned implementation source
• Qwen3-VL: from patches to DeepStack
• Qwen3.5: audit the visual input path
• Gemma 4: encoders, pooling and an encoder-free branch

Return to the learning map

Original teaching diagrams and snippets are identified separately from attributed paper figures. Paper reports, implementation observations, and analytical examples are distinguished. No research model runs in this page.