Qwen3-VL: from patches to DeepStack

An interactive field guide

Trace one image through the vision encoder, spatial merger and three residual feature injections.

Before this article: Visual tokenizers in multimodal models · RoPE-ViT

What was already possible

A semantic encoder needs a language interface

A pretrained visual encoder can return one continuous feature per patch. A language model expects vectors of its own hidden width and a manageable sequence length. A merger handles these interface differences. It is not an image decoder and does not select discrete image vocabulary entries.

The report starts from SigLIP-2 checkpoints and continues training at dynamic resolutions. SO-400M is the default encoder family; smaller 2B/4B language models use SigLIP2-Large. Calling the result an unchanged frozen SigLIP-2 would miss this continued training.

What changes, and why

Read from left to right. Select a module to connect its role, tensor shape, and calculation. Snippets are teaching pseudocode. Sources provide the paper method or audited implementation.

1. Pixels become a dynamic spatial gridSee the stage equations below2. Four spatial vectors become one language-width vectorSee the stage equations below3. DeepStack adds features without appending positionsSee the stage equations below

Pixels become a dynamic spatial grid

The 8B checkpoint uses a 27-block vision encoder of width 1152. Its spatial patch side is 16, temporal patch size is 2 and spatial merge side is 2. The image processor adjusts both sides to multiples of 32 under pixel-area limits. For a still image, repeated input frames satisfy the temporal patch interface; they do not add new observations.

pixels, grid_thw = image_processor(image)
z = patch_embed_3d(pixels)
z = z + interpolated_absolute_positions
z = vision_blocks(z, spatial_rotary_positions)

Training state: Static source inspection; training state is discussed below.

$$X_{\rm tube}\in\mathbb R^{N\times(2\cdot16\cdot16\cdot3)}\longrightarrow Z\in\mathbb R^{N\times1152}$$

Step-by-step explanation: Pixels become a dynamic spatial grid

Four spatial vectors become one language-width vector

The main merger normalizes visual vectors, concatenates each spatial 2×2 group, applies a two-layer MLP and returns the LLM width. For 512×512 prepared pixels, the encoder processes 1024 patch positions. The merger packs them into 256 vectors, each initially 4×1152=4608 wide, then projects to 4096 for 8B.

group = spatially_group(norm(z), side=2)
merged = fc2(gelu(fc1(group)))
inputs_embeds[image_mask] = merged

Training state: Static source inspection; training state is discussed below.

$$v_g=W_2\,\operatorname{GELU}\!\left(W_1\operatorname{concat}_{i\in g}\operatorname{LN}(z_i)+b_1\right)+b_2$$

Step-by-step explanation: Four spatial vectors become one language-width vector

DeepStack adds features without appending positions

The 8B visual block indices [8,16,24] are zero-based. Dedicated mergers turn those intermediate features into LLM-width vectors. After each of the first three language blocks, one feature set is added at visual positions only. Text positions do not receive that direct addition.

h = llm_block[layer](h)
if layer < len(deepstack_features):
    h[visual_mask] += deepstack_features[layer]

Training state: Static source inspection; training state is discussed below.

$$H^{\ell+1}_{\mathcal I}\leftarrow H^{\ell+1}_{\mathcal I}+M_\ell(Z^{k_\ell}),\qquad \ell\in\{0,1,2\}$$

Step-by-step explanation: DeepStack adds features without appending positions

Inspect the pinned source

$$N_{\rm visual}=\frac{H^\prime W^\prime}{32^2}$$

Pixels become a dynamic spatial grid

Teaching diagram · Qwen3-VL: visual features enter a language model Read each named row from left to right. Rows show distinct pathways or comparison cases. · Method sourceSelect the image to inspect it at full size.

The 8B checkpoint uses a 27-block vision encoder of width 1152. Its spatial patch side is 16, temporal patch size is 2 and spatial merge side is 2. The image processor adjusts both sides to multiples of 32 under pixel-area limits. For a still image, repeated input frames satisfy the temporal patch interface; they do not add new observations.

A learned Conv3d maps each 2×16×16 RGB block to a visual vector. Interpolated absolute position embeddings and visual 2D rotary embeddings supply spatial structure. The language backbone separately uses interleaved MRoPE. These position mechanisms belong to different stages.

$$X_{\rm tube}\in\mathbb R^{N\times(2\cdot16\cdot16\cdot3)}\longrightarrow Z\in\mathbb R^{N\times1152}$$

pixels, grid_thw = image_processor(image)
z = patch_embed_3d(pixels)
z = z + interpolated_absolute_positions
z = vision_blocks(z, spatial_rotary_positions)

• Vision and input code — Qwen3VLVisionModel

DeepStack adds features without appending positions

Teaching diagram · DeepStack: addition at existing visual positions Read each named row from left to right. Rows show distinct pathways or comparison cases. · Method sourceSelect the image to inspect it at full size.

The 8B visual block indices [8,16,24] are zero-based. Dedicated mergers turn those intermediate features into LLM-width vectors. After each of the first three language blocks, one feature set is added at visual positions only. Text positions do not receive that direct addition.

This is a residual feature injection, not cross-attention to a separate image memory. The final merged features already supplied the initial visual embeddings. Adding three feature sets later does not multiply the image sequence length by four. The 2B model instead selects [5,11,17], so layer indices must come from its own configuration.

I

The positions occupied by visual embeddings.

M

A dedicated merger for an intermediate visual layer.

$$H^{\ell+1}_{\mathcal I}\leftarrow H^{\ell+1}_{\mathcal I}+M_\ell(Z^{k_\ell}),\qquad \ell\in\{0,1,2\}$$

h = llm_block[layer](h)
if layer < len(deepstack_features):
    h[visual_mask] += deepstack_features[layer]

Work through the mechanism

Four spatial vectors become one language-width vector

The main merger normalizes visual vectors, concatenates each spatial 2×2 group, applies a two-layer MLP and returns the LLM width. For 512×512 prepared pixels, the encoder processes 1024 patch positions. The merger packs them into 256 vectors, each initially 4×1152=4608 wide, then projects to 4096 for 8B.

The processor orders patches so that these neighbors are grouped correctly. A naive reshape of an arbitrary raster sequence can group the wrong locations. The operation is learned compression of features, not average pooling and not nearest-code selection.

$$v_g=W_2\,\operatorname{GELU}\!\left(W_1\operatorname{concat}_{i\in g}\operatorname{LN}(z_i)+b_1\right)+b_2$$

group = spatially_group(norm(z), side=2)
merged = fc2(gelu(fc1(group)))
inputs_embeds[image_mask] = merged

Trace pixels to visual tokens

Shape arithmetic, without model inference. Image mode follows the pinned resize rules. Here a Qwen budget sets the pixel-area limit corresponding to a maximum visual-token count; it is not a billing estimate.

Height (pixels)Width (pixels)Sampled frames (video mode)

Stage / quantity	Result
Resized height	512
Resized width	512
Temporal groups	1
Repeated frames	1
Model patch-grid rows	32
Model patch-grid columns	32
Processor spatial units × time groups	1024
Model patch positions	1024
Effective visual tokens	256
LLM visual feature shape	[256, 4096]
Continuous scalar values	1048576

2×16×16 patches → ViT → 2×2 concat + MLP → LLM

27 vision blocks · width 1152 → 4096

Input: visual embeddings occupy image positions; text embeddings remain.

Pinned configuration for this variant

Reset

Effective tokens exclude padding slots, image boundaries, newlines, timestamp text and other text. Video mode does not sample frames or allocate video budgets: frame sides must be multiples of 32 for Qwen or 48 for Gemma. Repeating a still image satisfies the temporal patch interface; it adds no motion information.

Inspected checkpoint configurations

Checked: 2026-11-03. Each row links to a pinned version. Matching dimensions do not imply matching weights. The 12B patch column denotes the pixel block received by the model.

Variant	Model patch	Vision blocks / width	Spatial operation	LLM width	DeepStack zero-based blocks
Qwen3-VL-2B-Instruct	16×16	24 / 1024	2×2 concat + MLP	2048	5, 11, 17
Qwen3-VL-8B-Instruct	16×16	27 / 1152	2×2 concat + MLP	4096	8, 16, 24
Qwen3-VL-235B-A22B-Instruct	16×16	27 / 1152	2×2 concat + MLP	4096	8, 16, 24

“—” means no DeepStack injection in the inspected path, not an absence of visual information. All branches pass continuous vectors; text vocabulary sizes are not treated as visual codebooks.

Training and inference

Separate the training stages and the ablation setting

The report describes an alignment stage that updates only merger layers, then full-parameter pretraining with longer contexts. Its DeepStack ablation uses an internal 15B-A2B model trained on 200B tokens, without post-training. The reported average changes from 74.7 to 76.0 and DocVQA from 89.5 to 91.1. These are not scores for the public 8B checkpoint.

For video, frame sampling, spatial resizing, temporal grouping and textual timestamps all matter. The calculator starts from already sampled and resized video frames. If three frames enter a temporal-size-two encoder, repeating the last frame yields two time groups. Timestamp text costs remain outside its count.

What this does not establish

What the visual path does not supply

These continuous features feed text generation and visual reasoning. The audited path has no pixel decoder and no VQ codebook. Reusing it for reconstruction would require choosing a feature layer and training a decoder, then evaluating reconstruction separately from VQA.

The report and current implementation are separately pinned evidence. Undocumented training details remain unknown. The model-family selector below shows inspected configuration differences; it does not claim all similarly named checkpoints share identical weights.

• Report Figure 1 and architecture
• Qwen3.5 comparison

Check your understanding

Does DeepStack triple the visual context length?

No. It adds features at existing visual positions in early LLM layers.

Sources and next steps

• Qwen3-VL: from patches to DeepStack · 2511.21631v2
• Method / pinned implementation source
• Qwen3.5: audit the visual input path

Return to the learning map

Original teaching diagrams and snippets are identified separately from attributed paper figures. Paper reports, implementation observations, and analytical examples are distinguished. No research model runs in this page.