Qwen3.5: audit the visual input path

An interactive field guide

Native multimodal training does not tell you which visual modules are present. Follow the pinned implementation.

Before this article: Visual tokenizers in multimodal models · Qwen3-VL: from patches to DeepStack

What was already possible

Native multimodal training still uses a vision encoder

The official model card describes early fusion of multimodal tokens during training. This describes when modalities are trained together. It does not imply an encoder-free architecture. The same card calls the model a causal language model with a vision encoder.

We inspect 2B, 9B and 397B-A17B checkpoints. The 9B example has a 27-block, width-1152 visual Transformer and width-4096 language embeddings. The smaller 2B vision tower has 24 blocks and width 1024, with output width 2048. Architecture dimensions do not identify the training dataset or initialization checkpoint.

What changes, and why

Read from left to right. Select a module to connect its role, tensor shape, and calculation. Snippets are teaching pseudocode. Sources provide the paper method or audited implementation.

1. Follow the visual encoder, not the text-model labelSee the stage equations below2. The merger connects visual width to language widthSee the stage equations below3. A checked difference from Qwen3-VLSee the stage equations below

Follow the visual encoder, not the text-model label

The vision path uses 2×16×16 RGB tubelets, a learned Conv3d embedding, interpolated absolute positions, spatial rotary positions and vision Transformer blocks. The text backbone combines different attention mechanisms, but that does not make the vision tower a linear-attention encoder. Inspect the two modules separately.

z = patch_embed(pixel_values)
z += interpolate_absolute_positions(grid_thw)
for block in vision_blocks:
    z = block(z, rotary_positions)

Training state: Static source inspection; training state is discussed below.

$$Z=\operatorname{ViT}(\operatorname{PatchEmbed}(X)+E_{\rm abs};E_{\rm rotary})$$

Step-by-step explanation: Follow the visual encoder, not the text-model label

The merger connects visual width to language width

The final merger applies LayerNorm, packs spatial 2×2 groups and runs a GELU MLP. For a prepared 512×512 image, 1024×1152 encoder features become 256×4608 concatenated vectors, then 256×4096 language-width features. Concatenation preserves the four vectors as separate coordinates before the learned projection.

merged = vision_output.pooler_output
image_mask = locate_image_placeholders(input_ids)
inputs_embeds = inputs_embeds.masked_scatter(image_mask, merged)

Training state: Static source inspection; training state is discussed below.

$$[1024,1152]\rightarrow[256,4\cdot1152]\rightarrow[256,4096]$$

Step-by-step explanation: The merger connects visual width to language width

A checked difference from Qwen3-VL

All three inspected Qwen3.5 configurations have an empty deepstack_visual_indexes list. Configuration alone is not sufficient evidence, so we also trace the implementation. Qwen3_5VisionModel returns the final merger output and the multimodal forward path inserts it at input positions. It does not return the Qwen3-VL DeepStack feature list or add it after early LLM blocks.

# Inspected Qwen3.5 path
vision = visual(pixel_values, grid_thw)
llm_input[image_positions] = vision.pooler_output
# No intermediate DeepStack additions in this path.

Training state: Static source inspection; training state is discussed below.

Step-by-step explanation: A checked difference from Qwen3-VL

Inspect the pinned source

$$V=M(E(X)),\qquad H^0_{\mathcal I}=V$$

Follow the visual encoder, not the text-model label

Teaching diagram · Qwen3.5: a dedicated visual input path Read each named row from left to right. Rows show distinct pathways or comparison cases. · Method sourceSelect the image to inspect it at full size.

The vision path uses 2×16×16 RGB tubelets, a learned Conv3d embedding, interpolated absolute positions, spatial rotary positions and vision Transformer blocks. The text backbone combines different attention mechanisms, but that does not make the vision tower a linear-attention encoder. Inspect the two modules separately.

For a still image, temporal duplication creates one time group. The published processor uses a shared Qwen image-processing implementation. Its class name may contain an older model name; that is implementation reuse, not evidence that the checkpoint is an older model.

$$Z=\operatorname{ViT}(\operatorname{PatchEmbed}(X)+E_{\rm abs};E_{\rm rotary})$$

z = patch_embed(pixel_values)
z += interpolate_absolute_positions(grid_thw)
for block in vision_blocks:
    z = block(z, rotary_positions)

• Pinned vision model — Qwen3_5VisionModel

A checked difference from Qwen3-VL

Teaching diagram · Inspect the feature routes, not only the family name Read each named row from left to right. Rows show distinct pathways or comparison cases. · Method sourceSelect the image to inspect it at full size.

All three inspected Qwen3.5 configurations have an empty deepstack_visual_indexes list. Configuration alone is not sufficient evidence, so we also trace the implementation. Qwen3_5VisionModel returns the final merger output and the multimodal forward path inserts it at input positions. It does not return the Qwen3-VL DeepStack feature list or add it after early LLM blocks.

This is a code observation for the pinned checkpoints and implementation. It does not show that removing DeepStack caused any reported quality gain. The two families also differ in language models, data and training. A causal claim would need a matched ablation.

# Inspected Qwen3.5 path
vision = visual(pixel_values, grid_thw)
llm_input[image_positions] = vision.pooler_output
# No intermediate DeepStack additions in this path.

Work through the mechanism

The merger connects visual width to language width

The final merger applies LayerNorm, packs spatial 2×2 groups and runs a GELU MLP. For a prepared 512×512 image, 1024×1152 encoder features become 256×4608 concatenated vectors, then 256×4096 language-width features. Concatenation preserves the four vectors as separate coordinates before the learned projection.

The visual vectors replace image-placeholder embeddings through a mask. A repeated placeholder ID therefore marks positions; the image-specific content is carried by the inserted vectors. There is no nearest-neighbor codebook lookup in this inspected path.

$$[1024,1152]\rightarrow[256,4\cdot1152]\rightarrow[256,4096]$$

merged = vision_output.pooler_output
image_mask = locate_image_placeholders(input_ids)
inputs_embeds = inputs_embeds.masked_scatter(image_mask, merged)

Trace pixels to visual tokens

Shape arithmetic, without model inference. Image mode follows the pinned resize rules. Here a Qwen budget sets the pixel-area limit corresponding to a maximum visual-token count; it is not a billing estimate.

Height (pixels)Width (pixels)Sampled frames (video mode)

Stage / quantity	Result
Resized height	512
Resized width	512
Temporal groups	1
Repeated frames	1
Model patch-grid rows	32
Model patch-grid columns	32
Processor spatial units × time groups	1024
Model patch positions	1024
Effective visual tokens	256
LLM visual feature shape	[256, 4096]
Continuous scalar values	1048576

2×16×16 patches → ViT → 2×2 concat + MLP → LLM

27 vision blocks · width 1152 → 4096

Input: visual embeddings occupy image positions; text embeddings remain.

Pinned configuration for this variant

Reset

Effective tokens exclude padding slots, image boundaries, newlines, timestamp text and other text. Video mode does not sample frames or allocate video budgets: frame sides must be multiples of 32 for Qwen or 48 for Gemma. Repeating a still image satisfies the temporal patch interface; it adds no motion information.

Inspected checkpoint configurations

Checked: 2026-11-03. Each row links to a pinned version. Matching dimensions do not imply matching weights. The 12B patch column denotes the pixel block received by the model.

Variant	Model patch	Vision blocks / width	Spatial operation	LLM width	DeepStack zero-based blocks
Qwen3.5-2B	16×16	24 / 1024	2×2 concat + MLP	2048	—
Qwen3.5-9B	16×16	27 / 1152	2×2 concat + MLP	4096	—
Qwen3.5-397B-A17B	16×16	27 / 1152	2×2 concat + MLP	4096	—

“—” means no DeepStack injection in the inspected path, not an absence of visual information. All branches pass continuous vectors; text vocabulary sizes are not treated as visual codebooks.

Training and inference

Keep training claims separate from source observations

The model card reports a unified vision–language foundation and broad multimodal results. It does not provide, in the materials audited here, a complete vision-only pretraining recipe or an unambiguous initialization checkpoint for the visual tower. Those fields remain undisclosed in this survey. Similarity to Qwen3-VL does not fill them in.

The Qwen3.5 repository entry now redirects to a newer family. This article therefore cites fixed Qwen3.5 model revisions and source commits. It does not silently use a newer README or an Omni model report as evidence for these checkpoints.

• Pinned official model card

What this does not establish

What a reconstruction experiment would still need

Early fusion does not create a pixel decoder. A reconstruction study must select pre-merger or post-merger features, define how a decoder handles their spatial layout, and train against images. Text-generation success is not a reconstruction metric.

Use the calculator to inspect sequence and width changes, including repeated terminal video frames. It counts shapes only. It cannot rank Qwen3.5 against Qwen3-VL, estimate missing training cost, or predict how much pixel detail survives the merger.

Check your understanding

Can the visual initialization be inferred from matching layer counts?

No. Architecture and weight provenance are different evidence.

Sources and next steps

• Method / pinned implementation source

Return to the learning map

Original teaching diagrams and snippets are identified separately from attributed paper figures. Paper reports, implementation observations, and analytical examples are distinguished. No research model runs in this page.