Gemma 4: encoders, pooling and an encoder-free branch

An interactive field guide

Compare five variants without hiding the 12B architecture behind a family name.

Before this article: Visual tokenizers in multimodal models

What was already possible

One family, three visual configurations

E2B and E4B use a roughly 150M vision encoder with 16 blocks and width 768. The 31B and 26B-A4B variants use a roughly 550M encoder with 27 blocks and width 1152. The 12B variant removes the separate vision Transformer. Treating all five as one visual tokenizer would hide the main architectural difference.

The text vocabulary has about 262K entries. That is a text-tokenizer property, not the number of visual codes. Likewise, quantization-aware training for compact weights does not imply that image representations use a discrete VQ codebook.

What changes, and why

Read from left to right. Select a module to connect its role, tensor shape, and calculation. Snippets are teaching pseudocode. Sources provide the paper method or audited implementation.

1. Encode small patches before spatial averagingSee the stage equations below2. A token budget is an upper bound, not an exact countSee the stage equations below3. 12B projects larger pixel blocks without a vision towerSee the stage equations below

Encode small patches before spatial averaging

The encoder variants use 16×16 RGB patches. A learned projection and two-dimensional absolute positions feed bidirectional visual Transformer blocks with axial 2D rotary positions. After image-wide processing, groups of 3×3 spatial features are averaged, normalized/scaled and projected to the language width.

z = vision_transformer(patches16, positions_xy)
z = spatial_average_pool(z, kernel=3)
valid_z = remove_padding(z, valid_mask)
visual = language_projection(valid_z)

Training state: Static source inspection; training state is discussed below.

$$\bar z_{a,b}=\frac{1}{9}\sum_{i=0}^{2}\sum_{j=0}^{2}z_{3a+i,3b+j},\qquad V=\operatorname{Project}(\operatorname{Normalize}(\bar Z))$$

Step-by-step explanation: Encode small patches before spatial averaging

A token budget is an upper bound, not an exact count

The supported image budgets are 70, 140, 280, 560 and 1120 soft tokens. Set the pooled spatial unit m=16×3=48 pixels. The ideal target area is budget×48². Scale both original sides by the square root of target area divided by original area, then round each down to a multiple of 48.

H2, W2 = aspect_ratio_resize(H, W, max_soft_tokens=B)
patch_count = (H2 // 16) * (W2 // 16)
valid_soft_tokens = patch_count // 9

Training state: Static source inspection; training state is discussed below.

$$f=\sqrt{\frac{B(48)^2}{HW}},\quad H^\prime=48\left\lfloor\frac{fH}{48}\right\rfloor,\quad W^\prime=48\left\lfloor\frac{fW}{48}\right\rfloor,\quad N=\frac{H^\prime W^\prime}{48^2}\le B$$

Step-by-step explanation: A token budget is an upper bound, not an exact count

12B projects larger pixel blocks without a vision tower

The 12B model takes 48×48×3=6912 values per pixel block and uses a lightweight embedding path. Its configuration still lists processor patch_size=16 and pooling_kernel_size=3, but model_patch_size=48. The processor repacks 3×3 small pixel patches into a larger block. Repacking pixels is not averaging encoded features.

raw48 = repack_pixel_patches(raw16, group_side=3)
z = ln2(dense(ln1(raw48)))
z = ln3(z + position_x + position_y)
visual = multimodal_projection(rms_norm(z))

Training state: Static source inspection; training state is discussed below.

$$e_i=\operatorname{LN}\!\left(\operatorname{LN}(W\operatorname{LN}(x_i)+b)+E_x[x_i^{\rm pos}]+E_y[y_i^{\rm pos}]\right)$$

Step-by-step explanation: 12B projects larger pixel blocks without a vision tower

Inspect the pinned source

$$N_{\rm soft}=\frac{H^\prime W^\prime}{48^2}$$

Encode small patches before spatial averaging

Teaching diagram · Gemma 4: encode before pooling Read each named row from left to right. Rows show distinct pathways or comparison cases. · Method sourceSelect the image to inspect it at full size.

The encoder variants use 16×16 RGB patches. A learned projection and two-dimensional absolute positions feed bidirectional visual Transformer blocks with axial 2D rotary positions. After image-wide processing, groups of 3×3 spatial features are averaged, normalized/scaled and projected to the language width.

Pooling reduces nine feature positions to one after the visual tower has already exchanged information. It is not equivalent to averaging raw pixels before the encoder. The LLM widths are variant-specific: 1536 for E2B, 2560 for E4B, 5376 for 31B and 2816 for 26B-A4B.

$$\bar z_{a,b}=\frac{1}{9}\sum_{i=0}^{2}\sum_{j=0}^{2}z_{3a+i,3b+j},\qquad V=\operatorname{Project}(\operatorname{Normalize}(\bar Z))$$

z = vision_transformer(patches16, positions_xy)
z = spatial_average_pool(z, kernel=3)
valid_z = remove_padding(z, valid_mask)
visual = language_projection(valid_z)

• Pinned encoder implementation

12B projects larger pixel blocks without a vision tower

Teaching diagram · Gemma 4 12B: pixel regrouping is not feature pooling Read each named row from left to right. Rows show distinct pathways or comparison cases. · Method sourceSelect the image to inspect it at full size.

The 12B model takes 48×48×3=6912 values per pixel block and uses a lightweight embedding path. Its configuration still lists processor patch_size=16 and pooling_kernel_size=3, but model_patch_size=48. The processor repacks 3×3 small pixel patches into a larger block. Repacking pixels is not averaging encoded features.

The inspected implementation applies LayerNorm, a dense projection, LayerNorm, factorized x/y position embeddings, another LayerNorm, then an RMSNorm and a multimodal projection. The output width is 3840. The report’s “single large matmul” describes the main visual projection; the full interface includes these additional operations.

$$e_i=\operatorname{LN}\!\left(\operatorname{LN}(W\operatorname{LN}(x_i)+b)+E_x[x_i^{\rm pos}]+E_y[y_i^{\rm pos}]\right)$$

raw48 = repack_pixel_patches(raw16, group_side=3)
z = ln2(dense(ln1(raw48)))
z = ln3(z + position_x + position_y)
visual = multimodal_projection(rms_norm(z))

• Pinned encoder-free implementation — Gemma4UnifiedVisionEmbedder

Work through the mechanism

A token budget is an upper bound, not an exact count

The supported image budgets are 70, 140, 280, 560 and 1120 soft tokens. Set the pooled spatial unit m=16×3=48 pixels. The ideal target area is budget×48². Scale both original sides by the square root of target area divided by original area, then round each down to a multiple of 48.

Integer grids can change the aspect ratio slightly and leave part of the budget unused. In report Figure 2, a 572×1024 image (width×height) with an illustrative budget of 10 becomes 96×192: 72 small patches and eight soft tokens. Ten illustrates the algorithm; it is not an accepted budget in the public processor. The code also handles extremely narrow inputs when a rounded side would become zero.

$$f=\sqrt{\frac{B(48)^2}{HW}},\quad H^\prime=48\left\lfloor\frac{fH}{48}\right\rfloor,\quad W^\prime=48\left\lfloor\frac{fW}{48}\right\rfloor,\quad N=\frac{H^\prime W^\prime}{48^2}\le B$$

H2, W2 = aspect_ratio_resize(H, W, max_soft_tokens=B)
patch_count = (H2 // 16) * (W2 // 16)
valid_soft_tokens = patch_count // 9

• Resize implementation and edge handling

Original figure: the budget need not be filled

Original Figure 2 · Gemma 4 Width×height is 572×1024 → 96×192. Budget 10 is illustrative, not a processor option. · Gemma Team et al. · CC BY 4.0 · PDF rendered and converted to WebP · CC BY 4.0Select the image to inspect it at full size.

The figure uses a budget of 10 to illustrate rounding. The interactive model presets use only the public supported image budgets.

Trace pixels to visual tokens

Shape arithmetic, without model inference. Image mode follows the pinned resize rules. Here a Qwen budget sets the pixel-area limit corresponding to a maximum visual-token count; it is not a billing estimate.

Height (pixels)Width (pixels)Sampled frames (video mode)

Stage / quantity	Result
Resized height	768
Resized width	768
Temporal groups	1
Repeated frames	0
Model patch-grid rows	48
Model patch-grid columns	48
Processor spatial units × time groups	2304
Model patch positions	2304
Effective visual tokens	256
LLM visual feature shape	[256, 5376]
Continuous scalar values	1376256

16×16 patches → ViT → 3×3 average pool → LLM

27 vision blocks · width 1152 → 5376

Input: visual embeddings occupy image positions; text embeddings remain.

Pinned configuration for this variant

Reset

Effective tokens exclude padding slots, image boundaries, newlines, timestamp text and other text. Video mode does not sample frames or allocate video budgets: frame sides must be multiples of 32 for Qwen or 48 for Gemma. Repeating a still image satisfies the temporal patch interface; it adds no motion information.

Inspected checkpoint configurations

Checked: 2026-11-03. Each row links to a pinned version. Matching dimensions do not imply matching weights. The 12B patch column denotes the pixel block received by the model.

Variant	Model patch	Vision blocks / width	Spatial operation	LLM width	DeepStack zero-based blocks
gemma-4-E2B-it	16×16	16 / 768	3×3 average	1536	—
gemma-4-E4B-it	16×16	16 / 768	3×3 average	2560	—
gemma-4-12B-it	48×48	0 / 3840	Pixel projection	3840	—
gemma-4-31B-it	16×16	27 / 1152	3×3 average	5376	—
gemma-4-26B-A4B-it	16×16	27 / 1152	3×3 average	2816	—

“—” means no DeepStack injection in the inspected path, not an absence of visual information. All branches pass continuous vectors; text vocabulary sizes are not treated as visual codebooks.

Training and inference

Compare pathways under their own training conditions

The report states that the separate vision encoders are frozen during pretraining, while the encoder-free 12B model is trained from scratch. The public inference class does not by itself enforce the historical optimizer schedule. A reproduction needs explicit parameter groups and the matching training recipe.

A 48-pixel output unit appears in both routes. In one, small patches interact in a pretrained tower before pooling; in the other, the language backbone must process directly projected larger blocks. Equal output counts do not mean equal visual computation or equal detail retention.

What this does not establish

What remains unknown or outside this comparison

The inspected report does not justify naming one exact SigLIP initialization checkpoint for every encoder variant. We preserve this uncertainty. The comparison concerns visual input and text output, not an image decoder. Audio and multi-token prediction are outside this page’s audit.

For reconstruction research, distinguish pre-pooling patch features, pooled features and the encoder-free projected blocks. They have different training histories and spatial interfaces. Training a new pixel decoder on any one of them is a separate experiment.

Check your understanding

Does processor patch_size=16 prove that Gemma 4 12B has a patch-16 vision Transformer?

No. The processor repacks pixels into 48×48 blocks; the model uses a projection path without that vision tower.

Sources and next steps

• Gemma 4: encoders, pooling and an encoder-free branch · 2607.02770v2
• Method / pinned implementation source

Return to the learning map

Original teaching diagrams and snippets are identified separately from attributed paper figures. Paper reports, implementation observations, and analytical examples are distinguished. No research model runs in this page.