AToken: connect semantic, continuous, and discrete views

An interactive field guide

A unified tokenizer need not expose exactly the same representation to every downstream task.

Before this article: RAE · RoPE-ViT · UniTok

What was already possible

A semantic feature can support recognition but carry a large continuous payload. A compressed latent can support reconstruction. Discrete codes make categorical generation possible. AToken connects these representations and extends the visual interface beyond fixed-size images.

What changes, and why

Start from a SigLIP2 visual tower. Introduce low-dimensional continuous latents and optional quantization, with a progressive training curriculum. Transformer processing uses sparse coordinates (t,x,y,z) for image, video, and 3D inputs. Learned 2D absolute positions are retained alongside added 4D RoPE to preserve pretrained spatial structure.

Read from left to right. Select a module to connect its role, tensor shape, and calculation. Snippets are teaching pseudocode. Sources provide the paper method or audited implementation.

1. Pretrained semanticsimage patches → semantic embeddings2. Compact representationssemantic → continuous latent → optional IDs3. Position-aware reconstructiontokens + (t,x,y,z) → output patches

Pretrained semantics

Preserve useful SigLIP2 initialization.

semantic = pretrained_visual_tower(patches,ape)

Compact representations

Task interfaces may access different stages.

latent = compress(semantic)
codes = quantize(latent) if discrete_mode else latent

Position-aware reconstruction

Add multi-axis rotary phases while retaining the specified APE.

phase = sum(freq[axis]*coord[axis] for axis in axes)
output = decoder(tokens,rotary_phase=phase)

Inspect the pinned source

$$\phi_j(p)=\omega_{t,j}t+\omega_{x,j}x+\omega_{y,j}y+\omega_{z,j}z$$

Work through the mechanism

The coordinate demo keeps a fixed relative image displacement in x and y. Change the relative time and depth to see their contribution to one rotary phase. For a still image, t and z can both be zero. The four axes describe input coordinates; they do not mean each image has four nonzero spatial dimensions.

The illustrated frequencies are chosen for arithmetic, not copied model parameters. The real representation also contains content and pretrained absolute positions. This makes AToken a useful counterexample to the rule “adding RoPE requires removing all APE.”

Change a parameter, inspect the computed values, then compare this small example with the model described above.

Analytical teaching model · controls change the example, not a trained research model.

Relative time Δt

0

Relative depth Δz

0

Reset

Invented frequencies isolate the four-axis phase. Still images can keep time and depth differences at zero.

Computed values for the current example

Quantity	Value
Phase (radians)	1.500
Unit-pair dot product	0.071

Training is not sampling

Training uses perceptual and Gram-matrix objectives in place of adversarial training. A Gram matrix compares feature correlations, so it can constrain texture statistics without a learned discriminator. No GAN still leaves dependence on perceptual features. Different tasks may read different representation stages.

What this does not establish

This article follows the paper and author page; no reproducible official code commit was verified. Do not combine the best semantic score and best compression result as if they used one identical bottleneck. For an image-only implementation, keep image-grid coordinates distinct from global slot identities before expanding to video or 3D.

Check your understanding

Why can an image use a four-axis position interface?

Unused axes can be fixed to zero. The interface can represent multiple modalities without requiring every modality to vary along every axis.

Sources and next steps

• AToken · 2509.14476v2
• Method / pinned implementation source

Return to the learning map

Diagrams and snippets are teaching reconstructions. Paper reports, implementation observations, and analytical examples are distinguished. No research model runs in this page.