RAE-T2I: scale a shared feature space

An interactive field guide

A shared image representation can support separate understanding and generation paths. It need not imply one universal output head.

Before this article: RAE · Semantic representations · Diffusion and flow matching

What was already possible

The original RAE studies diffusion in pretrained feature spaces, including class-conditioned image generation. Text-to-image generation adds rich language conditions, larger data distributions, and demands such as rendering readable text. These can expose weaknesses that ImageNet reconstruction does not measure.

What changes, and why

Freeze a SigLIP-2 So400M patch-14 encoder. At 224² input it gives 256 features of width 1152. Train a ViT image decoder. For generation, an LLM with learned queries and an MLP connector conditions a flow DiT. For understanding, another projector sends features from the same frozen encoder to the LLM. The representation is shared; the task heads differ.

Read from left to right. Select a module to connect its role, tensor shape, and calculation. Snippets are teaching pseudocode. Sources provide the paper method or audited implementation.

1. Shared frozen features224² → [B,256,1152]2. Language-conditioned flowLLM queries → connector → DiT conditions3. Feature-to-pixel decoder[B,256,1152] → RGB

Shared frozen features

The understanding path uses a separate projector into an LLM.

z = frozen_siglip2(image)
understanding_tokens = understanding_projector(z)

Language-conditioned flow

The flow DiT generates a complete feature grid.

condition = connector(llm(text,learned_queries))
z_generated = flow_sampler(dit,condition)

Feature-to-pixel decoder

Train on data that covers the reconstruction details required by the task.

image = rae_decoder(z_generated)

Inspect the pinned source

$$\alpha=\sqrt{256C/4096},\qquad t^\prime=\frac{\alpha t}{1+(\alpha-1)t}$$

Work through the mechanism

The schedule demo uses m=256C and reference dimension n=4096. Set C=1152 and t=0.5: α=√72≈8.485 and shifted time is about 0.895. Under the data-at-zero convention, the shift places more probability on noisier training states. These arithmetic values illustrate the schedule, not a measured quality gain.

The decoder trains on about 73M examples, including web, synthetic, and text-rendering data. The paper finds that general web training alone is insufficient for reliable glyph recovery. A good natural-image rFID is therefore not a substitute for a dedicated text-reconstruction evaluation.

Change a parameter, inspect the computed values, then compare this small example with the model described above.

Analytical teaching model · controls change the example, not a trained research model.

Latent channels C

768

Base time t × 100

50

Reset

The straight path has known velocity. A learned model must estimate velocity; this demo does not measure its error.

Computed values for the current example

Quantity	Value
α = √(256C / 4096)	6.928
base t	0.500
t	0.874
x_t = (1 − t) × 2 + t × (−1)	-0.622
v = ε − x	-3

Training is not sampling

The decoder objective combines L1, LPIPS, Gram, and GAN terms. The generation study compares DiTs from 0.5B to 9.8B. Dimension-aware scheduling remains important, while some small-model improvements, such as a wide DDT head or decoder noise, become less useful at the larger text-to-image scale.

Reported evidence, not a local reproduction

Setting	Metric	Value	Interpretation
SigLIP-2 RAE: ImageNet / YFCC / Text	rFID ↓	0.435 / 0.702 / 1.621	Reconstruction table, datasets reported separately
FLUX VAE: same table columns	rFID ↓	0.288 / 0.410 / 0.638	RAE is not uniformly better at reconstruction

What this does not establish

The main comparison fixes 256 visual tokens, but RAE outputs 224² while VAE outputs 256²; feature width and pretraining cost also differ. This is a system comparison at fixed sequence length, not a completely matched rate/capacity study. Similar understanding results reflect the shared frozen understanding encoder, not proof that generation training caused a large understanding gain.

Open the derivation

The time shift rescales the ratio t/(1−t) by α. Solving this ratio equation yields the displayed fractional transform. It preserves 0 and 1 and moves intermediate times upward when α is larger than one.

$$\frac{t^\prime}{1-t^\prime}=\alpha\frac{t}{1-t}\quad\Longrightarrow\quad t^\prime=\frac{\alpha t}{1+(\alpha-1)t}$$

Check your understanding

Does sharing the representation make this a continuous image-token causal AR model?

No. The image generator is still a diffusion/flow DiT. Representation sharing and generative factorization are separate design choices.

Sources and next steps

• RAE-T2I · 2601.16208v1
• Method / pinned implementation source

Return to the learning map

Diagrams and snippets are teaching reconstructions. Paper reports, implementation observations, and analytical examples are distinguished. No research model runs in this page.