EOSTok: make prediction preserve the image

An interactive field guide

An easier token prediction problem is not always a better image model. The tokenizer could simply discard information.

Before this article: TiTok · Attention and autoregression · Semantic representations

What was already possible

Two-stage systems train the tokenizer first, then train an autoregressive prior over fixed codes. End-to-end training could coordinate them, but a next-token loss alone rewards predictability. A collapsing or smaller effective vocabulary can lower prediction difficulty while harming image fidelity.

What changes, and why

EOSTok jointly trains a 1D semantic tokenizer and causal image generator. IBQ uses softmax/straight-through assignments to connect gradients through quantization. Autoregressive Prediction Reconstruction, or APR, decodes teacher-forced predicted tokens to pixels and applies L2 plus LPIPS. DINOv2 alignment targets spatial hidden features, rather than forcing global latent slots to be pixel locations.

Read from left to right. Select a module to connect its role, tensor shape, and calculation. Snippets are teaching pseudocode. Sources provide the paper method or audited implementation.

1. Differentiable assignments[B,K,D] → code probabilities and IDs2. Teacher-forced predictiontrue prefix → next-code probabilities3. APR pixel supervisionpredicted codes → reconstructed pixels

Differentiable assignments

IBQ connects encoder, codebook, and generator gradients.

assignment = softmax(code_logits)
zq = straight_through_assignment(assignment) @ codebook

Teacher-forced prediction

Input embeddings can be formed from assignment probabilities.

inputs = assignment @ token_embeddings
predicted = causal_generator(inputs[:,:-1])

APR pixel supervision

Keep context assumptions visible when interpreting gradients.

prediction_image = decoder(predicted_codes)
loss_apr = l2(prediction_image,image)+lpips(prediction_image,image)

Inspect the pinned source

$$\hat x=2p,\quad L=(2p-x)^2,\quad \frac{dL}{dp}=4(2p-x)$$

Work through the mechanism

The demo uses two codes that decode to scalar values 0 and 2. A soft probability p gives expected output 2p. Pixel error changes with the predicted probability and produces a gradient. This isolates why reconstruction can supervise a predicted code distribution. It is a teaching relaxation, not the full IBQ estimator or free-running APR training.

Teacher forcing supplies true previous tokens when predicting the next one. APR therefore evaluates predictions under those contexts. It is not an unbiased gradient through a complete freely sampled image trajectory. The distinction matters when interpreting what end-to-end training actually optimizes.

Change a parameter, inspect the computed values, then compare this small example with the model described above.

Analytical teaching model · controls change the example, not a trained research model.

Probability of code 1

0.5

Pixel target

1

Reset

Differentiable soft expectation is a teaching relaxation. It is not free-running discrete generation or the complete APR implementation.

Computed values for the current example

Quantity	Value
Soft decoded prediction 2p	1.000
Teaching pixel MSE	0.000
d MSE / dp	0.000

Training is not sampling

In the encoder, image patches interact bidirectionally; queries use causal interaction, can read all patches, and cannot be read back by patches. The tokenizer and generator use learned absolute positions for the 1D latent sequence. APR adds another decoder forward pass, while semantic alignment adds a pretrained teacher dependency.

Reported evidence, not a local reproduction

Setting	Metric	Value	Interpretation
EOSTok-L, 50 epochs, baseline	rFID ↓ / gFID ↓ / AR accuracy ↑	1.09 / 3.82 / 11.8%	End-to-end ablation table; no guidance
Naive end-to-end	rFID ↓ / gFID ↓ / AR accuracy ↑	4.92 / 8.01 / 30.2%	Higher accuracy, worse images
Add APR	rFID ↓ / gFID ↓ / AR accuracy ↑	1.02 / 3.32 / 11.9%	Use table baseline accuracy; prose differs slightly

What this does not establish

Use the matched 50-epoch ablation below to see the failure of naive joint training. Higher AR accuracy is not sufficient: the easiest codes can give worse images. Main guided results also use AutoGuidance for some sizes, which must not be treated as ordinary CFG with identical cost.

Check your understanding

Why can AR accuracy rise while image quality falls?

The tokenizer may make its codes less informative and easier to predict. Pixel reconstruction and generated-image metrics test information that token accuracy does not.

Sources and next steps

• EOSTok · 2605.00503v2
• Method / pinned implementation source

Return to the learning map

Diagrams and snippets are teaching reconstructions. Paper reports, implementation observations, and analytical examples are distinguished. No research model runs in this page.