CausalFusion: two axes of generation

An interactive field guide

Generate a subset, denoise it, then let later subsets depend on its clean values.

Before this article: DiT · MAR · Attention and autoregression

What was already possible

DiT denoises a whole latent grid together. MAR computes context then runs a small per-token diffusion head. CausalFusion instead lets the main Transformer participate in iterative denoising of the current token subset while enforcing causal dependence between subsets.

What changes, and why

There is an outer axis for autoregressive subsets and an inner axis for diffusion time. A generalized causal mask allows the current noisy subset to use earlier clean subsets while preventing earlier positions from reading current or future targets. Noise levels and group membership must be part of the training design.

Read from left to right. Select a module to connect its role, tensor shape, and calculation. Snippets are teaching pseudocode. Sources provide the paper method or audited implementation.

1. Choose subsets and timestokens → ordered groups with noise levels2. Generalized causal attentionearlier clean + current noisy → prediction3. Nested generationAR group loop × diffusion loop

Choose subsets and times

Training samples a structured denoising problem.

groups = sample_partition(tokens)
noisy = corrupt_current_groups(groups,times)

Generalized causal attention

Earlier positions cannot read later targets.

mask = group_causal_mask(groups)
v = transformer(noisy,condition=earlier_clean,mask=mask)

Nested generation

The main denoising backbone runs inside the group loop.

for group in groups:
    for t,t_next in schedule:
        state[group] = denoise_step(transformer,state,group,t,t_next)

Inspect the pinned source

$$p(x)=\prod_g p_\theta(x_{G_g}\mid x_{G_{<g}}),\quad p_\theta(x_{G_g}\mid\cdot)\ \text{uses iterative denoising}$$

Work through the mechanism

Increase group size in the demo: fewer outer rounds are needed. Increase inner steps: each group needs more denoising calls. Unlike MAR, these repeated calls involve the main Transformer. The same group-step count therefore need not have the same cost in both models.

A schedule with one large group resembles whole-grid denoising; many small groups add more autoregressive conditioning. Moving between these endpoints changes the prediction problem. A model trained mainly on one endpoint need not work well at another.

Change a parameter, inspect the computed values, then compare this small example with the model described above.

Analytical teaching model · controls change the example, not a trained research model.

Tokens generated per group

4

Inner denoising steps

4

Reset

Numbers show generation groups, not token values. Network size and caching still determine runtime.

Computed values for the current example

Quantity	Value
Outer groups	4
Inner steps per group	4
Group × step evaluations	16

Training is not sampling

Uniformly sampling the number of AR rounds can overemphasize easy tasks with little new content per prediction. The paper changes round sampling and increases early-step loss weights. Its multimodal experiment mixes text-before-image and image-before-text examples for generation and captioning, with a small text-loss weight.

What this does not establish

The paper reports serious degradation for some training/inference schedule mismatches. Position encodings also vary by experiment: sinusoidal, learned, or a text/image combination. Do not summarize all configurations as RoPE. Captioning and text-to-image evidence does not establish open-domain VQA or general reasoning.

Check your understanding

Can we freely reduce outer rounds after training without checking quality?

No. That changes how much content is predicted from each context and may move outside the trained schedule distribution.

Sources and next steps

• CausalFusion · 2412.12095v2
• Method / pinned implementation source

Return to the learning map

Diagrams and snippets are teaching reconstructions. Paper reports, implementation observations, and analytical examples are distinguished. No research model runs in this page.