A small language for generative models

An interactive field guide

A generator must choose among many possible images. Before studying its network, separate a value, a distribution, and a conditional distribution.

What was already possible

A tensor is an array with a shape. B is the number of examples, C the number of channels, and H and W the image height and width. An image batch often has shape [B,C,H,W]. A sequence often has shape [B,N,D]: N positions, D numbers at each position. A token can be an integer ID or a vector.

What changes, and why

A latent z is an internal representation. An encoder maps an observed image x into z. A decoder maps z back to an image or to a distribution over images. A prior describes which z values the model expects before it sees x. Reconstruction starts from an observed image; generation may start from a prior sample.

Read from left to right. Select a module to connect its role, tensor shape, and calculation. Snippets are teaching pseudocode. Sources provide the paper method or audited implementation.

1. Observex: [B,C,H,W]2. Representz: [B,N,D]3. Modelp(z_i | context)

Observe

The input is a value, not a density.

x = batch["image"]

Represent

A learned map changes the representation.

z = encoder(x)

Model

The head assigns probabilities or defines a density sampler.

next_z = sample(head(context))

Inspect the pinned source

$$p(z_1,\ldots,z_N)=\prod_{i=1}^{N}p(z_i\mid z_{<i})$$

Work through the mechanism

Use the controls to change two binary probabilities. If P(first=1)=0.6 and P(second=1|first=1)=0.7, the joint event has probability 0.42. The remaining event with first=1 has probability 0.18. These add to 0.6. The conditional distribution must still sum to one.

The chain rule does not assume that positions are independent. It only chooses an order for writing the joint distribution. A causal model learns these conditional distributions with a mask. Continuous variables use densities rather than probabilities of exact values; their densities need not be less than one.

Change a parameter, inspect the computed values, then compare this small example with the model described above.

Analytical teaching model · controls change the example, not a trained research model.

P(first = 1)

0.6

P(second = 1 | first = 1)

0.7

Reset

The second probability is conditional. Multiplying marginal probabilities would assume independence.

Computed values for the current example

Quantity	Value
P(1,1) = P(1) P(1|1)	0.420
P(1,0) = P(1) [1 − P(1|1)]	0.180

Training is not sampling

A loss is a training objective, not automatically a measure of image quality. Negative log-likelihood rewards probability assigned to observed data. Squared error corresponds to a Gaussian likelihood only with stated variance assumptions. Averaging over pixels versus summing changes its scale.

What this does not establish

A smaller tensor is not necessarily a better representation. Count positions, channels, precision, and the cost of the model that consumes them. Do not compare 256 integer codes with 256 vectors as if both carry the same information.

Check your understanding

Is a continuous token already a probability distribution?

No. It is a value. A model must separately define how possible values are distributed.

Sources and next steps

• Method / pinned implementation source
• Autoencoders
• Variational autoencoders
• Attention and autoregression
• Evaluation
• Diffusion and flow matching

Return to the learning map

Diagrams and snippets are teaching reconstructions. Paper reports, implementation observations, and analytical examples are distinguished. No research model runs in this page.