Autoencoders: a useful bottleneck

An interactive field guide

Can a small internal description retain what is needed to rebuild an image? An autoencoder learns that description and its reader together.

Before this article: Probability and tensors

What was already possible

A fixed compression method chooses its transform in advance. An autoencoder learns an encoder E and decoder D from examples. It is trained to make D(E(x)) close to x. The intermediate value z is often called a latent. This does not require a probabilistic model.

What changes, and why

The bottleneck forces a choice about what to preserve. It can restrict dimensions, add noise, impose quantization, or constrain the representation through a loss. Without such restrictions, an expressive network may learn an unhelpful copy operation.

Read from left to right. Select a module to connect its role, tensor shape, and calculation. Snippets are teaching pseudocode. Sources provide the paper method or audited implementation.

1. Encoder[B,3,H,W] → [B,C,h,w]2. Bottleneckz: constrained capacity3. Decoder[B,C,h,w] → [B,3,H,W]

Encoder

Downsampling changes spatial size, not just channel count.

z = E(x)

Bottleneck

All reconstruction information must pass through this interface.

z_small = constrain(z)

Decoder

Predict pixels and compare against the original input.

x_hat = D(z_small)
loss = mean((x_hat - x)**2)

Inspect the pinned source

$$\hat{x}=D_\theta(E_\phi(x)),\qquad L=\frac{1}{M}\sum_{j=1}^{M}(x_j-\hat{x}_j)^2$$

Work through the mechanism

The teaching example retains a limited set of signal frequencies. Increase the kept coefficients until the detail frequency becomes available. The sudden improvement comes from this explicit basis rule. A neural encoder learns its own nonlinear representation, so its curve need not have this shape.

Change a parameter, inspect the computed values, then compare this small example with the model described above.

Analytical teaching model · controls change the example, not a trained research model.

Kept basis coefficients

3

Input frequency

4

Reset

Grey: target signal. Green: decoded teaching signal. No trained image model is used.

Computed values for the current example

Quantity	Value
Toy signal MSE	0.045
Signal samples	48

Training is not sampling

A common image encoder uses convolutions with stride to reduce H and W while increasing channels. The decoder upsamples the features. A pixel MSE penalizes every mismatch, but averaging several plausible details may lower MSE while making an image look blurred.

After training, reconstruction is one encoder pass followed by one decoder pass. Randomly choosing a latent is a different operation: the decoder may never have seen that region. An ordinary autoencoder provides no guarantee that standard Gaussian samples will decode well.

What this does not establish

Reconstruction does not prove semantic understanding. A model can preserve texture while failing to group objects by meaning. VAE adds a probabilistic latent structure; VQ-VAE adds a discrete dictionary. Both keep the encoder–bottleneck–decoder skeleton.

Check your understanding

Why not sample arbitrary z after training an AE?

The training codes occupy a learned subset. Arbitrary values may lie outside it; no prior has been fitted to describe that subset.

Sources and next steps

• Method / pinned implementation source
• Variational autoencoders
• Evaluation
• Diffusion and flow matching
• Semantic representations
• ViT

Return to the learning map

Diagrams and snippets are teaching reconstructions. Paper reports, implementation observations, and analytical examples are distinguished. No research model runs in this page.