CLIP, OpenCLIP, SigLIP and SigLIP 2

An interactive field guide

Follow two encoders into one similarity matrix, then separate the changes in loss, data and training.

Before this article: ViT · Semantic representations

What was already possible

Natural language changes the supervision interface

A fixed classifier has one learned weight vector for each training class. CLIP instead learns an image encoder and a text encoder from paired images and descriptions. It can later encode a new list of class descriptions as classifier weights. Earlier work already connected language and images; CLIP combines a simple contrastive objective with broad paired data and model scaling. Its image encoder can be a modified ResNet or a ViT. The ViT branch is the main drawing here, not a claim that every CLIP model is a ViT.

What changes, and why

Read from left to right. Select a module to connect its role, tensor shape, and calculation. Snippets are teaching pseudocode. Sources provide the paper method or audited implementation.

1. Natural language changes the supervision interfaceimage batch and text batch → paired embeddings2. Two towers meet in a shared embedding spaceU, V: B × d; no tower weight sharing3. A batch becomes B×B matching questionsscores: B × B; positive pairs on diagonal4. Zero-shot classification reuses the text towerimage vector: d; C candidate text vectors

Natural language changes the supervision interface

A fixed classifier has one learned weight vector for each training class. CLIP instead learns an image encoder and a text encoder from paired images and descriptions. It can later encode a new list of class descriptions as classifier weights. Earlier work already connected language and images; CLIP combines a simple contrastive objective with broad paired data and model scaling. Its image encoder can be a modified ResNet or a ViT. The ViT branch is the main drawing here, not a claim that every CLIP model is a ViT.

# Teaching pseudocode; not a complete training implementation.
images, captions = paired_batch()
image_features = image_encoder(images)
text_features = text_encoder(captions)

Training state: Training and inference roles are separated below.

Step-by-step explanation: Natural language changes the supervision interface

Two towers meet in a shared embedding space

The image tower pools an image feature, then projects it to dimension d. The text tower tokenizes a description, applies a causal Transformer, selects its end-of-text representation and projects to the same d. Normalize both vectors to unit length. Their dot product is then cosine similarity. Shared output dimension does not mean shared encoder weights. Original CLIP uses a learned positive logit scale. It has no cross-attention between image patches and words in this dual-encoder path.

# Teaching pseudocode; not a complete training implementation.
U = l2_normalize(image_projection(image_features))
V = l2_normalize(text_projection(eos_features))
S = exp(logit_scale) * U @ V.T

Training state: Training and inference roles are separated below.

$$u_i=\frac{W_vf_v(x_i)}{\|W_vf_v(x_i)\|_2},\quad v_j=\frac{W_tf_t(t_j)}{\|W_tf_t(t_j)\|_2},\quad S_{ij}=e^a u_i^Tv_j$$

Step-by-step explanation: Two towers meet in a shared embedding space

A batch becomes B×B matching questions

Arrange B paired images and texts in the same order. Row i compares image i with every text; column j compares text j with every image. Diagonal entries are the paired positives. Image-to-text cross-entropy normalizes each row; text-to-image cross-entropy normalizes each column. Average both directions. Increasing one negative score changes the softmax probabilities of other entries in that row or column. Unpaired entries are treated as negatives even when their content is actually compatible, so noisy and repeated captions matter.

# Teaching pseudocode; not a complete training implementation.
labels = arange(batch_size)
row_loss = cross_entropy(S, labels)
column_loss = cross_entropy(S.T, labels)
loss = (row_loss + column_loss) / 2

Training state: Training and inference roles are separated below.

$$\mathcal L_{CLIP}=-\frac{1}{2B}\sum_i\left[\log\frac{e^{S_{ii}}}{\sum_j e^{S_{ij}}}+\log\frac{e^{S_{ii}}}{\sum_j e^{S_{ji}}}\right]$$

Step-by-step explanation: A batch becomes B×B matching questions

Zero-shot classification reuses the text tower

Write a prompt for each candidate class, such as “a photo of a dog”. Encode and normalize each prompt. These text vectors form the columns of a classifier. Compare a new image vector with all candidates and choose the largest score. The resulting probability depends on the supplied candidate list, prompt wording and scale; it is not an unconditional probability that the world contains that class. Prompt ensembling averages normalized descriptions for a class and normalizes the resulting class vector. No image decoder or autoregressive text generator is required for this decision.

# Teaching pseudocode; not a complete training implementation.
class_vectors = encode_and_normalize(class_prompts)
image_vector = encode_and_normalize(image)
prediction = argmax(image_vector @ class_vectors.T)

Training state: Training and inference roles are separated below.

$$\hat y=\arg\max_c\,u(x)^Tv(t_c)$$

Step-by-step explanation: Zero-shot classification reuses the text tower

Inspect the pinned source

$$u_i=\frac{W_vf_v(x_i)}{\|W_vf_v(x_i)\|_2},\quad v_j=\frac{W_tf_t(t_j)}{\|W_tf_t(t_j)\|_2},\quad S_{ij}=e^a u_i^Tv_j$$

Two towers meet in a shared embedding space

The image tower pools an image feature, then projects it to dimension d. The text tower tokenizes a description, applies a causal Transformer, selects its end-of-text representation and projects to the same d. Normalize both vectors to unit length. Their dot product is then cosine similarity. Shared output dimension does not mean shared encoder weights. Original CLIP uses a learned positive logit scale. It has no cross-attention between image patches and words in this dual-encoder path.

$$u_i=\frac{W_vf_v(x_i)}{\|W_vf_v(x_i)\|_2},\quad v_j=\frac{W_tf_t(t_j)}{\|W_tf_t(t_j)\|_2},\quad S_{ij}=e^a u_i^Tv_j$$

CLIP: separate image and text towers, one paired similarity objective

CLIP: separate image and text towers, one paired similarity objective Original teaching diagram; exact configurations are explained in the article. · Paper sourceScroll on narrow screens; select the figure for full size.

Work through the mechanism

Read the architecture and the equations together. Each version changes a specific part of the learning system.

A batch becomes B×B matching questions

Arrange B paired images and texts in the same order. Row i compares image i with every text; column j compares text j with every image. Diagonal entries are the paired positives. Image-to-text cross-entropy normalizes each row; text-to-image cross-entropy normalizes each column. Average both directions. Increasing one negative score changes the softmax probabilities of other entries in that row or column. Unpaired entries are treated as negatives even when their content is actually compatible, so noisy and repeated captions matter.

$$\mathcal L_{CLIP}=-\frac{1}{2B}\sum_i\left[\log\frac{e^{S_{ii}}}{\sum_j e^{S_{ij}}}+\log\frac{e^{S_{ii}}}{\sum_j e^{S_{ji}}}\right]$$

OpenCLIP makes the training ecosystem inspectable

OpenCLIP is an open implementation, training toolkit and collection of checkpoints. It is not a single new loss that replaces CLIP. The reproducible scaling study varies model size, LAION data scale and the number of samples seen while retaining the contrastive objective. Dataset size and samples seen are different quantities because training can revisit examples. Report the architecture and training-data tag together: a ViT-L/14 name alone does not identify one embedding distribution or one set of weights.

SigLIP replaces the batch normalization in the objective

SigLIP treats every image–text pair as a binary classification example. Let yij be +1 on the diagonal and −1 elsewhere. A learned scale and bias produce the logit. The loss applies log-sigmoid independently to each labeled pair, without row or column softmax denominators. It still uses negative pairs, and processing all pairs still has quadratic pair count. The original paper divides the summed pair loss by B, not B². Its pairwise structure permits chunked computation without a global softmax normalizer; it does not remove shared-parameter coupling between pairs.

$$\mathcal L_{SigLIP}=-\frac1B\sum_{i,j}\log\sigma\!\left(y_{ij}(e^a u_i^Tv_j+b)\right)$$

CLIP-related methods: separate the ecosystem from the objective

CLIP-related methods: separate the ecosystem from the objective Original teaching diagram; exact configurations are explained in the article. · Paper sourceScroll on narrow screens; select the figure for full size.

Same pair scores: softmax competition or binary logistic losses

Same pair scores: softmax competition or binary logistic losses Original teaching diagram; exact configurations are explained in the article. · Paper sourceScroll on narrow screens; select the figure for full size.

One similarity matrix, two objectives

Fixed teaching vectors. We compute probabilities and losses, not training stability or model quality.

Temperature τ

0.20

Mismatch the first two captions (diagonal labels stay positive)Sigmoid bias b

−1.00

Image unit-vector angles: 0°, 75°, 150°. Text angles: 10°, 65°, 160°. Switch between similarities and probabilities; rows are images and columns are texts.

Image / text	0	1	2
0	0.9432	0.0567	0.0001
1	0.0561	0.9334	0.0105
2	0.0002	0.0111	0.9887

I→T:

0.0463

· T→I:

0.0463

CLIP:

0.0463

· SigLIP:

1.2550

SigLIP divides the summed pair loss by batch size B, as in the paper, not B². Loss scales differ; a smaller number does not identify a better model.

Reset

Training is not sampling

Zero-shot classification reuses the text tower

Write a prompt for each candidate class, such as “a photo of a dog”. Encode and normalize each prompt. These text vectors form the columns of a classifier. Compare a new image vector with all candidates and choose the largest score. The resulting probability depends on the supplied candidate list, prompt wording and scale; it is not an unconditional probability that the world contains that class. Prompt ensembling averages normalized descriptions for a class and normalizes the resulting class vector. No image decoder or autoregressive text generator is required for this decision.

$$\hat y=\arg\max_c\,u(x)^Tv(t_c)$$

SigLIP 2 improves both global and local features

SigLIP 2 combines sigmoid image–text training with a training-only decoder for captioning and localization, then adds self-distillation and masked patch prediction. The released vision encoder is not that caption decoder. Pooled vectors support image–text matching; unpooled patch features support dense tasks and visual-language connectors. Fixed-resolution variants and NaFlex are different recipes. NaFlex preserves aspect ratio approximately, adds patch coordinates and padding masks, and supports several sequence lengths. Its adaptation stage omits the self-distillation and masked-prediction losses to manage cost.

What this does not establish

What a VLM takes from a CLIP-family encoder

A multimodal language model often takes patch features before global pooling, then changes their width or spatial count with a connector. The learned text vocabulary belongs to a text tokenizer; it is not a codebook for those visual vectors. Qwen3-VL continues training a SigLIP-2-initialized vision tower, so its final representation is not identical to the original contrastive checkpoint. For reconstruction, an RAE decoder must learn to map the selected features back to pixels. Alignment quality alone does not guarantee that all small text, geometry or texture survives.

Check your understanding

Are these features discrete reconstruction codes?

No. The backbone emits continuous features. Prediction prototypes or text IDs do not make them a VQ codebook. Reconstruction requires a separately specified decoder and training objective.

Sources and next steps

• CLIP and SigLIP series · 2103.00020v1
• Method / pinned implementation source
• RAE
• Qwen3-VL: from patches to DeepStack

Return to the learning map

Original teaching diagrams and snippets are identified separately from attributed paper figures. Paper reports, implementation observations, and analytical examples are distinguished. No research model runs in this page.