RoPE-ViT: position enters the comparison

An interactive field guide

Absolute position adds an address to a token. Rotary position changes how two tokens compare.

Before this article: ViT · Attention and autoregression

What was already possible

ViT adds learned absolute position embeddings at its input. RoPE was already used in sequence models to encode relative displacement through rotations of query and key vectors. RoPE-ViT studies how to extend this mechanism to two spatial axes.

What changes, and why

Treat each pair of Q/K channels as a 2D vector. At position (x,y), rotate the pair by phase ωx x + ωy y. Axial RoPE assigns channel pairs to one axis at a time. Mixed RoPE lets a pair use both axes and learns the frequencies. This permits diagonal frequency directions directly. It does not mean axial Transformers cannot recognize diagonal objects.

Read from left to right. Select a module to connect its role, tensor shape, and calculation. Snippets are teaching pseudocode. Sources provide the paper method or audited implementation.

1. CoordinatesN → [N,2]2. Mixed phases[layers,heads,pairs,N]3. Attention scores[B,heads,N,N]

Coordinates

Build x and y from the actual patch grid.

x = index % width
y = index // width

Mixed phases

Each frequency pair combines both spatial axes.

phase = omega_x * x + omega_y * y
q_rot = rotate_pairs(q, phase)
k_rot = rotate_pairs(k, phase)

Attention scores

Apply the usual scaled dot product to rotated features.

scores = q_rot @ k_rot.transpose(-1,-2) / sqrt(head_dim)

Inspect the pinned source

$$(R_{\phi(p)}q)^T(R_{\phi(s)}k)=q^TR_{\phi(s)-\phi(p)}k$$

Work through the mechanism

Both Q and K are rotated. Their dot product depends on the difference of their phases. Moving both positions by the same displacement cancels from that difference. Content still matters: the rotation does not replace the learned Q/K projections.

The demo keeps relative y displacement at 2 and changes relative x displacement and frequency direction. It shows one unit-frequency channel pair. A real head sums many channel pairs with different frequencies and content.

Change a parameter, inspect the computed values, then compare this small example with the model described above.

Analytical teaching model · controls change the example, not a trained research model.

Relative x displacement

2

Frequency direction (degrees)

45

Reset

One channel pair, one unit frequency. Real attention sums many pairs with content-dependent Q and K.

Computed values for the current example

Quantity	Value
Mixed phase (radians)	2.828
Rotated unit-vector dot product	-0.951

Training is not sampling

The pinned implementation excludes the class token from rotation, uses float32 for frequency arithmetic, and keeps mixed frequencies trainable. RoPE acts inside attention at each layer. It can coexist with an absolute position table. The classification evidence below uses DeiT-III training for 400 epochs at 224².

Reported evidence, not a local reproduction

Setting	Metric	Value	Interpretation
ViT-S, APE, test 224 / 512	Top-1 accuracy (%)	80.4 / 75.4	Table 5, same trained model at two resolutions
ViT-S, mixed, test 224 / 512	Top-1 accuracy (%)	80.9 / 79.1	Classification only; higher is better

What this does not establish

Resolution extrapolation on classification does not establish better reconstruction or text rendering. Also, grid shape matters: equal token counts do not imply equal coordinates. A rectangular-grid implementation must invalidate position caches when height or width changes. Unanchored TiTok slots do not acquire 2D coordinates just because their count is a square.

Open the derivation

A rotation matrix is orthogonal. Moving the rotation on Q through the transpose changes its sign. Two planar rotations add angles, so the result contains only their phase difference.

$$R_{\phi(p)}^TR_{\phi(s)}=R_{-\phi(p)}R_{\phi(s)}=R_{\phi(s)-\phi(p)}$$

For a linear coordinate phase, a shared translation cancels. This is a property of the rotary comparison, not proof that the full finite-image network is translation invariant.

$$\phi(p+u)-\phi(s+u)=\omega^T(p-s)$$

Check your understanding

Why rotate both Q and K?

Their paired rotations make the comparison depend on phase difference. Rotating only Q leaves an absolute phase dependence.

Sources and next steps

• RoPE-ViT · 2403.13298v2
• Method / pinned implementation source
• AToken

Return to the learning map

Diagrams and snippets are teaching reconstructions. Paper reports, implementation observations, and analytical examples are distinguished. No research model runs in this page.