Vector
\(\mathbf v=(x,y)\)
A point or arrow in a 2D plane.
Guided reference · 4 steps
Build RoPE one idea at a time.
Begin with one 2D rotation, collect the identities it needs, derive relative position, then finish with sign and implementation checks.
Step 1 · Represent the same turn three ways
One pair, three views
Complex
\(z=x+iy\)
Rotate by multiplying \(ze^{i\phi}\).
Matrix
\(R_\phi\mathbf v\)
Apply the rotation matrix below.
\[R_\phi=\begin{bmatrix}\cos\phi&-\sin\phi\\[2pt]\sin\phi&\cos\phi\end{bmatrix}\]
Checkpoint: point to \(x\), \(y\), and \(\phi\) in each representation before moving on.
Step 2 · Gather the tools used by the proof
Five identities worth retrieving
| Unit circle | \(\cos^2\phi+\sin^2\phi=1\) | Explains length preservation. |
| Composition | \(R_\alpha R_\beta=R_{\alpha+\beta}\) | Successive rotations add angles. |
| Inverse | \(R_\phi^{-1}=R_\phi^{\mathsf T}=R_{-\phi}\) | Transpose reverses the turn. |
| Pair frequency | \(\theta_i=10000^{-2i/d}\) | Different pairs rotate at different rates. |
| Position angle | \(\phi_i(m)=m\theta_i\) | Absolute position becomes phase. |
Checkpoint: explain why \(R_m^{\mathsf T}=R_{-m}\) without multiplying the matrices.
Step 3 · Make relative position appear
Derive the attention comparison
- Rotate a query at position \(m\): \(\mathbf q'_m=R_m\mathbf q_m\).
- Rotate a key at position \(n\): \(\mathbf k'_n=R_n\mathbf k_n\).
- Form their dot product: \((\mathbf q'_m)^{\mathsf T}\mathbf k'_n=(R_m\mathbf q_m)^{\mathsf T}(R_n\mathbf k_n)\).
- Reverse multiplication through the transpose: \(=\mathbf q_m^{\mathsf T}R_m^{\mathsf T}R_n\mathbf k_n\).
- Reverse \(m\), then add \(n\): \(R_m^{\mathsf T}R_n=R_{-m}R_n=R_{n-m}\).
- Therefore: \((\mathbf q'_m)^{\mathsf T}\mathbf k'_n=\mathbf q_m^{\mathsf T}R_{n-m}\mathbf k_n\).
Interpretation: the query and key each receive an absolute-position rotation, but their comparison contains only the relative displacement \(n-m\). RoPE rotates \(Q\) and \(K\); \(V\) is not rotated in the original formulation.
Step 4 · Test the derivation against common mistakes
Common failure modes
| Symptom | Likely mistake | Check |
|---|---|---|
| Relative sign is reversed | Changed dot-product order or transpose order. | Write \((R_m\mathbf q)^{\mathsf T}(R_n\mathbf k)\) one step at a time. |
| Vector length changes | Signs in the rotation matrix are wrong. | Columns must be unit and perpendicular. |
| Dimensions are mixed | Pair layout differs from the assumed adjacent pairing. | Confirm the implementation's rotate-half convention. |
| Degrees appear in code | Angle units were confused. | Trigonometric functions consume radians. |
Final check: reproduce \(R_m^{\mathsf T}R_n=R_{n-m}\), then state which of \(Q\), \(K\), and \(V\) RoPE changes.
Primary source: Su et al., RoFormer, Sections 3.1–3.2. Ask the teaching agent to unpack any line you cannot reproduce without looking.