A Short Course for Modern Large Language ModelsFormula sheet

Lesson 9 of 9 · Capstone derivation

Absolute rotations reveal relative distance.

Your win: derive the RoPE identity from first principles, audit its sign, and teach the algorithm back without notes.

18 minutesNeeds: all prior lessonsOutcome: derive \(R_m^{\mathsf T}R_n=R_{n-m}\)
Mission achieved when: every line below has both an algebraic reason and a geometric sentence you can say aloud.

Set up two absolute positions

Let the query be at position \(m\) and the key at position \(n\). For one dimension pair with frequency \(\theta\):

\[\mathbf q'_m=R_{m\theta}\mathbf q_m,\qquad \mathbf k'_n=R_{n\theta}\mathbf k_n.\]

To keep notation light, write \(R_m\) for \(R_{m\theta}\) and \(R_n\) for \(R_{n\theta}\).

Derive the score

\[ \begin{aligned} (\mathbf q'_m)^{\mathsf T}\mathbf k'_n &=(R_m\mathbf q_m)^{\mathsf T}(R_n\mathbf k_n)\\ &=\mathbf q_m^{\mathsf T}R_m^{\mathsf T}R_n\mathbf k_n\\ &=\mathbf q_m^{\mathsf T}R_{-m}R_n\mathbf k_n\\ &=\mathbf q_m^{\mathsf T}R_{n-m}\mathbf k_n. \end{aligned} \]
  1. Substitute the rotated query and key.
  2. Use \((A\mathbf x)^{\mathsf T}=\mathbf x^{\mathsf T}A^{\mathsf T}\).
  3. Use \(R_m^{\mathsf T}=R_{-m}\): transpose reverses the query rotation.
  4. Use \(R_\alpha R_\beta=R_{\alpha+\beta}\): \(-m+n=n-m\).
Geometric and matrix views of composing a key rotation with the inverse query rotation
Use this board as a sign audit: rotate by \(n\theta\), then undo \(m\theta\). With the convention used in this course, the net rotation is \((n-m)\theta\), so the correct matrix is \(R_{n-m}\).

Numbers substituted: \(m=5\), \(n=8\), \(\theta=0.1\)

The absolute angles are \(0.5\) and \(0.8\) radians. Their difference is \(0.3\) radians. Substitute numerical sine and cosine values:

\[ R_{0.5}\approx \begin{bmatrix} 0.8776&-0.4794\\ 0.4794&0.8776 \end{bmatrix}, \qquad R_{0.8}\approx \begin{bmatrix} 0.6967&-0.7174\\ 0.7174&0.6967 \end{bmatrix}. \]
\[ \begin{aligned} R_{0.5}^{\mathsf T}R_{0.8} &\approx \begin{bmatrix} 0.8776&0.4794\\ -0.4794&0.8776 \end{bmatrix} \begin{bmatrix} 0.6967&-0.7174\\ 0.7174&0.6967 \end{bmatrix}\\ &\approx \begin{bmatrix} 0.9553&-0.2955\\ 0.2955&0.9553 \end{bmatrix} =R_{0.3}. \end{aligned} \]

The matrix product is the rotation for \((n-m)\theta=(8-5)(0.1)=0.3\) radians.

Numbers substituted: the resulting score

Take \(\mathbf q=\mathbf k=(1,0)\). The rotated vectors are \((\cos0.5,\sin0.5)\) and \((\cos0.8,\sin0.8)\). Their dot product is:

\[ \begin{aligned} (R_{0.5}\mathbf q)^{\mathsf T}(R_{0.8}\mathbf k) &=\cos0.5\cos0.8+\sin0.5\sin0.8\\ &=\cos(0.8-0.5)\\ &=\cos0.3\approx0.9553. \end{aligned} \]

The two absolute angles disappear from the final comparison; only their \(0.3\)-radian separation remains.

Exact values: \(m=2\), \(n=7\), \(\theta=\pi/6\)

\[ (n-m)\theta=(7-2)\frac{\pi}{6}=\frac{5\pi}{6}, \qquad R_{5\pi/6}= \begin{bmatrix} -\sqrt3/2&-1/2\\ 1/2&-\sqrt3/2 \end{bmatrix}. \]
rad/token

mθ = 1.75nθ = 2.80relative angle (n−m)θ = 1.05 rad

Sign audit for the supplied image: with \(R_\theta=\begin{bmatrix}\cos\theta&-\sin\theta\\\sin\theta&\cos\theta\end{bmatrix}\) and the score \((R_m\mathbf q)^{\mathsf T}(R_n\mathbf k)\), the result is \(R_{n-m}\). The second image's headline shows \(R_{m-n}\), but its matrix algebra, geometric caption, and the first image all show \(R_{n-m}\). Reversing query/key order would reverse the sign.

Restore all dimension pairs

The full \(R_m\) is block diagonal. Pair \(i\) uses angle \(m\theta_i\), but the same identity holds in every block:

\[ R_m= \begin{bmatrix} R_{m\theta_0}&0&\cdots&0\\ 0&R_{m\theta_1}&\cdots&0\\ \vdots&\vdots&\ddots&\vdots\\ 0&0&\cdots&R_{m\theta_{d_h/2-1}} \end{bmatrix}, \qquad R_m^{\mathsf T}R_n=R_{n-m}. \]

Final retrieval check

What relative rotation appears in \((R_m\mathbf q)^{\mathsf T}(R_n\mathbf k)\)?

Capstone exercises

  1. Derive \((R_2\mathbf q)^{\mathsf T}(R_7\mathbf k)=\mathbf q^{\mathsf T}R_5\mathbf k\) one line at a time.
  2. If \(\theta=0.2\), \(m=3\), and \(n=9\), calculate both absolute angles and the relative angle.
  3. Expand \(R_\theta^{\mathsf T}\) and prove directly that it equals \(R_{-\theta}\).
  4. Explain why rotating both vectors by the same extra offset \(c\) does not change their relative rotation.
  5. Find and correct the sign inconsistency in the supplied second image.
  6. Without notes, give the five-part teach-back listed below.
Check solutions
  1. \((R_2\mathbf q)^{\mathsf T}(R_7\mathbf k)=\mathbf q^{\mathsf T}R_2^{\mathsf T}R_7\mathbf k=\mathbf q^{\mathsf T}R_{-2}R_7\mathbf k=\mathbf q^{\mathsf T}R_5\mathbf k\).
  2. \(m\theta=0.6\), \(n\theta=1.8\), and \((n-m)\theta=1.2\) radians.
  3. Swap rows and columns, then use cosine evenness and sine oddness.
  4. \((n+c)-(m+c)=n-m\).
  5. The headline says \(R_{m-n}\); under the displayed convention it should say \(R_{n-m}\).
  6. Use the rubric below; ask the teaching agent for assessment.

Two-minute teach-back

  1. Why does RoPE group dimensions in pairs?
  2. What does position \(m\) do to pair \(i\)?
  3. Why does the rotation preserve vector magnitude?
  4. Why is the transpose a negative rotation?
  5. How does the dot product turn \(m\) and \(n\) into \(n-m\)?

Evidence of mastery is not recognizing the answer on this page. It is reconstructing the chain after a delay and applying it to a fresh \(m,n,\theta\) example.

Primary source: Su et al., RoFormer, Sections 3.1–3.2, especially Equations 11–16.

Ask the teaching agent to assess your teach-back. A passed teach-back should become the first learning record and determine what comes next.