A Short Course for Modern Large Language ModelsFormula sheet

Lesson 8 of 9 · RoPE assembly

Each pair carries a different clock.

Your win: explain why adjacent dimensions are paired and why the pairs rotate at geometrically spaced rates.

14 minutesNeeds: radians, rotationOutcome: interpret \(\theta_i\)
Mission link: the supplied diagram's fast and slow rotations are the multi-scale positional signal used by RoPE.

Position becomes phase

For dimension pair \(i\), choose an angular frequency \(\theta_i\). At token position \(m\), rotate by:

\[\phi_i(m)=m\theta_i.\]

Moving forward one token adds \(\theta_i\) radians. Moving forward \(r\) tokens adds \(r\theta_i\) radians.

The original frequency schedule

\[\theta_i=10000^{-2i/d_h},\qquad i=0,1,\ldots,\frac{d_h}{2}-1.\]

Here \(d_h\) is the query/key width of one attention head. The frequencies decrease geometrically: early pairs turn quickly; later pairs turn slowly.

position m = 20

Example: \(d_h=8\)

PairDimensions\(\theta_i\)Period \(2\pi/\theta_i\)
0\((q_0,q_1)\)\(1\)\(\approx6.28\) tokens
1\((q_2,q_3)\)\(0.1\)\(\approx62.8\) tokens
2\((q_4,q_5)\)\(0.01\)\(\approx628\) tokens
3\((q_6,q_7)\)\(0.001\)\(\approx6283\) tokens

Fast pairs are sensitive to small shifts. Slow pairs preserve distinct phase over longer ranges. Together, the stack offers multiple positional scales.

Overview of RoPE dimension pairs, frequency scales, and relative position inside attention
Read the first two columns with this lesson: adjacent feature pairs form 2D planes, and each plane receives its own frequency. Lesson 9 derives the relative-position term shown on the right.

Retrieval check

Which frequency gives the longer period?

Practice before moving on

  1. For \(d_h=8\), list all four adjacent dimension pairs.
  2. Calculate the angle for \(m=7\) and \(\theta_i=0.1\).
  3. Calculate the relative angle for positions \(m=5\), \(n=8\), and \(\theta_i=0.1\).
  4. Estimate the period for \(\theta_i=0.01\).
  5. Explain why using only \(\theta=1\) would make long-range positions ambiguous.
Check solutions
  1. \((q_0,q_1),(q_2,q_3),(q_4,q_5),(q_6,q_7)\).
  2. \(0.7\) radians.
  3. \((8-5)(0.1)=0.3\) radians.
  4. \(2\pi/0.01\approx628\) tokens.
  5. The phase repeats every \(2\pi\), so one fast clock wraps after only about 6.28 tokens; slower clocks retain other scales.
Why pairs? a proper rotation needs a plane. Pairing two real dimensions creates that plane; assigning one \(\theta_i\) to it creates one positional clock.

Primary source: Su et al., RoFormer, Section 3.2.2. Compare Equations 14–16 with the pair table above.

Ask the teaching agent to calculate the schedule for a different head dimension if the exponent still feels opaque.