A Short Course for Modern Large Language ModelsFormula sheet

Lesson 7 of 9 · Attention bridge

Attention begins with a comparison.

Your win: locate the query-key dot product where RoPE injects position, without learning the full Transformer.

10 minutesNeeds: dot productsOutcome: identify the RoPE insertion point
Mission link: RoPE matters because it changes the score used to decide which keys a query aligns with.

Three roles

Query \(\mathbf q_m\)

What token \(m\) is looking for.

Key \(\mathbf k_n\)

What token \(n\) offers for matching.

Value \(\mathbf v_n\)

The information carried if the match receives weight.

The score before softmax

\[s_{m,n}=\frac{\mathbf q_m^{\mathsf T}\mathbf k_n}{\sqrt{d_k}}.\]

The dot product supplies alignment. The factor \(1/\sqrt{d_k}\) controls scale. Softmax later converts all scores for query \(m\) into weights, but RoPE acts earlier, on \(\mathbf q_m\) and \(\mathbf k_n\).

RoPE changes the comparison frame

\[ \mathbf q'_m=R_m\mathbf q_m,\qquad \mathbf k'_n=R_n\mathbf k_n,\qquad \mathbf v'_n=\mathbf v_n. \]

Queries and keys rotate according to their positions. Values remain unchanged in the original RoPE formulation. The rotated score is:

\[s'_{m,n}=\frac{(R_m\mathbf q_m)^{\mathsf T}(R_n\mathbf k_n)}{\sqrt{d_k}}.\]

A tiny alignment example

Let \(\mathbf q=(1,0)\). Then \(\mathbf q^{\mathsf T}(1,0)=1\), while \(\mathbf q^{\mathsf T}(0,1)=0\). Rotating a key changes its alignment with the query even though its length stays one.

Retrieval check

Which pair does original RoPE rotate?

Practice before moving on

  1. Compute \((1,2)^{\mathsf T}(3,4)\).
  2. Which has greater alignment with \((1,0)\): \((0.8,0.2)\) or \((0.2,0.8)\)? Show the two dot products.
  3. Write the scaled score for \(d_k=4\), \(\mathbf q=(1,2)\), and \(\mathbf k=(3,4)\).
  4. State in one sentence why RoPE rotates \(Q\) and \(K\), not only \(V\).
  5. If \(R_m=I\) and \(R_n\) is a quarter turn, compute \((R_m(1,0))^{\mathsf T}(R_n(1,0))\).
Check solutions
  1. \(11\).
  2. \((0.8,0.2)\); the scores are \(0.8\) and \(0.2\).
  3. \(11/\sqrt4=5.5\).
  4. The query-key dot product creates the comparison score that needs positional information.
  5. \((1,0)^{\mathsf T}(0,1)=0\).
Keep the scope narrow: to understand RoPE, you need the query-key score. You do not yet need multi-head concatenation, masking, or the output projection.

Primary source: Vaswani et al., “Attention Is All You Need,” Section 3.2.1. Read Equation 1 and the paragraph immediately below it.

Ask the teaching agent to trace one query against two keys if the three Q/K/V roles still blur together.