Guided reference · 5 steps

Walk through one decoder block.

Follow the residual stream through attention, verify every tensor shape, inspect masking and softmax, then finish with normalization and SwiGLU.

Step 1 · See the complete path before opening it

Block equations

\[ \bar X=\operatorname{Norm}(X^{(\ell)}), \qquad Q=\bar XW_Q,\quad K=\bar XW_K,\quad V=\bar XW_V, \] \[ Q'=\operatorname{RoPE}(Q),\qquad K'=\operatorname{RoPE}(K), \] \[ P=\operatorname{softmax}\!\left(\frac{Q'K'^{\mathsf T}}{\sqrt{d_h}}+M_{\text{causal}}\right), \qquad O_{\text{attn}}=\operatorname{Concat}(P_1V_1,\ldots,P_hV_h)W_O, \] \[ Y=X^{(\ell)}+O_{\text{attn}}, \qquad X^{(\ell+1)}=Y+\operatorname{FFN}(\operatorname{Norm}(Y)). \]

Checkpoint: find the two residual additions and verify that both return \([B,T,d_{\text{model}}]\).

Step 2 · Keep the tensor-shape ledger

Track every axis

Quantity	Shape	Invariant to remember
\(X,\bar X,Q,K,V\)	\(B\times T\times d_{\text{model}}\)	Projection acts on the final feature axis.
Split Q, K, V	\(B\times h\times T\times d_h\)	\(d_{\text{model}}=h\,d_h\); every head keeps all \(T\) tokens.
Scores and probabilities	\(B\times h\times T\times T\)	Rows are queries; columns are keys.
One head output	\(B\times T\times d_h\)	\(P_rV_r\) blends value vectors.
Concatenated/output state	\(B\times T\times d_{\text{model}}\)	Required for residual addition.

Checkpoint: explain why heads split the feature axis \(d_{\text{model}}\), not the token axis \(T\).

Step 3 · Add position, then prevent peeking

Rotate Q and K; mask future keys

Causal mask

\[ M_{ij}=\begin{cases}0,&j\le i,\\-\infty,&j>i.\end{cases} \]

Add it before softmax. Future weights then become zero.

RoPE identity

\[ (R_mq)^{\mathsf T}(R_nk)=q^{\mathsf T}R_{n-m}k. \]

With the course's counterclockwise convention, the sign is \(n-m\).

Checkpoint: RoPE changes score geometry; the causal mask changes which key positions are allowed.

Step 4 · Turn one score row into weights

Numerical softmax check

\[ \operatorname{softmax}(0.7,1.1,0.2,-\infty) \approx(0.323,0.482,0.196,0). \]

The three displayed rounded finite weights total \(1.001\); the unrounded values total exactly \(1\).

Checkpoint: the masked future position receives probability zero, while the allowed positions sum to one.

Step 5 · Normalize and transform each token's features

Normalization and SwiGLU

\[ \operatorname{LN}(x)=\gamma\odot\frac{x-\mu}{\sqrt{\sigma^2+\varepsilon}}+\beta, \qquad \operatorname{RMSNorm}(x)=\gamma\odot\frac{x}{\sqrt{\frac1d\sum_rx_r^2+\varepsilon}}, \] \[ \operatorname{FFN}(x) =\left(\operatorname{SiLU}(xW_{\text{gate}})\odot(xW_{\text{up}})\right)W_{\text{down}}. \]

Role check: Q asks, K is compared, V carries content. RoPE rotates Q and K only. Attention mixes across token positions; the FFN transforms each token independently.

See RESOURCES.md for the primary papers and Lesson 23 for a fully substituted walkthrough.