Walk Through a Transformer Block

Walk Through a Transformer Block A workflow diagram generated by Archify. 01 / Causal self-attention 02 / Residual stream 03 / Token-wise feed-forward Input X^(l) · [B, T, D] · Residual stream Input X^(l) [B, T, D] RMSNorm · normalize each token · Causal self-attention RMSNorm normalize each token Q/K/V · H heads · Causal self-attention Q/K/V H heads RoPE · rotate Q + K · Causal self-attention · V same RoPE rotate Q + K V same Attend · mask + V · Causal self-attention · softmax Attend mask + V softmax W_O · join H · Causal self-attention · [B, T, D] W_O join H [B, T, D] Y = X + A · A: attention output · Residual stream Y = X + A A: attention output RMSNorm · normalize Y · Token-wise feed-forward RMSNorm normalize Y SwiGLU FFN · gate, expand, project · Token-wise feed-forward · mix features SwiGLU FFN gate, expand, project mix features X' = Y + F · F: feed-forward output · Residual stream X' = Y + F F: feed-forward output carry X^(l) carry Y