A Short Course for Modern Large Language ModelsGlossary

Lesson 10 of 23 · LLM preliminaries

Text becomes IDs, then learned vectors.

Your win: explain the two distinct operations that turn a string into the \(B\times T\times d_{\text{model}}\) tensor consumed by a decoder.

16 minutesNeeds: vectors and matricesOutcome: trace text to embeddings
Module bridge: Transformer blocks never receive raw words. They receive learned vectors selected by integer token IDs.

Tokenization chooses reusable text pieces

A tokenizer maps text to a sequence of vocabulary entries. Character tokenization is simple but long; whole-word tokenization is compact but struggles with unseen words. Subword tokenizers reuse common pieces such as low, er, and est.

Five-stage diagram showing how byte-pair encoding repeatedly merges frequent adjacent symbols into reusable subword tokens
Byte-pair encoding learns a vocabulary from repeated local merges. The tokenizer and the neural network are separate learned systems.

Numbers substituted: one short sequence

Suppose the tokenizer returns:

\[\text{``I like cats''}\longrightarrow [1,2,3].\]

The integers are labels, not meanings. Token ID \(2\) does not mean “twice as much” as token ID \(1\).

Embedding lookup selects matrix rows

Let the vocabulary size be \(V=5\) and model width be \(D=3\). The learned embedding table is \(E\in\mathbb R^{5\times3}\). For the IDs \([1,2,3]\):

\[ E=\begin{bmatrix} 0.1&0.0&-0.2\\ 0.2&-0.1&0.5\\ 0.7&0.3&-0.2\\ -0.4&0.9&0.1\\ 0.0&0.2&0.4 \end{bmatrix},\qquad E[[1,2,3]]=\begin{bmatrix} 0.2&-0.1&0.5\\ 0.7&0.3&-0.2\\ -0.4&0.9&0.1 \end{bmatrix}. \]
Diagram showing token IDs selecting rows from a trainable embedding matrix and producing one dense vector per token
For batched IDs with shape \([B,T]\), embedding lookup adds the feature axis and returns \([B,T,D]\).
Keep the boundary clear: tokenization decides which IDs represent the string. The embedding layer decides which learned \(D\)-dimensional vectors those IDs retrieve.

Code checkpoint · lookup preserves batch and time

Show the PyTorch embedding lookup
import torch
from torch import nn

token_ids = torch.tensor([[1, 2, 3], [3, 2, 1]])  # [B=2, T=3]
embedding = nn.Embedding(num_embeddings=5, embedding_dim=3)
x = embedding(token_ids)                           # [B=2, T=3, D=3]

assert x.shape == (2, 3, 3)

Trace it: Which output axis comes from embedding_dim, and which two axes came directly from token_ids?

Retrieval check

What does token ID \(37\) do inside an embedding layer?

Practice before moving on

  1. Explain one tradeoff between character tokens and whole-word tokens.
  2. If \(V=32{,}000\) and \(D=512\), what is the embedding-weight shape?
  3. For token IDs with shape \(4\times128\), what is the embedding output shape when \(D=512\)?
  4. Using the displayed matrix \(E\), write the vector selected by token ID \(4\).
  5. Why is token ID \(200\) not numerically “larger in meaning” than token ID \(10\)?
  6. State the full pipeline from a text string to hidden-state tensor \(X\).
Check solutions
  1. Characters handle unseen text but create longer sequences; whole words create shorter sequences but require a huge vocabulary and fail on unseen forms.
  2. \(32{,}000\times512\).
  3. \(4\times128\times512\).
  4. \((0.0,0.2,0.4)\).
  5. IDs are arbitrary vocabulary indices, not ordered measurements.
  6. Text → tokenizer → token IDs \([B,T]\) → embedding lookup → \(X\in\mathbb R^{B\times T\times D}\).

Sources: Sennrich et al., Neural Machine Translation of Rare Words with Subword Units; PyTorch nn.Embedding.