01 — Architecture
The Big Picture
Before Transformers, sequence models were dominated by RNNs and LSTMs — architectures that processed tokens one at a time, left to right. This made parallelization impossible and caused gradients to vanish over long sequences.
The Transformer replaced recurrence entirely with self-attention, allowing every token to directly attend to every other token in one step.
Core Insight
Instead of reading a sentence word-by-word, the Transformer reads the entire sequence at once and learns which words are relevant to each other — regardless of how far apart they are.
Click any block in the architecture diagram to learn what it does.
Encoder
Input Tokens
Input Embedding
+ Positional Encoding
Multi-Head Self-Attention
Add & Norm
Feed-Forward Network
Add & Norm
Encoder Output (×N)
Decoder
Output Tokens (shifted)
Output Embedding
+ Positional Encoding
Masked Multi-Head Attention
Add & Norm
Cross-Attention
Add & Norm
Feed-Forward Network
Linear + Softmax
Output Probabilities
// select a block to explore
Click any colored block →
Each component plays a specific role. Explore them by clicking.
02 — Input Representation
Embeddings & Positional Encoding
The Transformer maps tokens to dense vectors in a high-dimensional space (typically d_model = 512). Since all tokens are processed simultaneously, positional encodings inject sequence order information.
E(pos, 2i) = sin(pos / 100002i/d) E(pos, 2i+1) = cos(pos / 100002i/d)
Why Sinusoids?
Sine and cosine at different frequencies create a unique fingerprint for every position. The model can learn relative distances because PE(pos+k) is a linear function of PE(pos).
▲ Positional encoding heatmap — position on x-axis, embedding dimension on y-axis.
Tokenize the input text
Text is split into subword tokens using BPE or WordPiece. "unhappiness" → ["un", "##happi", "##ness"]. Each token gets a unique integer ID from a fixed vocabulary of ~30k–50k tokens.
Look up token embeddings
Each token ID indexes into an embedding matrix E ∈ ℝ^(vocab_size × d_model). This lookup produces a d_model-dimensional vector that is learned during training.
Add positional encodings
PE vectors are computed using sinusoidal functions and added to token embeddings. The result encodes both "what word" and "where in sequence" in a single vector.
Feed combined vectors into encoder
The summed vectors form input sequence X ∈ ℝ^(seq_len × d_model). All positions flow in parallel through attention and FFN layers.
03 — Core Mechanism
Scaled Dot-Product Attention ●interactive
Attention asks: "Given a query, which keys are most relevant, and what are their values?" Every token generates a Query (Q), a Key (K), and a Value (V).
Attention(Q, K, V) = softmax( QKT / √dk ) · V
What information am I looking for? Computed as Q = X·W_Q
What information do I offer? High Q·K dot product = strong relevance. K = X·W_K
What information do I contain? The weighted sum of values becomes the output. V = X·W_V
Why divide by √dk? As dimension grows, dot products get very large, pushing softmax into near-zero gradient regions. Scaling stabilizes training.
Self-Attention Visualizer
Select a word to see which tokens it attends to. Brighter cells = higher attention weight.
The
animal
didn't
cross
the
street
because
it
was
tired
← Click a word to explore its attention pattern
Self-Attention vs Cross-Attention
In Self-Attention, Q, K, and V all come from the same sequence. In Cross-Attention (decoder only), Q comes from the decoder and K, V come from the encoder output.
04 — Parallelism
Multi-Head Attention ●interactive
Multi-Head Attention runs h attention heads in parallel, each with its own projections, capturing different types of relationships simultaneously.
MultiHead(Q,K,V) = Concat(head1, ..., headh) · W_O
In the original paper, h = 8 heads with d_k = d_v = 64. Click each head to see what it specializes in.
Head 1
Syntax / subject-verb
Head 2
Coreference resolution
Head 3
Local context window
Head 4
Long-range dependencies
Head 5
Semantic similarity
Head 7
Positional attention
Head 8
Rare/salient tokens
05 — Encoder
The Encoder Stack
The encoder transforms the input sequence into a rich contextual representation. It consists of N = 6 identical layers, each with two sub-layers.
// Each Encoder Layer
Multi-Head Self-Attention
Feed-Forward Network (×2 linear)
// Feed-Forward Network
A two-layer MLP applied independently to each position:
FFN(x) = max(0, xW₁ + b₁)W₂ + b₂
d_ff = 2048 (4× wider than d_model). Stores factual knowledge.
// Residual & Layer Norm
Each sub-layer uses a residual connection:
output = LayerNorm(x + Sublayer(x))
Allows gradients to flow directly, enabling training of very deep stacks.
What does the encoder actually learn?
After 6 layers, the word "bank" in "river bank" will have a very different vector than "bank" in "deposit at the bank". The encoder has resolved ambiguity through context.
06 — Decoder
The Decoder Stack
The decoder generates output tokens auto-regressively — one at a time. It has three sub-layers per layer.
// Each Decoder Layer
Cross-Attention (Q←dec, KV←enc)
// The Causal Mask
Position i can only attend to positions ≤ i. Prevents looking at future tokens.
▲ Attention mask (cyan=allowed, dark=masked)
// Output Distribution
Linear projection → vocab_size, then softmax for next-token probability.
07 — Optimization
Training the Transformer
The model is trained with teacher forcing — ground-truth previous tokens are fed as decoder input, stabilizing training.
Loss Function
Cross-entropy loss over the output vocabulary at every position: L = −(1/T) Σ log P(y_t | y_<t, x)
Label Smoothing
Instead of one-hot targets, smooth the distribution (ε=0.1). Improves generalization and prevents overconfidence.
Warmup Schedule
Learning rate increases linearly for warmup_steps, then decays as inverse square root. Prevents early large updates from destabilizing.
Dropout
Applied to attention weights (p=0.1) and between sub-layers. Prevents co-adaptation of neurons.
Adam Optimizer
β₁=0.9, β₂=0.98, ε=10⁻⁹. Adaptive per-parameter learning rates make transformer training robust.
Key scaling numbers from the original paper:
d_model = 512d_ff = 2048h = 8 headsN = 6 layers~65M parameters100K steps
08 — Reference
Key Terms Glossary
Token
A unit of text — typically a word or subword. "playing" might split into ["play","##ing"].
Embedding
A dense vector representation of a token. Similar tokens cluster together in embedding space.
Attention Weight
A scalar in [0,1] indicating how much one token attends to another. All weights for a query sum to 1.
d_model
The dimensionality of all internal representations. Every layer produces vectors of this size.
d_k / d_v
Dimensionality of queries/keys and values within each attention head. d_k = d_model / h.
Residual Connection
Adding the input directly to the sub-layer output: out = LayerNorm(x + F(x)). Enables deep network training.
Layer Normalization
Normalizes activations across the feature dimension. Stabilizes training of deep networks.
Auto-regressive
Generating outputs one token at a time, feeding each output back as input for the next step.
Teacher Forcing
During training, feeding ground-truth previous tokens to the decoder instead of its own predictions.
Softmax
Converts raw scores into a probability distribution. exp(xᵢ) / Σexp(xⱼ)
Cross-Entropy Loss
Measures how well predicted probabilities match ground truth. Lower = better.
Encoder-Decoder
The original Transformer — encoder reads input, decoder generates output. Used for translation.