Step by Step
E
Encoder — building rich contextual understanding
The encoder stack reads the input sequence and builds rich contextual representations using self-attention, where every word attends to every other word in the input.
Example: an encoder processing a sentence to be translated, building a deep contextual representation of its full meaning before any translation output begins.
D
Decoder — generating output token by token
The decoder stack generates the output sequence one token at a time, attending both to the encoder's output and to its own previously generated tokens.
Example: a decoder generating a translated sentence one word at a time, referring back to the encoder's understanding of the original sentence and to the words it has already generated.
A
Attention — connecting any position to any other, regardless of distance
The attention mechanism lets any position in a sequence directly influence and be influenced by any other position, regardless of how far apart they are — this is exactly what solved the long-range dependency problem that plagued RNNs.
Example: attention letting the very first word of a long document directly influence the model's understanding of the very last word, without that information needing to pass sequentially through every word in between.
Applied Walkthrough
1
A translation system needs to convert a long sentence in one language into another, correctly capturing relationships between words that are far apart in the sentence.
2
The encoder first reads the entire input sentence, using self-attention to build a rich contextual representation where every word has already "seen" every other word.
3
The decoder then generates the translated output one token at a time, attending both to this rich encoder representation and to the tokens it has already generated in the output so far.
4
Because attention allows any position to directly connect to any other regardless of distance, this architecture handles long-range word relationships far better than the sequential, memory-based approach an RNN would use.
Exam Application
Exams test whether you can describe the distinct roles of the encoder (understanding, via self-attention) and decoder (generation, attending to both encoder output and its own prior outputs), and whether you understand attention as the unifying mechanism that solved RNNs' long-range dependency problem.
⚠ Common Trap
The most common trap is assuming every modern Transformer-based model uses both an encoder AND a decoder. In practice, many well-known models use only one or the other — BERT is encoder-only (for understanding tasks), while GPT and Claude are decoder-only (for generation tasks); only some models (like T5) use the full encoder-decoder structure.
✓ Quick Self-Check
1. What is the encoder's primary job in a Transformer?
Reading the input sequence and building rich contextual representations using self-attention.
Tap to reveal / hide
2. What is the decoder's primary job in a Transformer?
Generating the output sequence token by token, attending to both the encoder's output and its own previous outputs.
Tap to reveal / hide
3. What does the attention mechanism specifically solve that RNNs struggled with?
The long-range dependency problem, by letting any position directly influence any other regardless of distance.
Tap to reveal / hide
4. Do all Transformer-based models use both an encoder and a decoder?
No — some (like BERT) are encoder-only, some (like GPT and Claude) are decoder-only, and some (like T5) use the full encoder-decoder structure.
Tap to reveal / hide
5. What kind of tasks are encoder-only models like BERT typically used for?
Understanding tasks, rather than generation tasks.
Tap to reveal / hide