Step by Step
R
RNN — sequential processing, one step at a time
RNNs process a sequence one step at a time, passing a hidden state forward — like reading a book one word at a time and updating your memory as you go.
Example: an RNN processing a 100-word sentence must process word 1, then word 2, then word 3, and so on sequentially, one at a time.
N
(The problem) — early information fades by the end
By the time an RNN reaches word 100 of a long sequence, information from word 1 has often nearly vanished from memory, due to the vanishing gradient problem over long sequences.
Example: an RNN processing a long document might have essentially forgotten a character's name introduced in the very first sentence by the time it reaches the last paragraph.
N
(Transformers) — processing everything simultaneously
Transformers process all words in a sequence simultaneously using attention, letting every word directly see every other word at once, rather than passing information sequentially step by step.
Example: a Transformer processing that same 100-word sentence can directly relate word 1 to word 100 in a single attention computation, without any information having to pass sequentially through every word in between.
⭐
Why this made Transformers win — speed and long-range dependencies
This parallelism made Transformers dramatically faster to train on GPUs (since all positions can be processed simultaneously rather than sequentially) and much better at capturing long-range dependencies — which is why RNNs are now largely obsolete for most NLP tasks.
Example: training a Transformer on a large text corpus can take full advantage of GPU parallelism, while an RNN's inherently sequential nature limits how much it can be parallelized during training.
Applied Walkthrough
1
A model needs to correctly relate a pronoun near the end of a very long paragraph back to a noun introduced at the very beginning of that same paragraph.
2
An RNN processing this paragraph sequentially, one word at a time, risks having nearly forgotten that early noun by the time it reaches the pronoun many words later, due to the vanishing gradient problem.
3
A Transformer instead processes the entire paragraph simultaneously, letting the pronoun's representation directly attend back to that early noun in a single attention computation, regardless of how many words separate them.
4
This combination of training speed (through parallelism) and long-range dependency handling is exactly why Transformers displaced RNNs as the dominant architecture for most modern NLP tasks.
Exam Application
Exams test whether you understand the core architectural difference (RNN: sequential with a carried hidden state; Transformer: parallel via attention) and specifically why this difference led to Transformers displacing RNNs — training speed on GPUs and superior long-range dependency handling.
⚠ Common Trap
The most common trap is assuming RNNs are completely obsolete for every possible task. While Transformers dominate most modern NLP applications, RNNs (and their LSTM/GRU variants) can still appear in some specialized or resource-constrained contexts — but for the vast majority of mainstream NLP work today, Transformers are the standard choice.
✓ Quick Self-Check
1. How does an RNN process a sequence?
One step at a time, passing a hidden state forward sequentially.
Tap to reveal / hide
2. What problem do RNNs face with very long sequences?
Vanishing gradients — information from early in the sequence can nearly vanish by the time later positions are processed.
Tap to reveal / hide
3. How does a Transformer process a sequence, compared to an RNN?
All at once, in parallel, using attention to let every word directly see every other word simultaneously.
Tap to reveal / hide
4. What two advantages did this parallelism give Transformers over RNNs?
Dramatically faster training on GPUs, and better handling of long-range dependencies.
Tap to reveal / hide
5. What has largely happened to RNNs for most modern NLP tasks?
They have become largely obsolete, replaced by Transformers as the dominant architecture.
Tap to reveal / hide