Step by Step
F
Forget gate — deciding what to erase
Controls what information from the previous memory state should be discarded, since not everything from earlier in the sequence remains relevant.
Example: forgetting details about a subject that's no longer the topic of conversation several sentences later.
I
Input gate — deciding what new information to add
Controls what new information from the current input should be added to the memory state.
Example: adding a newly introduced character's name to memory, since it may be relevant later in the sequence.
O
Output gate — deciding what to expose
Controls what portion of the current memory state should actually be exposed as the output at this step.
Example: exposing only the currently relevant part of memory needed to predict the next word, while keeping other stored information hidden internally.
Applied Walkthrough
1
A vanilla RNN processing a very long document struggles to remember details from many sentences earlier, due to vanishing gradients over long sequences.
2
An LSTM processing the same document uses its forget gate to discard irrelevant earlier details, its input gate to add new relevant information as it's encountered, and its output gate to expose only what's currently needed.
3
This gating mechanism allows the LSTM to selectively preserve important long-range information (like a character's name mentioned much earlier) far better than a vanilla RNN could.
4
Despite this improvement over vanilla RNNs, LSTMs (and their simplified cousin, GRUs) have now been largely replaced by Transformers for most modern NLP tasks, since Transformers handle long-range dependencies even more effectively via attention.
Exam Application
Exams test whether you can name and describe the three LSTM gates (forget, input, output) and whether you understand why LSTMs were developed specifically to address vanilla RNNs' vanishing gradient problem over long sequences. Also expect a note that GRUs are a simplified two-gate alternative with comparable performance.
⚠ Common Trap
The most common trap is assuming vanilla RNNs and LSTMs work identically, just with different names. Vanilla RNNs have no gating mechanism at all and suffer badly from vanishing gradients on long sequences; LSTMs specifically add the three-gate mechanism to solve this problem.
✓ Quick Self-Check
1. What does the forget gate in an LSTM control?
What information from the previous memory state should be discarded.
Tap to reveal / hide
2. What does the input gate in an LSTM control?
What new information from the current input should be added to the memory state.
Tap to reveal / hide
3. What does the output gate in an LSTM control?
What portion of the current memory state should be exposed as output at this step.
Tap to reveal / hide
4. What problem do vanilla RNNs suffer from that LSTMs were designed to fix?
Vanishing gradients over long sequences, causing them to "forget" early inputs.
Tap to reveal / hide
5. What architecture has now largely replaced LSTMs for most NLP tasks?
Transformers.
Tap to reveal / hide