Step by Step
T
Tokenize — splitting text into units
Raw text is split into individual tokens (words, subwords, or characters), the fundamental units the rest of the pipeline will work with.
Example: splitting the sentence "I love NLP" into the tokens "I," "love," "NLP."
N
Normalize — cleaning and standardizing
Text is lowercased, punctuation is handled, contractions are expanded, and traditionally high-frequency low-information stop words might be removed, with words further reduced to root forms via stemming or lemmatization.
Example: converting "Running" to "run" via lemmatization, or removing common stop words like "the" and "is" in traditional pipelines.
R
Represent — converting to numerical vectors
Text is converted into numerical vectors that models can actually process, using approaches ranging from simple word counts (Bag of Words, TF-IDF) to dense semantic embeddings.
Example: converting a document into a TF-IDF vector, or into a dense embedding vector using a model like BERT.
M
Model — applying an ML or DL algorithm
A machine learning or deep learning algorithm processes the represented text to perform the actual task, such as classification, generation, or translation.
Example: feeding the represented text into a trained classifier to predict sentiment (positive/negative).
E
Evaluate — measuring with task-specific metrics
Model performance is assessed using metrics appropriate to the specific task, such as accuracy for classification or BLEU for translation.
Example: evaluating a translation model's output quality using the BLEU metric.
Applied Walkthrough
1
A traditional NLP pipeline from a decade ago carefully applies stop word removal, stemming, and TF-IDF representation before feeding text into a classical ML model.
2
A modern LLM-based approach instead skips most of this traditional preprocessing entirely.
3
Ask: why can modern LLMs skip steps like stop word removal and stemming? Because they're trained end-to-end using BPE subword tokenization, learning to handle raw text directly without needing hand-crafted preprocessing.
4
This shift illustrates how the classic five-step pipeline still describes the conceptual stages of NLP, even though modern deep learning approaches have automated or bypassed several of the traditional manual steps.
Exam Application
Exams test whether you can name and describe all five pipeline stages in order, and whether you understand that modern LLMs, trained end-to-end with subword tokenization, skip most of the traditional manual preprocessing steps (like stop word removal and stemming) that older NLP pipelines relied on.
⚠ Common Trap
The most common trap is assuming modern LLMs still require the full traditional preprocessing pipeline (stop word removal, stemming, etc.). Modern LLMs are trained end-to-end using subword tokenization (like BPE), largely bypassing these traditional manual preprocessing steps.
✓ Quick Self-Check
1. What happens during the Tokenize step?
Raw text is split into individual tokens (words, subwords, or characters).
Tap to reveal / hide
2. What happens during the Normalize step?
Text is cleaned and standardized — lowercased, punctuation handled, contractions expanded, and words reduced to root forms.
Tap to reveal / hide
3. What happens during the Represent step?
Text is converted into numerical vectors that models can process, ranging from simple word counts to dense embeddings.
Tap to reveal / hide
4. Do modern LLMs still require the full traditional preprocessing pipeline?
No — they skip most of it, being trained end-to-end with BPE subword tokenization instead.
Tap to reveal / hide
5. What determines how model performance is measured in the Evaluate step?
Task-specific metrics appropriate to what the model is being used for, such as accuracy or BLEU.
Tap to reveal / hide