Step by Step
N
Normalization (Min-Max) — scales to a 0-1 range
Rescales every feature so its values fall between 0 and 1, based on the minimum and maximum values observed in the training data.
Example: a feature ranging from 20 to 200 gets rescaled so 20 becomes 0.0 and 200 becomes 1.0, with values in between proportionally scaled.
S
Standardization (Z-score) — mean 0, standard deviation 1
Rescales every feature so it has a mean of 0 and a standard deviation of 1, which is often preferred when data may contain outliers.
Example: a feature with mean 50 and standard deviation 10 gets transformed so its new mean is 0 and its new standard deviation is 1.
E
Encoding categoricals — one-hot vs. label encoding
One-hot encoding converts categories into separate binary columns (appropriate for non-ordinal categories); label encoding converts categories into integers (only appropriate when the categories have a genuine order).
Example: a "color" column (red, blue, green — no natural order) should use one-hot encoding, while a "size" column (small, medium, large — has a natural order) can use label encoding.
⚠
The golden rule — fit transformers on training data only
Always split data into training and test sets FIRST, then fit any scaler, encoder, or PCA transformer only on the training data — applying that same fitted transformer to the test data without re-fitting it.
Example: fitting a standard scaler on the full dataset (including test data) before splitting leaks information about the test set's distribution into training — a classic data leakage mistake.
Applied Walkthrough
1
A student fits a standardization scaler on the entire dataset, then splits it into training and test sets afterward.
2
Ask: has any test-set information leaked into the training process? Yes — the scaler's mean and standard deviation were computed using test data too.
3
This is data leakage: the test set is no longer a genuinely unseen, honest measure of performance, because its statistics already influenced the transformer.
4
The correct order: split the data first, fit the scaler only on training data, then apply that same fitted scaler to transform the test data without re-fitting it.
Exam Application
This split-before-fit rule is one of the most heavily tested practical mistakes in applied ML. Exams often describe a workflow and ask you to spot where the ordering was wrong. Also expect questions asking which algorithms require scaling (K-NN, SVM, neural networks, PCA) versus which don't (tree-based methods like Random Forest and XGBoost).
⚠ Common Trap
The most common trap is fitting any transformer (scaler, encoder, PCA) on the full dataset before splitting into train/test. This leaks test-set information into training and inflates reported performance. A second trap is using label encoding on non-ordinal categories, which falsely implies an order relationship (like blue > red) that doesn't exist.
✓ Quick Self-Check
1. What does normalization (Min-Max scaling) do to a feature?
Rescales its values to fall between 0 and 1, based on the training data's minimum and maximum.
Tap to reveal / hide
2. What does standardization (Z-score) do to a feature?
Rescales it to have a mean of 0 and a standard deviation of 1.
Tap to reveal / hide
3. When should you use one-hot encoding instead of label encoding?
When the categories have no natural order (like colors), to avoid implying a false ordinal relationship.
Tap to reveal / hide
4. What is the correct order: split data first, or fit transformers first?
Split first, then fit transformers only on the training data — fitting on the full dataset before splitting causes data leakage.
Tap to reveal / hide
5. Which types of algorithms typically do NOT require feature scaling?
Tree-based methods, such as decision trees, Random Forest, and XGBoost.
Tap to reveal / hide