P
Principal Components — orthogonal directions of maximum variance
PCA finds orthogonal (perpendicular) directions in the data called principal components, ordered by how much variance each one captures. PC1 captures the greatest amount of variance, PC2 the second greatest, and so on.
Example: in a dataset with many correlated features, PC1 might capture the single direction along which the data varies the most.
C
Choosing k — the scree plot and the elbow
A scree plot shows variance explained as a function of the number of components (k) kept. The optimal number of components is typically chosen where the curve "elbows" — often keeping enough components to preserve around 95% of the total variance.
Example: a scree plot might show the first 10 components already capturing 95% of total variance, suggesting those 10 components are sufficient, discarding the rest.
A
Always standardize first — the critical preprocessing step
Because PCA is based on variance, features with larger numeric scales will dominate the analysis unless all features are standardized first (mean 0, standard deviation 1).
Example: an unstandardized "income" feature (ranging into hundreds of thousands) would completely dominate PCA's identified directions of variance over a "years of experience" feature (ranging 0-40), unless both are standardized first.
Applied Walkthrough
1
A dataset includes both "income" (ranging into hundreds of thousands) and "age" (ranging 0-100) as features, and PCA is applied directly without any preprocessing.
2
Ask: will both features contribute fairly to the identified principal components? No — income's much larger numeric scale will dominate the variance calculations, effectively drowning out age's contribution.
3
This means the resulting principal components would mostly just reflect variance in income, largely ignoring the genuinely useful variance in age.
4
The fix: standardize both features (mean 0, standard deviation 1) before running PCA, ensuring each feature contributes proportionally to the identified directions of maximum variance.
Exam Application
Exams test whether you understand PCA's core goal (finding directions of maximum variance to reduce dimensionality while preserving structure), how to use a scree plot to choose the number of components to keep, and the critical importance of standardizing data first. Also expect a note distinguishing PCA (a linear technique usable for actual dimensionality reduction) from t-SNE and UMAP (nonlinear techniques used only for visualization).
⚠ Common Trap
The most common trap is applying PCA directly to unstandardized data with features on very different numeric scales. Without standardization, PCA's identified "directions of maximum variance" will simply reflect which features happen to have the largest numeric ranges, rather than genuinely meaningful structure in the data.
✓ Quick Self-Check
1. What does PCA find in the data?
Orthogonal directions of maximum variance, called principal components.
Tap to reveal / hide
2. What does PC1 represent?
The direction capturing the single greatest amount of variance in the data.
Tap to reveal / hide
3. What does a scree plot help you choose?
How many principal components (k) to keep, typically by finding where the variance-explained curve "elbows."
Tap to reveal / hide
4. Why must you standardize data before running PCA?
Because features with larger numeric scales would otherwise dominate the identified directions of variance, distorting the analysis.
Tap to reveal / hide
5. Are t-SNE and UMAP typically used for actual dimensionality reduction or just visualization?
Just visualization — they are nonlinear alternatives to PCA used primarily for visualizing high-dimensional data, not as a general-purpose dimensionality reduction step.
Tap to reveal / hide