Memory tricks that make machine learning algorithms click

Select the right algorithm for your problem type. Prepare data: clean, normalize, handle missing values, encode categoricals, split train/test. Learn: train the model -- algorithm finds optimal parameters. Interpret: examine what was learned. Test: evaluate on held-out test set -- never seen during training. Performance on test data is the only honest measure of model quality.

Linear Regression: y = mx + b, minimizes squared errors, predicts continuous values (house price, temperature). Logistic Regression: despite the name, used for CLASSIFICATION -- uses sigmoid function to output probability 0-1. Decision boundary at 0.5. Above = positive class. Below = negative class. This distinction appears on virtually every ML exam.

Decision Trees split data by feature questions using Gini impurity or Information Gain. Prone to overfitting alone. Random Forest (bagging): parallel trees on random data and feature subsets -- reduces variance. XGBoost (boosting): sequential trees, each corrects previous errors -- dominant for tabular data competitions. No feature scaling needed for tree-based methods.

Find K closest training examples (Euclidean distance), take majority vote (classification) or average (regression). Always normalize -- large-range features dominate distance. Curse of dimensionality: too many features makes distance meaningless. Choose K with cross-validation. K=1 overfits. Large K underfits.

SVM finds the hyperplane that maximally separates classes. Wider margin = better generalization. Kernel trick: maps data to higher dimensions where linearly separable -- RBF kernel handles nonlinear data. C parameter: low C = wider margin, more misclassifications tolerated. High C = narrow margin, fewer misclassifications. Best for high-dimensional sparse data.

Steps: choose K, randomly initialize K centroids, assign each point to nearest centroid, recalculate centroids as cluster means, repeat until stable. Elbow method: plot inertia vs K, choose where improvement slows. DBSCAN alternative: finds arbitrary-shaped clusters and outliers automatically without specifying K.

Normalization (Min-Max): scales to 0-1. Standardization (Z-score): mean=0, std=1. Scale for: K-NN, SVM, neural networks, PCA. NOT for: tree-based methods. One-hot encoding: categoricals to binary columns. Label encoding: integers to categories -- only when ordinal. PCA: reduce dimensions by keeping directions of maximum variance. Golden rule: fit transformers on training data only.

P(class|features) proportional to P(class) times product of P(feature|class). Despite independence assumption being almost always violated, it works well for text -- word frequencies are roughly independent given class. Gaussian NB: continuous features. Multinomial NB: word counts. Laplace smoothing: add 1 to all counts to prevent zero probabilities.

Agent: learner. State: current situation. Action: choice made. Reward: feedback signal. Policy: strategy mapping states to actions. Goal: maximize expected cumulative reward. Exploration vs exploitation. Q-learning: learns value of state-action pairs. RLHF (Reinforcement Learning from Human Feedback): human raters rank outputs, reward model trained, RL maximizes it -- standard for aligning LLMs.

Trend: long-term direction. Seasonality: regular repeating patterns (daily, weekly, yearly). Autocorrelation: current value correlates with its own past values. Cycle: irregular multi-year patterns. NEVER shuffle before splitting -- always chronological. Classic models: ARIMA. Modern: LSTMs, Temporal CNNs, Transformer-based forecasters.

Isolation Forest: anomalies isolated by fewer random splits (short path = anomaly) -- fast, scalable, no distance metric needed. Statistical: z-score, flag beyond N standard deviations. Autoencoder: train on normal data only, flag high reconstruction error points as anomalies. Applications: fraud detection, network intrusion, manufacturing defects, medical outliers.

AutoML tools: Google AutoML (cloud, no-code), H2O AutoML (open-source), Auto-sklearn. NAS (Neural Architecture Search): discovered EfficientNet -- outperformed human-designed networks. Active Learning: model queries the most uncertain examples for labeling -- reduces labeling cost by 5-10x. Limitations: still requires clean data, correct evaluation metric, good problem definition.

Supervised learning: every training example has a label (the right answer). The model learns to predict labels. Unsupervised learning: no labels at all — the model finds hidden structure on its own. Think of it as school (supervised — teacher gives answers) vs exploration (unsupervised — figure it out yourself). The label is the dividing line between them.

Imagine you're blindfolded on a hilly landscape. Your goal is to reach the lowest valley (minimum loss). You feel the slope under your feet and take a small step downhill. Repeat thousands of times. That's gradient descent — the model checks which direction reduces error, takes a small step that direction, and repeats until it can't go lower. Learning rate = how big each step is. Too big = overshoot. Too small = takes forever.

Reinforcement learning has three parts: an Agent (the dog/AI), an Environment (the world it acts in), and a Reward signal (treat or punishment). The agent takes actions, receives rewards or penalties, and learns which actions maximize total reward over time. No labels needed — just trial, error, and feedback. This is how AlphaGo beat world chess champions and how robots learn to walk.

K-fold cross-validation splits data into K equal chunks. In each round, one chunk is the test set and the rest train the model. Rotate K times so every chunk gets tested once. Average the K scores for a reliable estimate. Why? One test split might be lucky or unlucky. K-fold gives a fair average across many splits. K=5 or K=10 are standard. Think of it as rotating who grades your exam to get a fair average score.

A single decision tree can overfit badly. A Random Forest grows hundreds of trees on random data subsets and averages their predictions. More diverse opinions = more stable result. This is ensemble learning — combining many weak learners into one strong learner. Bagging (Random Forest) averages parallel models. Boosting (XGBoost) trains models sequentially, each fixing the last one's errors. Either way: the forest always beats the tree.

Picture a dartboard. Bias = how far from the bullseye your average shot lands (systematic error — wrong aim). Variance = how spread out your shots are (inconsistency). High Bias = underfitting — model too simple, misses the pattern every time. High Variance = overfitting — model too complex, nails training data but wild on new data. The goal: low bias AND low variance. Increasing model complexity reduces bias but raises variance — the unavoidable tradeoff.

Overfitting: training accuracy very high, validation accuracy much lower — the model memorized the training set including its noise. Like a student who memorized every practice exam verbatim but can't handle new questions. Underfitting: both training AND validation accuracy are low — the model hasn't learned the pattern at all. The diagnostic: plot learning curves. A big gap between training and validation = overfitting. Both curves low and flat = underfitting.

Bagging (Bootstrap Aggregating): trains many models IN PARALLEL on random data subsets, averages their predictions. Reduces variance. Random Forest is the gold standard. Boosting: trains models SEQUENTIALLY — each new model focuses on examples the previous one got wrong. Reduces bias. XGBoost, LightGBM, and AdaBoost are the most powerful implementations. Rule of thumb: Random Forest when you need speed and reliability. XGBoost when you need maximum accuracy on tabular data.

Regularization adds a penalty to the loss function for large weights — taxing the model for complexity. L1 (Lasso): penalty = sum of absolute values of weights. Drives unimportant weights to exactly zero — built-in feature selection. L2 (Ridge): penalty = sum of squared weights. Shrinks all weights toward zero but rarely to exactly zero. Elastic Net combines both. Use L1 when you suspect only a few features matter. Use L2 when all features contribute something. Lambda controls the penalty strength.