Memory tricks that make search, optimization and classic AI click
From A-star search to genetic algorithms -- these memory tricks lock in the classical AI algorithms, optimization methods, and ensemble techniques that power modern AI systems.
Or continue to the sub-topics below for more specialized Study Rooms and Forums
AI Algorithms
Memory Tricks
Proven Mnemonics & Acronyms — fast to learn, hard to forget.
Search Algorithms
A* = g(n) + h(n) -- actual cost PLUS heuristic estimate to goal
BFS AND DFS AND UNIFORM COST AND A-STAR
A-star is optimal if the heuristic is admissible -- never overestimates the true cost
BFS (queue, level-by-level): complete and optimal for unweighted graphs. DFS (stack, goes deep first): memory efficient but not optimal. Uniform Cost Search: optimal for weighted graphs. A* Search: f(n) = g(n) + h(n). g(n) = actual cost from start. h(n) = heuristic estimate to goal. Optimal if heuristic is admissible (never overestimates). Used in GPS navigation and game pathfinding.
BFS
Queue (FIFO), level-by-level, complete, optimal for unweighted
DFS
Stack (LIFO), memory efficient, NOT optimal
A* f(n)=g(n)+h(n)
Optimal if heuristic admissible (never overestimates)
Admissible examples
Straight-line distance, misplaced tiles in 8-puzzle
Minimax
MINIMAX -- MAXimize your score, MINimize opponent score at alternating levels
ASSUMES OPTIMAL OPPONENT -- ALWAYS PLAN FOR WORST CASE
Alpha-beta pruning eliminates branches that cannot affect the final decision -- free speed
Minimax for two-player zero-sum games (chess, tic-tac-toe). MAX player (you) chooses action maximizing score. MIN player (opponent) minimizes your score. Alternate MAX and MIN levels. Alpha-Beta Pruning: prune branches where alpha is >= beta -- same result, dramatically fewer nodes. Reduces from O(b^d) to O(b^(d/2)) in best case. AlphaZero: minimax plus deep learning plus self-play -- beat world champions.
Minimax
Alternate MAX (your move) and MIN (opponent move) levels
Alpha-beta pruning
Prune when alpha >= beta -- same result, far fewer nodes
CSP = Variables, Domains, Constraints must all be satisfied simultaneously
SUDOKU AND MAP COLORING AND SCHEDULING ARE ALL CSPs
MRV heuristic -- assign the most constrained variable first -- fails early and saves time
Variables (cells, regions), Domains (digits 1-9, colors), Constraints (no repeats, adjacent regions differ). Backtracking: assign one variable at a time, backtrack when constraint violated. MRV (Minimum Remaining Values): assign most constrained variable first. AC-3 Arc Consistency: preprocessing that removes domain values with no compatible assignment.
Variables
What needs to be assigned (cells, regions, time slots)
Domains
Possible values for each variable
Constraints
Rules that must hold (no repeats, adjacency, order)
MRV heuristic
Assign most constrained variable first -- catches failures early
Genetic Algorithms
SELECT, CROSS, MUTATE -- evolution as an optimization algorithm
POPULATION AND FITNESS AND SELECTION AND CROSSOVER AND MUTATION
Mutation maintains diversity -- without it the population converges to a local optimum
Population: candidate solutions. Fitness function: evaluate solution quality. Selection: fitter solutions more likely to reproduce. Crossover: combine two parents to produce offspring. Mutation: randomly alter part of a solution -- maintains diversity, avoids local optima. Repeat for many generations. Used when search space is huge or gradient is unavailable.
Population
Set of candidate solutions
Fitness function
Evaluates how good each solution is
Crossover
Combine parents -- offspring inherits from both
Mutation
Random change -- prevents convergence to local optima
Bayesian Networks
BAYES NET -- probabilistic graphical model where ARROWS MEAN CAUSES
DIRECTED ACYCLIC GRAPH WITH CONDITIONAL PROBABILITY TABLES
Given its parents, a node is conditionally independent of all non-descendants
Directed Acyclic Graph (DAG). Nodes = random variables. Edges = direct probabilistic influence (X to Y means X directly influences Y). Each node stores a Conditional Probability Table (CPT) given its parents. Applications: medical diagnosis, spam filtering, fault diagnosis. Inference: given evidence, compute probability of query variable. Inference is NP-hard in general.
Nodes
Random variables (symptoms, diseases, sensor readings)
Edges
X to Y means X directly influences Y
CPT
P(node|parents) -- conditional probability table at each node
Inference
Given evidence, compute P(query) -- NP-hard in general
Gradient Descent
GRADIENT DESCENT goes DOWNHILL -- like a ball rolling to the lowest point
BATCH AND SGD AND MINI-BATCH AND ADAM -- FOUR VARIANTS
Learning rate is the most critical hyperparameter -- too high diverges, too low is very slow
Batch GD: gradient on entire dataset -- stable, slow. SGD: gradient on one example -- fast, very noisy. Mini-batch (32-256 samples): standard in deep learning -- best of both worlds. Adam: adaptive per-parameter learning rate -- most popular optimizer, works with defaults (lr=0.001). Momentum: adds inertia, avoids oscillation, escapes saddle points.
Batch GD
Entire dataset per update -- stable but slow
SGD
One example per update -- fast, noisy, can escape local optima
Mini-batch (32-256)
Standard in deep learning -- best compromise
Adam
Adaptive per-parameter LR -- use by default, start lr=0.001
Ensemble Methods
BAG and BOOST -- Bagging = parallel, Boosting = sequential error correction
RANDOM FOREST (BAGGING) AND XGBOOST (BOOSTING) AND STACKING
Ensembles consistently outperform single models -- they win Kaggle competitions
Bagging: parallel independent models on random subsets, average predictions -- reduces variance. Random Forest = bagging of decision trees. Boosting: sequential, each model corrects previous errors -- reduces bias. XGBoost and LightGBM = gradient boosting. Stacking: meta-model learns to combine base model outputs. No Free Lunch: try ensembles before complex single models.
Bagging of decision trees -- reliable off-the-shelf algorithm
XGBoost
Gradient boosting -- most powerful for tabular data
PCA
PCA finds the DIRECTIONS OF MAXIMUM VARIANCE in the data
REDUCE DIMENSIONS WHILE PRESERVING THE MOST IMPORTANT STRUCTURE
Always standardize data before PCA -- large-scale features will otherwise dominate
PCA finds orthogonal directions (principal components) of maximum variance. PC1: greatest variance direction. Project data onto top k components. Scree plot: variance explained vs k -- choose where curve elbows (~95% variance kept). Always standardize first. t-SNE and UMAP: nonlinear alternatives for visualization only.
PC1
Direction of greatest variance in the data
Scree plot
Variance explained vs k -- pick the elbow (~95% total)
Standardize first
Large-scale features dominate without standardization
t-SNE and UMAP
Nonlinear -- visualization only, not for feature extraction
Monte Carlo
SAMPLE then AVERAGE -- Monte Carlo estimates by running many random simulations
LAW OF LARGE NUMBERS: RANDOM SAMPLE AVERAGE CONVERGES TO TRUE VALUE
Error decreases as 1 over sqrt(N) -- quadruple samples to halve the error
Run many random simulations, average results -- converges to true value as samples increase. Monte Carlo Tree Search (MCTS): simulate random game playouts from each position to guide search -- used in AlphaGo. MCMC (Markov Chain Monte Carlo): sample from complex probability distributions -- used in Bayesian inference.
Core idea
Many random samples, take average, converges to true value
MCTS
Game AI -- simulate playouts to estimate position value
MCMC
Sample from complex distributions -- Bayesian inference
Error rate
Decreases as 1/sqrt(N) -- 4x samples to halve error
Association Rules
SUPPORT, CONFIDENCE, LIFT -- the three metrics that make bread-to-butter meaningful
MARKET BASKET ANALYSIS -- ITEMS THAT FREQUENTLY APPEAR TOGETHER
Apriori uses anti-monotonicity -- if A,B is infrequent then A,B,C must also be infrequent
Support: how often do X and Y appear together? Confidence (A to B): P(B given A). Lift: is the relationship stronger than chance? Lift greater than 1 = positive association beyond chance. Apriori: uses anti-monotonicity to prune itemset search efficiently. Applications: recommendations, fraud detection, drug interactions.
Support
Frequency of itemset -- how common is the pair?
Confidence (A to B)
P(B|A) -- given A, how often is B also present?
Lift greater than 1
Items appear together more than expected by chance
Apriori
Prune by anti-monotonicity -- efficient frequent itemset mining
Dynamic Programming
DIVIDE into subproblems, STORE solutions, REUSE -- never solve the same subproblem twice
OPTIMAL SUBSTRUCTURE PLUS OVERLAPPING SUBPROBLEMS = USE DP
Viterbi algorithm uses DP for HMMs -- speech recognition and POS tagging both use it
DP solves complex problems by breaking into overlapping subproblems, solving each once, storing results (memoization). Two conditions: optimal substructure and overlapping subproblems. Top-down (memoization): recursive with caching. Bottom-up (tabulation): iterative. Viterbi algorithm: DP for finding most likely sequence in a Hidden Markov Model -- used in speech recognition and POS tagging.
Optimal substructure
Optimal solution contains optimal sub-solutions
Overlapping subproblems
Same subproblems solved repeatedly in naive recursion
Memoization
Store results -- top-down recursive approach
Viterbi
DP for HMM sequences -- speech recognition and POS tagging
🎯 Exam Favorite
K-NN = Ask your K NEAREST NEIGHBORS and go with the majority vote
DEMOCRACY BY DISTANCE
K-Nearest Neighbors — the simplest classifier
K-NN is beautifully simple: to classify a new point, find the K training examples closest to it (by distance) and let them vote. Majority wins. K=1 means one nearest neighbor decides. K=5 means the 5 closest points vote. Large K = smoother boundary (less overfit). Small K = jagged boundary (more overfit). No training phase — just store all data and compute distances at prediction time. Lazy learning at its finest.
🧠 Vivid Story
SVM = Finding the WIDEST STREET between two neighborhoods
MAXIMUM MARGIN HYPERPLANE
Support Vector Machine — the widest street metaphor
SVM finds the boundary line (hyperplane) that separates two classes with the maximum margin — the widest possible street between the two neighborhoods. The data points closest to the boundary are called support vectors (the houses on the edge of the street). Wider street = more confident separation = better generalization. The kernel trick lets SVM handle data that can't be separated by a straight line by projecting it into higher dimensions.
🔑 Key Distinction
CLASSIFICATION predicts a CATEGORY — REGRESSION predicts a NUMBER
CAT OR NUMBER — THE FUNDAMENTAL OUTPUT TYPE
Classification vs Regression — never confuse them again
The output type determines which algorithm you need. Classification: output is a discrete category — spam/not spam, cat/dog, disease/healthy. Regression: output is a continuous number — house price, temperature tomorrow, patient age. Same input data, completely different question. Logistic regression (confusingly named) is a classifier. Linear regression predicts numbers. Always ask: am I predicting a category or a value?
💡 Concept Anchor
DECISION TREE = Playing 20 QUESTIONS — each question splits the possibilities
ASK · SPLIT · REPEAT UNTIL PURE
Decision trees — the 20 questions algorithm
A decision tree asks a series of yes/no questions about features, splitting the data at each step. Each split is chosen to maximize information gain — making the resulting groups as pure as possible. Terminal nodes (leaves) contain the final prediction. It's exactly like 20 Questions: "Is it alive? Does it have legs? Does it bark?" The tree stops when leaves are pure or a depth limit is reached. Interpretable but prone to overfitting without pruning.
📅 Quick Reference
K-MEANS = Assign to nearest CENTROID, move centroid, repeat until stable
THREE STEPS: ASSIGN · UPDATE · CONVERGE
K-means clustering — the three-step dance
K-means finds K clusters in unlabeled data: Step 1 — randomly place K centroids. Step 2 — assign every point to its nearest centroid. Step 3 — move each centroid to the average position of its assigned points. Repeat steps 2–3 until centroids stop moving. The algorithm converges to stable clusters. You must choose K in advance (a weakness). Great for customer segmentation, document grouping, and image compression.
⭐ Most Important
CHOOSE YOUR WEAPON — Classification, Regression, Clustering, or Dimensionality Reduction?
MATCH THE ALGORITHM TO THE TASK TYPE
The algorithm selection framework every student needs
Before choosing any algorithm, identify the task type. Classification: predict a discrete category (spam/not spam). Regression: predict a continuous number (house price). Clustering: group unlabeled data by similarity (K-means). Dimensionality Reduction: compress features while keeping structure (PCA). Getting this wrong means applying the wrong family of algorithms entirely. Always ask: What type of output do I need?
Continuous output — linear regression, ridge, lasso, gradient boosting
Clustering
Group unlabeled data — K-means, DBSCAN, hierarchical clustering
Dim reduction
Compress features — PCA, t-SNE, UMAP
🐍 Code
from sklearn.linear_model import LogisticRegression, LinearRegression from sklearn.tree import DecisionTreeClassifier from sklearn.cluster import KMeans from sklearn.decomposition import PCA # Pick based on task type — not familiarity
🎯 Exam Favorite
PCA = Find the AXES OF MAXIMUM SPREAD and project data onto them
PRINCIPAL COMPONENT ANALYSIS — COMPRESS WITHOUT LOSING SHAPE
PCA for dimensionality reduction — the exam staple
PCA finds the directions (principal components) along which the data varies the most, then projects the data onto those axes. PC1 captures the most variance, PC2 the next most (perpendicular to PC1), and so on. You then keep only the top K components, discarding the rest. Use case: 500 features → 10 principal components that capture 95% of variance. Speeds up training, reduces overfitting, enables visualization in 2D/3D.
PC1
Direction of maximum variance in the data
PC2
Direction of second-most variance, perpendicular to PC1
n_components
How many PCs to keep — choose by explained variance ratio
🐍 Code
from sklearn.decomposition import PCA pca = PCA(n_components=0.95) # keep 95% of variance X_reduced = pca.fit_transform(X_scaled) print(pca.explained_variance_ratio_) # variance per component
🔑 Key Distinction
DBSCAN finds BLOBS OF ANY SHAPE — K-means only finds ROUND BLOBS
DENSITY-BASED vs CENTROID-BASED CLUSTERING
When to use DBSCAN instead of K-means
K-means assumes clusters are roughly spherical and requires you to specify K upfront. DBSCAN (Density-Based Spatial Clustering) finds clusters of any shape — it groups points that are densely packed and labels sparse outliers as noise. No need to specify K. Perfect for: geographic data (cities form irregular shapes), anomaly detection (outliers labeled automatically), real-world messy clusters. Weakness: sensitive to the epsilon (neighborhood radius) hyperparameter.
K-means
Round blobs, must specify K, no outlier detection
DBSCAN
Any shape, auto-detects K, automatically labels outliers
Use DBSCAN when
Clusters are irregular shapes, or you need outlier detection
NAIVE BAYES = Fast, Simple, Surprisingly Powerful — especially for text
BAYES THEOREM APPLIED TO CLASSIFICATION
Why Naive Bayes works despite its naive assumption
Naive Bayes classifies by applying Bayes theorem, assuming all features are conditionally independent given the class. This assumption is almost never true — hence "naive." Yet it works remarkably well for text classification (spam filtering, sentiment analysis) because word frequencies are approximately independent. Training is instant — just count frequencies. Prediction is a few multiplications. Accuracy often rivals far more complex models on text data. Always try it as a baseline.
Assumption
All features independent given class — almost never true, often works anyway
Best for
Text classification — spam, sentiment, document categorization
Advantage
Extremely fast to train, works well with small data, handles high dimensions
🐍 Code
from sklearn.naive_bayes import MultinomialNB from sklearn.feature_extraction.text import CountVectorizer vec = CountVectorizer() X = vec.fit_transform(texts) nb = MultinomialNB().fit(X, labels)
0
Correct
0
Missed
0
Remaining
What does this mean / stand for?
0
Correct
0
Wrong
0
Remaining
🔗 Related Sub-Subjects
📊 Machine Learning
The ML framework that all these algorithms fit into — supervised, unsupervised, training pipelines.
A: The No Free Lunch theorem states that no single algorithm performs best across all possible problems. Every algorithm makes assumptions — and those assumptions help on some problems but hurt on others. This is why model selection matters: you must evaluate multiple algorithms on your specific data, not just pick the most popular one.
Q: What is the difference between parametric and non-parametric models?
A: Parametric models have a fixed number of parameters regardless of training data size — they summarize data into parameters (e.g., linear regression has one weight per feature). Non-parametric models grow with training data size — K-NN stores all training points, decision trees can grow as large as needed. Non-parametric models are more flexible but slower at prediction time.
Q: When would you use logistic regression over a decision tree?
A: Logistic regression: when you need a probabilistic output (confidence scores), when you need interpretability (feature weights), when data is linearly separable, when you have limited data. Decision tree: when relationships are non-linear, when you need if-then rules for human interpretation, when you have categorical features without encoding.
Q: What is feature scaling and which algorithms require it?
A: Feature scaling standardizes features to a common range. Distance-based algorithms require it: K-NN (distance calculations skewed by large-scale features), K-means (centroids pulled toward high-magnitude features), SVM (margin depends on feature scale), PCA (variance dominated by large-scale features). Tree-based algorithms (Random Forest, XGBoost) do NOT require scaling — they split on thresholds, not distances.
Q: What is the elbow method in K-means clustering?
A: The elbow method helps select K. Plot the within-cluster sum of squares (inertia) for K=1 through K=10. As K increases, inertia decreases. Find the K where the rate of decrease sharply changes — the "elbow." Beyond the elbow, adding more clusters gives diminishing returns. The elbow represents the best tradeoff between cluster compactness and model simplicity.