Memory tricks that make computer vision and image AI click
From pixel to prediction -- these memory tricks lock in CNNs, object detection, image segmentation, generative vision models, and the architectures that gave machines the ability to see.
Or continue to the sub-topics below for more specialized Study Rooms and Forums
Computer Vision
Memory Tricks
Proven Mnemonics & Acronyms — fast to learn, hard to forget.
CV Fundamentals
PIXELS to FEATURES to PREDICTIONS -- the three levels of what a CNN sees
EDGES THEN SHAPES THEN OBJECTS -- LEARNED HIERARCHICALLY FROM DATA
Before deep learning: hand-crafted SIFT and HOG features. After: learned automatically from data.
Early layers detect low-level features (edges, corners, color gradients). Middle layers detect textures, shapes, and patterns. Deep layers detect semantic concepts (faces, cars, dogs). This hierarchy mirrors the human visual cortex. AlexNet (2012): first deep CNN to win ImageNet -- crushed all competitors and launched the deep learning revolution in computer vision.
Early layers
Edges, corners, color gradients -- low-level features
Middle layers
Textures, shapes, and patterns -- mid-level features
Deep layers
Semantic concepts: faces, cars, dogs -- high-level features
AlexNet 2012
First deep CNN win at ImageNet -- launched CV revolution
Object Detection
YOLO -- You Only Look Once -- detect all objects in one single forward pass
CLASSIFICATION = WHAT AND DETECTION = WHAT + WHERE AND SEGMENTATION = WHAT + WHICH PIXELS
IoU = Intersection over Union -- measures overlap between predicted and ground truth box
Classification: what is in this image. Detection: WHAT and WHERE -- bounding box plus class label for each object. Segmentation: which exact pixels belong to each object. YOLO: one-stage detector, divides image into grid, predicts boxes and classes simultaneously -- real-time speed. Faster R-CNN: two-stage, slower but more accurate. IoU: area of overlap divided by area of union -- threshold at 0.5 for acceptable detection.
Classification
Single label for whole image -- is there a cat?
Detection
Bounding box (x,y,w,h) + class label per object
Segmentation
Pixel-level mask per object -- most precise
YOLO
One forward pass, real-time speed -- slight accuracy tradeoff
U-Net Segmentation
U-NET looks like a U -- encoder compresses down, decoder expands back up with skip connections
ENCODER SHRINKS TO CAPTURE CONTEXT -- DECODER EXPANDS TO RECOVER SPATIAL DETAIL
Skip connections preserve fine spatial detail that is lost during max pooling
U-Net: contracting path (encoder) compresses spatial dimensions while increasing channels -- captures what is in the image. Expanding path (decoder) restores spatial resolution -- recovers where. Skip connections: copy encoder feature maps directly to corresponding decoder layer -- preserve fine spatial detail lost during downsampling. Dominant for medical image segmentation. SAM (Meta 2023): foundation model -- segment any object with a click.
Brightness, contrast, saturation -- color invariance
MixUp
Blend two images and labels -- smooth decision boundaries
AutoAugment
Learned augmentation policy -- often best performance
Vision Transformer
ViT -- split image into PATCHES, treat each patch like a TOKEN, run Transformer
AN IMAGE IS WORTH 16 by 16 WORDS -- THE PAPER THAT LAUNCHED VISION TRANSFORMERS
SWIN Transformer: hierarchical ViT with shifted windows -- efficient for detection and segmentation
ViT (Dosovitskiy et al., 2020): split image into 16x16 patches, flatten, project to embedding, add position embedding, run through standard Transformer encoder. Global context from layer 1 -- unlike CNNs that build context gradually. Needs more data than CNNs (no inductive bias for locality). SWIN Transformer: hierarchical ViT with shifted windows, efficient for high-resolution detection and segmentation.
Patch embedding
16x16 pixel patches treated like word tokens
Position embedding
Required -- Transformer has no inherent sense of spatial order
Global attention
Every patch attends to every other from layer 1
SWIN Transformer
Hierarchical + shifted windows -- efficient for dense prediction
Generative Vision
VAE encodes to SMOOTH space -- GAN has two fighters -- Diffusion REVERSES NOISE
THREE WAYS TO TEACH A MACHINE TO GENERATE REALISTIC IMAGES
Diffusion models now dominate -- more stable training, better diversity, easier text conditioning
VAE: encode image to distribution (mean and variance), sample, decode -- smooth latent space enables interpolation. GAN: Generator vs Discriminator, adversarial training, sharp results but mode collapse and instability. Diffusion: add noise over T steps, train to denoise, generate by starting from noise -- most stable training, best diversity, dominant for image generation (Stable Diffusion, DALL-E 3, Midjourney).
VAE
Smooth latent space, enables interpolation -- foundation of Stable Diffusion
GAN
Generator vs Discriminator -- sharp but unstable, mode collapse risk
Diffusion
Most stable, best quality, easiest text conditioning -- dominant now
ControlNet
Adds spatial conditioning (pose, depth, edges) to diffusion models
Face Recognition
EMBED then COMPARE -- face recognition maps faces to vectors then finds nearest match
DETECTION AND ALIGNMENT AND EMBEDDING AND COMPARISON -- FOUR STEP PIPELINE
ArcFace: current state of art -- additive angular margin loss for face embeddings
Pipeline: detect faces (MTCNN, RetinaFace), align using landmark points, encode face into compact vector (128 or 512 dimensions with ArcFace), compare via cosine similarity to known embeddings. Triplet loss: pull same-person embeddings together, push different people apart. Bias: higher error rates for dark-skinned women. EU AI Act bans real-time biometric surveillance in public spaces.
Detection
Find faces in image -- MTCNN, RetinaFace
Alignment
Normalize face orientation using landmark points
Embedding
CNN encodes identity into compact vector (ArcFace is state of art)
Bias
Higher error rates for dark-skinned women -- documented in MIT study
Autonomous Driving
SENSE then PERCEIVE then PLAN then ACT -- the four-stage AV pipeline
CAMERA AND LIDAR AND RADAR AND SENSOR FUSION -- THE SENSING DEBATE
SAE Level 2 is Tesla Autopilot -- human still monitors and is responsible
SAE Levels: L0 (none) to L5 (full autonomy any condition). L2: partial, human monitors (Tesla Autopilot). L4: high automation, limited geofenced area (Waymo One). Camera-only (Tesla) vs sensor fusion (Waymo). LiDAR: precise 3D depth, expensive. Camera: cheap, texture-rich, needs depth estimation. Radar: all-weather, velocity measurement. Long tail problem: rare edge cases too rare to appear in training data -- kills autonomous driving reliability.
SAE L2
Partial automation -- human monitors and is legally responsible
SAE L4
High automation -- limited geofenced area, Waymo in Phoenix and SF
Camera-only (Tesla)
Cost reduction -- camera provides texture but needs depth estimation
Long tail problem
Rare edge cases not in training data -- major unsolved challenge
Medical Imaging
READ the scan better than a doctor -- but only when told exactly what to look for
AI AUGMENTS RADIOLOGISTS -- AI PLUS PHYSICIAN OUTPERFORMS EITHER ALONE
Grad-CAM highlights which pixels most influenced the prediction -- required for clinical trust
Proven: diabetic retinopathy (matches ophthalmologist), CheXNet chest X-ray pneumonia detection, lymph node metastasis detection. Challenges: distribution shift (fails at hospitals not in training data), class imbalance (rare diseases), FDA approval required (510(k)). Grad-CAM: highlight influential image regions -- required for clinician trust. 3D U-Net: standard for CT and MRI volumetric segmentation.
Diabetic retinopathy
Google AI matches ophthalmologist accuracy
CheXNet
Stanford AI outperformed radiologists on pneumonia from chest X-rays
Distribution shift
Trained on teaching hospital data -- fails at rural clinics
Grad-CAM
Highlights influential pixels -- required for clinician trust
CV Metrics
mAP -- mean Average Precision -- the standard metric for object detection
AP = AREA UNDER PRECISION-RECALL CURVE FOR ONE CLASS
COCO mAP averages over IoU thresholds 0.5 to 0.95 -- much harder than Pascal VOC at 0.5 only
Classification: Top-1 accuracy, Top-5 accuracy. Object Detection: mAP -- compute AP (area under precision-recall curve) for each class, average across classes. COCO mAP: average over IoU thresholds 0.5 to 0.95 -- much harder than Pascal VOC (IoU=0.5 only). Segmentation: mIoU (mean Intersection over Union across all semantic classes). FPS: frames per second -- real-time requires 30+ FPS.
Top-1 accuracy
Is the top prediction the correct class?
mAP
Average AP across all classes and IoU thresholds
COCO vs VOC
COCO: IoU 0.5-0.95 (hard). VOC: IoU=0.5 only (easier).
mIoU
Mean Intersection over Union -- standard for segmentation
Optical Flow
FLOW = how pixels MOVE between frames -- a velocity field for every pixel in the video
LUCAS-KANADE IS CLASSICAL -- RAFT IS CURRENT STATE OF ART
Two-stream networks: spatial stream (RGB) plus temporal stream (optical flow) for action recognition
Optical flow: 2D vector (dx, dy) per pixel between consecutive frames showing apparent motion. Classical: Lucas-Kanade (local patch matching), Horn-Schunck (global smoothness). Deep learning: FlowNet, PWC-Net, RAFT (current state of art). Applications: video compression (encode motion), action recognition, video stabilization, slow-motion generation (interpolate frames), object tracking.
Optical flow output
(dx, dy) velocity vector per pixel between frames
Lucas-Kanade
Classical local patch matching -- assumes constant brightness
RAFT
Current state of art deep optical flow estimation
Two-stream networks
RGB appearance + optical flow motion for action recognition
🎯 Exam Favorite
OBJECT DETECTION = LOCATE + CLASSIFY — draw a box AND label what's inside it
BOUNDING BOX + CLASS LABEL
Object detection vs image classification
Image classification answers "What is in this image?" — one label for the whole image (cat). Object detection answers "What is where?" — draw bounding boxes around each object AND label each one (cat at top-left, dog at bottom-right, person in center). Detection is classification + localization. YOLO (You Only Look Once) is the famous real-time object detector — it processes the entire image in one pass, making it fast enough for video.
🧠 Vivid Story
IMAGE SEGMENTATION = COLORING BOOK — color every pixel the color of its object
PIXEL-LEVEL CLASSIFICATION
Semantic vs Instance segmentation
Segmentation goes beyond bounding boxes — it classifies every single pixel. Semantic segmentation: color all pixels of the same class the same color (all cars = red, all roads = gray, all sky = blue). Instance segmentation: distinguish individual objects of the same class (car #1 = bright red, car #2 = dark red). Used in self-driving cars (understand the entire scene pixel by pixel), medical imaging (outline a tumor precisely), and satellite imagery analysis.
🔑 Key Distinction
DATA AUGMENTATION = Teaching a kid to recognize a CAT even when it's upside down, sideways, or far away
FLIP · ROTATE · CROP · BRIGHTEN
Data augmentation — making small datasets go further
Computer vision models need massive training data. Data augmentation artificially expands the dataset by applying transformations: horizontal flip, rotation, random crop, brightness/contrast change, adding noise. A single cat photo becomes 20 training examples. The model learns that a cat flipped upside down is still a cat — improving robustness. This is why modern CV models generalize well even when real data is limited. Augmentation happens on-the-fly during training.
💡 Concept Anchor
FEATURE MAPS = X-RAY VISION — each filter reveals a different hidden pattern in the image
EACH FILTER = ONE PATTERN DETECTOR
Feature maps in CNNs — what the network actually sees
After each convolution layer, the network produces feature maps — one per filter. Each feature map highlights where a particular pattern (edge, curve, texture) appears in the image. Early layers: simple features (horizontal edges, vertical edges). Middle layers: shapes (curves, corners). Deep layers: complex parts (eyes, wheels, faces). Visualizing feature maps shows what the network has learned to see — like giving it X-ray goggles tuned to different patterns.
📅 Quick Reference
TRANSFER LEARNING IN CV = Borrowing EYES trained on millions of images and adapting them to your task
IMAGENET FEATURES → YOUR TASK
Transfer learning in computer vision — why you rarely train from scratch
Training a CNN from scratch needs millions of images. Instead: take ResNet, VGG, or EfficientNet pre-trained on ImageNet (1.2 million images, 1000 classes) — the early layers already detect edges, textures, and shapes. Freeze those layers. Replace the final classification layer with your own. Train only the last few layers on your small dataset. The borrowed eyes already know what visual features look like — you just teach them to recognize your specific objects.
⭐ Most Important
PIXELS → FEATURES → PREDICTIONS — the three-level hierarchy every CV model follows
LOW LEVEL · MID LEVEL · HIGH LEVEL
The three levels of what a CNN actually learns to see
Every CNN builds a visual hierarchy: Low-level features (edges, corners, colors, textures — detected by early layers, reusable across all tasks). Mid-level features (shapes, patterns, object parts — eyes, wheels, doors — detected by middle layers). High-level features (whole objects — faces, cars, buildings — detected by deep layers, task-specific). This hierarchy is why transfer learning works: the low and mid-level features trained on ImageNet transfer to almost any vision task.
Early layers
Edges, corners, textures — universal, transfer to any task
Middle layers
Shapes, patterns, object parts — semi-universal
Deep layers
Specific objects and concepts — task-specific, often replaced
🐍 Code
import torchvision.models as models resnet = models.resnet50(pretrained=True) # Freeze early layers (universal features) for param in list(resnet.parameters())[:-10]: param.requires_grad = False # Only train the final layers on your task
🎯 Exam Favorite
IoU = OVERLAP / UNION — the standard metric for how well your bounding box fits
INTERSECTION OVER UNION — OBJECT DETECTION ACCURACY
IoU and mAP — how object detectors are evaluated
IoU (Intersection over Union) measures bounding box quality: area where predicted and ground truth boxes overlap, divided by total area covered by both. IoU=1.0 = perfect match. IoU=0.5 = standard threshold for "correct detection." A detection is counted as correct only if IoU ≥ threshold AND class label matches. mAP (mean Average Precision) averages the AP score across all classes and IoU thresholds — the standard benchmark for object detectors (COCO, Pascal VOC).
IoU
Overlap area / Union area — 0 to 1, higher = better box placement
0.5 threshold
Standard: prediction "correct" if IoU ≥ 0.5 and class matches
mAP
Mean Average Precision — averaged across all classes and thresholds
Discriminative vs Generative computer vision tasks
Discriminative CV: given an image, output a label or bounding box. Classification (what?), detection (what and where?), segmentation (what at every pixel?). Generative CV: given a prompt or noise, create a new image. GANs, diffusion models, VAEs. The line is blurring: modern vision-language models (CLIP, DALL-E, Stable Diffusion) combine both — understanding AND generating images from the same representation space.
Discriminative
Image → label or structured output. Classification, detection, segmentation.
Generative
Noise or text → new image. GANs, diffusion, VAEs.
Vision-Language
Understand AND generate — CLIP, DALL-E, GPT-4V, Gemini Vision
🐍 Code
# Discriminative: classify an image from torchvision import models, transforms model = models.resnet50(pretrained=True); model.eval() # Generative: create an image from text from diffusers import StableDiffusionPipeline pipe = StableDiffusionPipeline.from_pretrained("runwayml/stable-diffusion-v1-5")
💡 Concept Anchor
NON-MAXIMUM SUPPRESSION = Delete DUPLICATE boxes — keep only the most confident one per object
THE CLEANUP STEP EVERY DETECTOR NEEDS
Why NMS is essential in object detection
Object detectors predict hundreds of overlapping bounding boxes around each object. Without cleanup, you get 50 boxes around the same car. Non-Maximum Suppression fixes this: sort all boxes by confidence score, keep the highest-confidence box, delete any remaining boxes with IoU > threshold against the kept box, repeat. Result: one clean box per object. NMS is the final step in YOLO, Faster R-CNN, and every real-time detector.
Step 1
Sort all predicted boxes by confidence score, highest first
Step 2
Keep the top box; delete all boxes with IoU > threshold against it
Step 3
Repeat until no boxes remain — one clean box per object
🐍 Code
# NMS in torchvision from torchvision.ops import nms import torch boxes = torch.tensor([[100,100,200,200],[105,105,205,205]], dtype=torch.float) scores = torch.tensor([0.9, 0.75]) keep = nms(boxes, scores, iou_threshold=0.5) # Returns indices of boxes to keep
0
Correct
0
Missed
0
Remaining
What does this mean / stand for?
0
Correct
0
Wrong
0
Remaining
🔗 Related Sub-Subjects
🔬 Deep Learning
CNNs, Vision Transformers, and diffusion models — the architectures powering computer vision.
Q: What is the difference between object detection and image segmentation?
A: Object detection: predict bounding boxes and class labels for each object — the output is a rectangle. Segmentation goes further: semantic segmentation classifies every single pixel (all car pixels = red, all road pixels = gray). Instance segmentation distinguishes individual objects of the same class (car #1 vs car #2). Detection is faster and sufficient for counting or locating. Segmentation is needed when exact shape matters — medical imaging, autonomous driving, satellite analysis.
Q: How does YOLO achieve real-time object detection?
A: YOLO (You Only Look Once) processes the entire image in a single forward pass — unlike two-stage detectors (Faster R-CNN) which first propose regions then classify them. YOLO divides the image into a grid, predicts bounding boxes and class probabilities simultaneously for all grid cells. This makes it dramatically faster (30-100+ FPS) at some cost to accuracy on small objects. YOLOv8 is the current state-of-the-art real-time detector.
Q: What is data augmentation and why is it essential for CV?
A: Data augmentation artificially expands the training dataset by applying transformations: horizontal flip, rotation, random crop, brightness/contrast change, Gaussian noise, cutout. A single image becomes 10-50 training examples. This teaches the model that a cat flipped or rotated is still a cat — building spatial invariance. Without augmentation, CV models severely overfit on small datasets. Always apply augmentation during training; never during validation or testing.
Q: Explain how a GAN generates photorealistic images.
A: A GAN has two networks in competition: Generator takes random noise and produces a fake image. Discriminator receives real and fake images and tries to tell them apart. Generator loss = how often the discriminator correctly identifies its output as fake. Discriminator loss = how often it is fooled. They compete: as the discriminator improves, the generator must produce more realistic fakes. After training converges, the generator creates images indistinguishable from real photos — never having seen the real images directly.
Q: What is optical flow in computer vision?
A: Optical flow estimates the motion of pixels between consecutive video frames — producing a vector field where each pixel has a velocity (direction and speed). Used in: video compression (motion compensation), action recognition (understanding movement), autonomous vehicles (tracking moving objects), and video stabilization. Classic methods: Lucas-Kanade. Deep learning: FlowNet, RAFT. The output is a 2D vector for every pixel representing how far and in which direction it moved.