Step by Step
F
Flow vectors — a 2D motion field per pixel
Optical flow computes a 2D vector (dx, dy) for every pixel, representing its apparent motion between two consecutive video frames.
Example: a pixel on a moving car's bumper having an optical flow vector pointing in the direction the car is traveling, with magnitude reflecting its speed on screen.
C
Classical methods — Lucas-Kanade and Horn-Schunck
Lucas-Kanade estimates flow using local patch matching, assuming motion is consistent within a small neighborhood; Horn-Schunck instead assumes global smoothness across the whole image.
Example: Lucas-Kanade tracking a specific small patch of pixels frame to frame, assuming that patch's motion is locally consistent.
D
Deep learning methods — FlowNet, PWC-Net, RAFT
Modern deep learning approaches to optical flow include FlowNet, PWC-Net, and RAFT, with RAFT representing the current state of the art.
Example: RAFT computing highly accurate optical flow between two video frames, outperforming classical methods like Lucas-Kanade.
⭐
Two-stream networks — combining spatial and temporal information
Two-stream networks combine a spatial stream (processing regular RGB frames) with a temporal stream (processing optical flow), specifically to improve action recognition by incorporating explicit motion information alongside static appearance.
Example: a two-stream network recognizing the action "running" by combining what a person looks like in a single RGB frame (spatial stream) with how their body parts are moving across frames (temporal/optical-flow stream).
Applied Walkthrough
1
A team is building an action recognition system that needs to distinguish between similar-looking but differently-moving actions, like "waving" versus "reaching."
2
Using only static RGB frames (a spatial stream alone), the system might struggle to distinguish these actions if a single frame looks similar for both.
3
Adding a temporal stream, processing optical flow computed with a modern method like RAFT, provides explicit motion information showing exactly how body parts are moving across frames.
4
Combining both streams in a two-stream network lets the system leverage both static appearance (spatial stream) and explicit motion patterns (temporal stream) together, significantly improving action recognition accuracy for motion-dependent distinctions.
Exam Application
Exams test whether you understand what optical flow actually represents (a 2D motion vector per pixel between frames) and whether you can distinguish classical methods (Lucas-Kanade, Horn-Schunck) from modern deep learning methods (FlowNet, PWC-Net, RAFT). Also expect a question on why two-stream networks combine spatial and temporal information specifically for action recognition.
⚠ Common Trap
The most common trap is assuming a single RGB frame alone provides sufficient information for action recognition. Some actions are difficult or impossible to distinguish from a single static frame — the whole point of the temporal (optical flow) stream is to provide explicit motion information that static appearance alone cannot capture.
✓ Quick Self-Check
1. What does optical flow represent for a given pixel?
A 2D vector (dx, dy) representing that pixel's apparent motion between two consecutive frames.
Tap to reveal / hide
2. Name one classical optical flow method.
Lucas-Kanade (local patch matching) or Horn-Schunck (global smoothness).
Tap to reveal / hide
3. What deep learning method represents the current state of the art for optical flow?
RAFT.
Tap to reveal / hide
4. What two streams does a two-stream network combine?
A spatial stream (RGB frames) and a temporal stream (optical flow).
Tap to reveal / hide
5. Why is the temporal (optical flow) stream specifically useful for action recognition?
It provides explicit motion information that a single static RGB frame alone cannot capture, helping distinguish similar-looking but differently-moving actions.
Tap to reveal / hide