Step by Step
E
Encoder (contracting path) — capturing what's in the image
The encoder path compresses spatial dimensions while increasing the number of channels, capturing increasingly abstract information about WHAT is present in the image, at the cost of losing precise spatial detail.
Example: the encoder progressively shrinking a 256x256 image down through several pooling layers while extracting increasingly abstract feature representations.
D
Decoder (expanding path) — recovering where things are
The decoder path restores spatial resolution step by step, recovering WHERE things are located within the image, working back up toward the original image dimensions.
Example: the decoder progressively upsampling the compressed representation back toward the original 256x256 resolution, recovering precise spatial location information.
S
Skip connections — preserving fine spatial detail
Skip connections copy encoder feature maps directly across to their corresponding decoder layer, preserving fine spatial detail that would otherwise be lost during the encoder's downsampling (max pooling) process.
Example: a skip connection directly passing high-resolution edge detail from an early encoder layer straight to the corresponding decoder layer, preserving precision that pure upsampling alone would lose.
⭐
SAM — Meta's foundation model for segmentation
SAM (Segment Anything Model, Meta 2023) is a foundation model that can segment any object in an image with just a simple click, representing a major generalization of segmentation beyond task-specific trained models like U-Net.
Example: using SAM to segment an arbitrary, previously-unseen object type in a photo with a single click, without needing to train a dedicated segmentation model for that specific object category.
Applied Walkthrough
1
A medical imaging team needs to segment tumors precisely at the pixel level from MRI scans, requiring both broad contextual understanding and fine spatial precision.
2
U-Net's encoder path compresses the image, capturing increasingly abstract contextual information about what tissue types and structures are present.
3
The decoder path then expands back up, recovering spatial resolution — but without help, this process alone would lose fine spatial detail during the encoder's earlier downsampling.
4
Skip connections solve this by directly passing high-resolution detail from corresponding encoder layers to the decoder, which is exactly why U-Net has become the dominant architecture for medical image segmentation specifically, where fine spatial precision genuinely matters.
Exam Application
Exams test whether you understand U-Net's core "U" shape (encoder compresses for context, decoder expands for spatial resolution) and specifically why skip connections are necessary — to preserve fine spatial detail otherwise lost during the encoder's downsampling/max pooling process.
⚠ Common Trap
The most common trap is assuming the decoder alone (without skip connections) could adequately recover the fine spatial detail lost during encoding. Skip connections are specifically necessary because pure upsampling in the decoder cannot recover detail that was destroyed during max pooling — the skip connections provide a direct pathway for that lost detail to reach the decoder.
✓ Quick Self-Check
1. What does the encoder (contracting) path of U-Net do?
Compresses spatial dimensions while increasing channels, capturing what is in the image.
Tap to reveal / hide
2. What does the decoder (expanding) path of U-Net do?
Restores spatial resolution, recovering where things are located in the image.
Tap to reveal / hide
3. What problem do skip connections specifically solve?
They preserve fine spatial detail that is otherwise lost during the encoder's downsampling (max pooling) process.
Tap to reveal / hide
4. What domain is U-Net particularly dominant in, and why?
Medical image segmentation, because fine spatial precision genuinely matters for that application.
Tap to reveal / hide
5. What is SAM (Segment Anything Model), and who created it?
A foundation model from Meta (2023) that can segment any object with just a click, generalizing beyond task-specific trained models.
Tap to reveal / hide