Step by Step
B
Batch size — how many examples per training step
Determines how many training examples are processed together before the model's weights are updated. Typical values range from 32 to 128.
Example: a batch size of 64 means the model computes an average gradient across 64 examples before making one weight update.
L
Layers & Learning rate — the two most consequential settings
The number of layers determines network depth, while the learning rate (widely considered the single most critical hyperparameter) controls how large each weight update step is. A common starting point for the Adam optimizer is a learning rate of 0.001.
Example: starting a new project with Adam at the default learning rate of 0.001 is a standard, reasonable first choice before any tuning.
A
Activation functions — chosen per layer type
The choice of activation function (typically ReLU for hidden layers, sigmoid/softmax for output layers) is itself a hyperparameter decision made before training begins.
Example: choosing ReLU for all hidden layers and softmax for a 10-class output layer, decided before any training starts.
N
Neurons per layer — network width
The number of neurons in each layer determines the network's width, affecting its capacity to represent complex patterns.
Example: choosing 256 neurons for a hidden layer rather than 64, giving the network more representational capacity at the cost of more parameters to train.
D
Dropout rate — how aggressively to regularize
The percentage of neurons randomly disabled during training (typically 20-50%) is set as a hyperparameter before training begins, balancing regularization against underfitting.
Example: choosing a 30% dropout rate as a starting point, then adjusting based on whether the model overfits or underfits.
Applied Walkthrough
1
A team is setting up training for a new model and needs to decide on several settings before training even starts.
2
They choose a batch size of 64, three hidden layers of 128 neurons each with ReLU activation, a dropout rate of 30%, and Adam with a 0.001 learning rate.
3
None of these values were learned from the data — they were all chosen by the developers ahead of time, based on common defaults and prior experience.
4
If training doesn't go well, the learning rate is typically the first hyperparameter worth investigating and adjusting, given how heavily final performance depends on it.
Exam Application
Exams test whether you understand hyperparameters are set BEFORE training by the developer, as opposed to model parameters (weights) which are learned FROM the data during training. Also expect questions on tuning methods — random search often outperforms grid search for the same computational budget, and transfer learning (starting from a pretrained model) dramatically helps with small datasets.
⚠ Common Trap
The most common trap is confusing hyperparameters (developer-chosen settings like learning rate and batch size) with parameters (the weights the model learns from data during training). These are fundamentally different categories, and mixing them up is a frequently tested error.
✓ Quick Self-Check
1. What is the difference between a hyperparameter and a parameter?
Hyperparameters are set by the developer before training begins; parameters (weights) are learned from the data during training.
Tap to reveal / hide
2. Which hyperparameter is considered the single most critical one to get right?
The learning rate.
Tap to reveal / hide
3. What is a common starting learning rate for the Adam optimizer?
0.001.
Tap to reveal / hide
4. What tuning method often outperforms grid search for the same computational budget?
Random search.
Tap to reveal / hide
5. What does transfer learning help with, especially for small datasets?
It dramatically improves performance by starting from a pretrained model rather than training from scratch.
Tap to reveal / hide