Step by Step
B
Bayes' theorem — the mathematical foundation
Naive Bayes computes the probability of a class given the observed features as proportional to the class's overall probability multiplied by the product of each feature's probability given that class.
Example: the probability an email is spam given its words is proportional to how common spam is overall, times how likely each individual word is to appear in spam emails.
N
The naive independence assumption — technically wrong, practically useful
Naive Bayes assumes every feature is independent of every other feature given the class — an assumption that is almost always technically violated in real data, yet the algorithm still performs surprisingly well, especially for text.
Example: in a spam email, the words "free" and "money" are not truly independent (they tend to co-occur), but treating them as independent still produces useful spam classifications.
V
Variants — Gaussian and Multinomial
Gaussian Naive Bayes handles continuous numeric features; Multinomial Naive Bayes handles discrete counts, like word frequency counts in text classification.
Example: classifying emails by word-count frequency uses Multinomial Naive Bayes, while classifying based on continuous sensor readings would use Gaussian Naive Bayes.
E
Enable with Laplace smoothing — avoiding zero probabilities
Adding 1 to every count (Laplace smoothing) prevents a single unseen word/feature combination from forcing an entire probability calculation to zero.
Example: without smoothing, a brand-new word never seen in spam training emails would make the entire probability calculation zero for any message containing it, even if every other word strongly suggests spam.
Applied Walkthrough
1
A spam filter using Naive Bayes encounters a word it has never seen in any training email.
2
Without correction, multiplying by a probability of exactly zero for that unseen word would zero out the entire prediction, regardless of other strong evidence.
3
Laplace smoothing fixes this by adding 1 to every count before calculating probabilities, ensuring no probability is ever exactly zero.
4
This lets the model still make a sensible prediction even when it encounters words or features it never saw during training.
Exam Application
Exams test whether you understand the independence assumption is technically false yet still practically useful, particularly for text classification where word frequencies are roughly independent given the class. Also expect questions distinguishing Gaussian Naive Bayes (continuous features) from Multinomial Naive Bayes (count-based features like word frequencies).
⚠ Common Trap
The most common trap is assuming the naive independence assumption being technically wrong means the algorithm doesn't work well in practice. It does — especially for text classification — precisely because the violations of independence tend to be mild enough not to destroy the overall probability ranking.
✓ Quick Self-Check
1. What does Naive Bayes "naively" assume about features?
That all features are independent of each other given the class — an assumption that is usually technically false.
Tap to reveal / hide
2. Why does Naive Bayes still work well for text classification despite this false assumption?
Because word frequencies are roughly independent given the class, so the assumption's violations are usually mild enough not to ruin predictions.
Tap to reveal / hide
3. When would you use Multinomial Naive Bayes instead of Gaussian Naive Bayes?
When features are discrete counts, like word frequencies in text, rather than continuous numeric values.
Tap to reveal / hide
4. What problem does Laplace smoothing solve?
It prevents a single unseen feature value from forcing the entire probability calculation to zero.
Tap to reveal / hide
5. What does Bayes' theorem compute in this context?
The probability of a class given the observed features, proportional to the class's overall probability times the product of each feature's probability given that class.
Tap to reveal / hide