Step by Step
D
Disparate impact — neutral on paper, harmful in practice
Disparate impact occurs when a facially neutral algorithm produces outcomes that disproportionately harm a protected group, even though the algorithm never explicitly uses the protected characteristic itself.
Example: a lending algorithm that never uses race as an input still producing loan approval outcomes that disproportionately disadvantage a particular racial group.
Z
ZIP code as a racial proxy — the classic example
An algorithm using ZIP code as a feature can effectively "learn" a racial proxy without ever seeing race explicitly, since ZIP code correlates strongly with race due to historical redlining practices.
Example: an algorithm using ZIP code to help predict loan risk inadvertently learning patterns that closely track racial demographics, purely because historical redlining created strong correlations between ZIP code and race.
P
Other common proxies — arrest record, education, name
Beyond ZIP code, other seemingly neutral features can also serve as demographic proxies: arrest record (correlating with policing bias), educational institution (correlating with socioeconomic status), and name (correlating with ethnicity).
Example: an algorithm using a job applicant's name as an input feature inadvertently learning ethnicity-correlated patterns, even though ethnicity itself was never an explicit input.
A
Auditing for disparate impact — testing outputs, not just inputs
Properly auditing for disparate impact requires testing the model's actual OUTPUTS across different demographic groups, not just inspecting which input features the model uses — a model can look neutral on its input list while still producing discriminatory outcomes.
Example: an auditor checking a lending algorithm's actual approval rates across different demographic groups, rather than just checking whether race was included as an explicit input feature.
Applied Walkthrough
1
A lending algorithm is designed to explicitly exclude race as an input feature, and its developers believe this makes it free of racial bias.
2
The algorithm does, however, use ZIP code as one of its input features to help assess loan risk.
3
Because ZIP code correlates strongly with race due to historical redlining, the algorithm effectively learns a racial proxy through this seemingly neutral feature, without race ever being an explicit input.
4
An audit that only checked the model's list of input features (confirming race wasn't explicitly included) would miss this problem entirely — properly detecting disparate impact requires testing the model's actual output patterns across demographic groups, not just inspecting its input feature list.
Exam Application
Exams test whether you understand disparate impact as distinct from explicit, intentional discrimination — a facially neutral algorithm can still produce disproportionately harmful outcomes for a protected group. Also expect you to identify common proxy variables (ZIP code, arrest record, educational institution, name) and to know that proper auditing requires testing OUTPUTS across demographic groups, not just inspecting inputs.
⚠ Common Trap
The most common trap is assuming that excluding a protected characteristic (like race) as an explicit input is sufficient to eliminate bias related to that characteristic. Proxy variables — like ZIP code, arrest record, educational institution, or name — can still carry substantial correlated demographic information, allowing disparate impact to occur even without the protected characteristic ever being directly used.
✓ Quick Self-Check
1. What is disparate impact?
When a facially neutral algorithm produces outcomes that disproportionately harm a protected group, even without explicitly using the protected characteristic.
Tap to reveal / hide
2. Why does ZIP code function as a racial proxy in many algorithms?
Because ZIP code correlates strongly with race, largely due to historical redlining practices.
Tap to reveal / hide
3. Name two other common proxy variables besides ZIP code.
Arrest record, educational institution, or name (any two).
Tap to reveal / hide
4. What does properly auditing for disparate impact require?
Testing the model's actual outputs across demographic groups, not just inspecting which input features it uses.
Tap to reveal / hide
5. Does excluding a protected characteristic as an explicit input guarantee the absence of disparate impact?
No — proxy variables can still carry substantial correlated demographic information, allowing disparate impact to occur regardless.
Tap to reveal / hide