FairSight revisited: the bias in my bias detector was twice what I claimed

FairSight was our group project for FIT1055 (IT Professional Practice and Ethics): a platform that audits AI hiring systems for bias and explains every decision. The project site is fairsight.org. I built the Streamlit prototype, which is now back online at prototype.fairsight.org.

Putting it back online was a good excuse to check its numbers properly, and they weren't what the code said they'd be.

What the prototype does

It's a dashboard with five pages:

1. Overview and bias metrics: hire rates by gender, score distributions, and the effect of gender-coded keywords.
2. Explainability: which features drive the model's scores, overall and for any one applicant.
3. Mitigation: apply a fix and compare the metrics before and after.
4. Applicant view: "why was this decision made?" in plain language, plus a human review queue.
5. Audit and compliance report: a downloadable summary of all of the above.

The data: a deliberately biased model

There's no real hiring data to work with, so the prototype generates 1,200 synthetic applicants, modelled on the widely reported case of Amazon's experimental recruiting tool, scrapped in 2018 for penalising women's CVs.

Each applicant gets merit features (technical score, interview score, years of experience). Then the "model" scores them with three discriminatory penalties built in on purpose:

gender_penalty  = 0.18 * is_female
keyword_penalty = 0.04 * min(gender_coded_keywords, 5)
college_penalty = 0.10 * womens_college
final_score = base_score - gender_penalty - keyword_penalty - college_penalty

The top 30% of scores are hired. The generator's docstring says this produces a hire rate for women about 25 percentage points lower than for men.

What it actually produces

Running the project's own metrics on its own data:

Metric	Value	What it means
Male hire rate	54.1%	353 of 653 men
Female hire rate	1.3%	7 of 547 women
Demographic parity difference	0.528	A 52.8-point gap, not 25
Disparate impact ratio	0.024	Women are hired at 2.4% of men's rate
Composite bias score	0.99	On a 0–1 scale

A worked example of the two headline metrics, using these numbers:

• Demographic parity difference = male hire rate − female hire rate = 0.541 − 0.013 = 0.528. Zero would mean equal hire rates.
• Disparate impact ratio = female hire rate ÷ male hire rate = 0.013 ÷ 0.541 ≈ 0.024. The usual threshold is the four-fifths rule: below 0.8 is evidence of adverse impact. This one is more than 30 times below it.

Why the gap doubled

The penalties are large compared with the spread of scores:

• The hiring cut-off (the 70th-percentile score) is about 0.44.
• The average total penalty for a woman is 0.32: 0.18 for gender, plus roughly 0.12 for gender-coded keywords (women average about three), plus the college penalty for one in four.
• The average penalty for a man is 0.02.

A penalty of 0.32 against a cut-off of 0.44 means almost no woman can be hired, however strong her CV. The penalties stack, and the docstring described one of them, not all three together.

For a teaching demo, an extreme bias still makes the point. But a tool whose whole purpose is honest measurement shouldn't misstate its own numbers. The fix is to tune the penalties until the measured gap matches the documented one, and to print the measured value in the docs rather than an intended one.

The "mitigation" only works because we wrote the bias

The mitigation page says it applies fairness-aware sample reweighting. The code does something simpler: it recomputes every score from the merit features alone, without the three penalty terms. The result looks great:

	Before	After
Female hire rate	1.3%	29.8%
Male hire rate	54.1%	30.2%
Disparate impact ratio	0.024	0.988

But this only works because we know exactly where the bias is: we put it there. In a real system you don't get a line saying gender_penalty = 0.18. Bias comes in through proxies: a women's college, a gap in employment, a hobby, wording in a CV. Deleting the obvious column ("fairness through unawareness") leaves the proxies doing the same job. That's how Amazon's model penalised women without ever being given gender.

So the honest label for that page is "what the hire rates would be without the planted bias", which is still a useful comparison. Real reweighting would weight training examples so that each group's outcomes are balanced, without assuming you know the cause.

The ethics part

FIT1055 was an ethics unit, so the prototype came with ethical pseudocode mapping every module to an ethical framework. We chose deontology as the main framework, because it turns into hard rules code can enforce:

RULE 1: GENDER_PENALTY          = FORBIDDEN
RULE 2: COLLEGE_TYPE_PENALTY    = FORBIDDEN
RULE 3: BIAS_DISCLOSURE         = MANDATORY
RULE 4: HUMAN_REVIEW_RIGHT      = PROTECTED
RULE 5: MERIT_FEATURES_ONLY     = REQUIRED

Virtue ethics came second, for the system's character: honesty in explanations, justice in scoring, responsibility for logging and flagging. We rejected pure utilitarianism, because maximising the overall hire rate could still justify penalising one group.

Rereading it now, my own findings above are RULE 3 in action. A bias tool that understates bias, or labels a fix as something it isn't, breaks the disclosure rule it was built to enforce.

What I took away

• Measure, don't assume. The docstring described the intended bias. One run of the metrics would have shown the real one.
• Name things for what they do. "Reweighting" and "remove the terms we added" are different techniques, and the difference is exactly what matters in the real world.
• Synthetic data proves the plumbing, not the method. The dashboard, metrics and explanations work. Whether the mitigation works can only be tested on bias you didn't write yourself.