Four layers of bias in hiring algorithms: why dropping gender fails

According to Reuters, a team at Amazon began building software to score job applicants' CVs in 2014; by 2015 it saw that the tool rated candidates unevenly by gender, and the company later dropped it (Dastin, 2018). If someone offers you CV-screening software, what should you ask? This post reads the Amazon recruiting-tool case through four layers of algorithmic bias and algorithmic fairness metrics, then turns each layer into a question for the vendor.

What did Reuters report about Amazon's tool, and what did it leave out?

Reuters, citing five people familiar with the effort, reports that the model was trained on CVs submitted over 10 years, most of them from men, and so taught itself that male candidates were preferable. It penalised CVs containing the word "women's" and downgraded graduates of two all-women's colleges (Dastin, 2018).

The team built 500 models and taught each to recognise about 50,000 terms from past CVs; verbs such as "executed" and "captured", more common on male engineers' CVs, were favoured. Amazon edited the models to be neutral to those terms, but the sources said that was no guarantee the machine would not find other ways of sorting candidates that discriminated. The article says the team was disbanded by the start of the previous year, which for an October 2018 piece means early 2017 (Dastin, 2018).

The article gives no accuracy figures and relies on anonymous sources. According to them, problems with the data also made the tool recommend unqualified candidates almost at random, and Amazon shut the project down. Amazon said recruiters never used the tool to evaluate candidates, but it did not dispute that they looked at its recommendations.

What are the four layers, and which did Amazon hit?

The four layers are skewed training data, proxy variables, structural injustice with feedback loops, and conflicting notions of fairness. On the Reuters account, Amazon hit the first two most clearly.

My reading: layer 1 is ten years of mostly male CVs, and layer 2 is the word "women's", the college names and the verbs. Reuters calls the male-heavy data a reflection of male dominance across the tech industry, which is layer 3, but it says nothing about a feedback loop.

Is dropping the gender column enough?

No, because other columns can rebuild it. Dwork and colleagues call ignoring protected attributes "fairness through blindness" and give redundant encoding as a successful attack on it: deleting the "sex" and "interested in men/women" fields from a profile almost surely does not hide sexual orientation, because group membership can be embedded in months of online activity (Dwork et al., 2012). Dattner and colleagues write that Facebook "likes" can be used to infer sexual orientation and race with considerable accuracy (Dattner et al., 2019).

A hypothetical example; every number is mine. A company receives 1,000 CVs, 500 from women and 500 from men. Of these, 200 women's CVs and 50 men's show a career gap of six months or more. The tool has no gender column, but it learned from earlier recruiters who rarely shortlisted CVs with gaps, so it rejects them.

Delete the "gap" column and the tool finds the next proxy. Reuters describes exactly this worry when it says Amazon edited out particular terms (Dastin, 2018).

Does more representative data fix structural injustice?

Only in part: it corrects a group being too thin or too thick in the data, not the kind of data collected. Dattner and colleagues note that developers balance training data so that all groups are represented, and that the more information the tool can learn from, the better it can control bias (Dattner et al., 2019). Zimmermann and colleagues say of countermeasures like this that they can address over- and under-representation but not the problem of what kind of data was collected in the first place (Zimmermann et al., 2020).

They use predictive policing: the data reflect which areas were patrolled most, the algorithm repeats the past, its predictions become new data, and injustice is entrenched further (Zimmermann et al., 2020). That is a feedback loop. The hiring version, by my reasoning: the tool shortlists few women, the firm hires few women, and the next training run treats that as evidence.

What does "fair" mean when two groups have different base rates?

Several things that cannot all hold. Binns lists demographic parity, equal opportunity, disparate mistreatment, counterfactual fairness and group-wise accuracy measures, and writes that some of them cannot be satisfied together except in rare and contrived circumstances (Binns, 2018).

Theorem 1 of Kleinberg and colleagues has precise conditions. There are two groups; each person has a feature vector; the tool places people in bins and gives each bin a score read as the probability of belonging to the positive class (here, people who are truly suited to the job). Three conditions (Kleinberg et al., 2017, p. 43:5):

• (A) Calibration within groups: of the people given score x, about a fraction x belong to the positive class, group by group.
• (B) Balance for the negative class: the average score of negative-class people is the same in both groups.
• (C) Balance for the positive class: the average score of positive-class people is the same in both groups.

If any scoring satisfies all three, then either prediction is perfect (every probability is 0 or 1) or the two groups have equal base rates (Kleinberg et al., 2017, p. 43:6). An approximate version holds too (Kleinberg et al., 2017, p. 43:6). The authors treat the balance conditions as generalisations of equal false-negative and false-positive rates between groups (Kleinberg et al., 2017, p. 43:3).

A hypothetical example with my own numbers. Each group has 1,000 applicants; the high band is 80% truly suited, the low band 10%, in both groups, so calibration holds. Base rates are 45% in group 1 and 24% in group 2.

The last two rows are the paper's two balance quantities. To make group 2 catch 89% of its suited people too, I would have to shortlist about 533 more from the low band: the list grows to 733, 68% of unsuited people get through, and only about 29% of the list is suited.

My reading of the library's table: it gives three metrics the same "impossibility" line, whereas the theorem concerns only the three conditions above. Binns adds that a metric that fits one context need not fit another (Binns, 2018).

What should you ask a vendor at each layer?

Ask for numbers, not reassurance. The table is my own tool, not a standard from any source.

Dattner and colleagues (article published April 2019) say many new candidate-assessment tools emerged from technology rather than validated methods, so may not control for discriminatory impact, and that in the US an employer may be liable for blind reliance on them (Dattner et al., 2019). Reuters quotes critics saying candidates may never know such a tool was used, which makes suing difficult (Dastin, 2018). On who answers when an automated system gets it wrong, see When an airline's chatbot gave wrong advice, the airline paid, a different case with the same question.

When does this approach not fit, and where does the post stop?

It fits if you receive enough CVs to test the tool on your own data, someone reads a sample of rejections, and the vendor can answer all four rows. It does not fit if the vendor will not say what the tool reads. Amazon is one company, reported through anonymous sources in 2018; I do not know what the tool does today.

Going deeper: what did I read, and what did I not?

I read the Reuters article (Dastin, published 10 October 2018) in full, through the Internet Archive capture. I read Binns (2018) and Zimmermann and colleagues (2020) in full. For Kleinberg and colleagues (2017) I read the opening, the statements of Theorems 1 and 2, the start of the proof and the conclusion. For Dwork and colleagues (2012) I read the arXiv version 1104.3913, not the ACM print: the introduction, section 3.1 and the appendix.

Dattner and colleagues (2019) sits behind the HBR paywall; I read only the article text embedded in the public page source, about 2,050 words. I have not read studies of how well any commercial CV-screening tool works, or Vietnamese employment or data-protection law.