Skip to content
HireRabbit.AI
Menu
Request accessSign in
Compliance11 min read

Adverse Impact Analysis for Resume Screening: How to Check Your Own Funnel Before Someone Else Does

The four-fifths rule, where to measure in your funnel, what NYC bias audits require, and what to do when a screening stage fails, with a worked example.

By the HireRabbit.AI team · Published · Last updated

On this page
  1. Why run the check now
  2. The four-fifths rule in plain terms
  3. A worked example
  4. Where to measure
  5. What a New York City bias audit requires
  6. Handling the demographic data
  7. What to do when a stage fails
  8. Keep the records
  9. A quarterly routine
  10. How HireRabbit.AI fits

This article is general information for HR and talent teams, not legal advice. Which tests apply, which groups you must report on and how long you keep the data differ by country, state and city. Check your plan with employment counsel before you rely on it.

If software scores your applicants, someone will eventually ask whether it scores everyone the same way. The question may come from a rejected candidate, a regulator, a court or your own board. The teams that handle it well are the ones who asked it first, with their own data, and wrote down what they found and what they changed.

This guide explains the standard test for adverse impact in plain terms, shows where to apply it in a hiring funnel, walks through a worked example with real numbers, and covers what to do when a stage fails. It also summarises what New York City's bias audit rule requires, since it is the most specific written standard for AI screening tools in the US and a useful template elsewhere.

Why run the check now

Four things changed between 2023 and 2026.

A court let an AI screening case proceed as a collective action. In Mobley v. Workday, a job seeker alleged that Workday's screening tools rejected him and others on the basis of age, race and disability. On 16 May 2025, Judge Rita Lin of the US District Court for the Northern District of California granted conditional certification of a collective under the Age Discrimination in Employment Act, covering applicants aged 40 or older who applied through Workday's platform since 24 September 2020. In February 2026 the court authorised notice to potential members. Whatever the eventual outcome, the case established that a vendor's scoring tool can be challenged across many employers at once, and that the employers' own funnel data will be evidence.

Federal guidance disappeared, but the statute did not. The EEOC published technical assistance on AI and Title VII in May 2023, including a worked explanation of the four-fifths rule for algorithmic tools. In late January 2025 the documents were removed from the EEOC website, and an April 2025 executive order directed federal agencies to deprioritise disparate impact enforcement. Title VII's disparate impact provision is statute, so private claims and state enforcement are unaffected. Several states are writing their own rules. The practical effect is that the test is now something employers run for themselves rather than something the federal government walks them through.

The EU AI Act treats recruitment AI as high risk. Article 26 puts obligations on the employer deploying such a system: assign human oversight to people with the competence, training and authority to exercise it, keep the system's logs for at least six months, tell workers' representatives and affected workers before use, and tell affected people that they are subject to the system. The obligations for these systems are scheduled to apply from 2027. The AI hiring laws guide covers the timeline.

Volume made it unavoidable. With applications per job above 240 on Greenhouse's benchmarks, almost every team is scoring applications with software, a checklist or both, and the first stage of the funnel is where most people are removed. That stage is also where an impact ratio is easiest to compute, because the numbers are large.

The four-fifths rule in plain terms

The US Uniform Guidelines on Employee Selection Procedures, at 29 CFR 1607.4(D), set the standard most audits still use:

A selection rate for any race, sex, or ethnic group which is less than four-fifths (4/5) (or eighty percent) of the rate for the group with the highest rate will generally be regarded by the Federal enforcement agencies as evidence of adverse impact.

Three terms carry the rule.

  • Selection rate is the share of a group that passes a stage. If 160 women applied and 28 were moved to interview, the selection rate for women at that stage is 17.5%.
  • Impact ratio is one group's selection rate divided by the highest group's selection rate.
  • The 0.80 line is where federal agencies have said they would generally see evidence of adverse impact.

The same paragraph adds the caveats that make the rule a screen rather than a verdict. Smaller differences can still count if they are significant in both statistical and practical terms. Larger differences may not count if they rest on small numbers. The rule tells you where to look, not what you will find.

The same section, 1607.4(A), tells employers to keep records showing the impact of their selection procedures by race, sex and ethnic group, and allows sampling when applicant volumes are high and procedures run often, as long as the sample is representative and large enough.

A worked example

A 300-person company fills a sales development role. 400 people apply. The team uses AI scoring with a cut-off, and resumes above the cut-off go to a recruiter screen. The table shows the first stage. Group sizes are illustrative, and the categories are the ones an EEO-1 report uses.

GroupAppliedAbove cut-offSelection rateImpact ratio
Men2406025.0%1.00
Women1602817.5%0.70

The impact ratio for women is 0.70, below 0.80. That is a flag, not a finding. The next questions are whether the difference is statistically significant at these sizes (with groups of 240 and 160 it is close to the line, and worth a proper test), and what in the scoring produced it.

Now the interview stage, counting only the people who passed the screen.

GroupScreened inInterviewedSelection rateImpact ratio
Men602033.3%0.62
Women281553.6%1.00

Here women are selected at a higher rate, and the ratio for men is 0.62, which is its own flag in the other direction. Looking at the whole funnel from application to interview would blur the two stages together: 20 of 240 men (8.3%) against 15 of 160 women (9.4%), a ratio of 0.89 for men and no flag at all. The overall number hides a problem at each stage. This is the single most useful habit in adverse impact work: compute the ratio for each stage separately, and treat the overall figure as a summary rather than a result.

Where to measure

Apply the test wherever a decision removes people, which in most funnels means at least four points.

  1. Application to screen. Who passed the first filter, whether that filter is a knockout question, an AI score cut-off or a recruiter skim.
  2. Screen to interview. Who got a conversation.
  3. Interview to offer.
  4. Offer to hire, which usually says more about the offer than about bias, but is cheap to include.

Run each stage by sex, by race and ethnicity, and by the combinations. New York City's rule for automated tools requires the intersectional categories as well, so that a problem affecting, for example, women in one ethnic group is not hidden by a healthy ratio for women overall.

Two practical rules about small groups. Where a category is a very small share of the applicants, the ratio will swing on one or two people and means little; New York City's rule lets auditors exclude categories below 2% of the data as long as they say so. Where the groups are larger but still modest, a few dozen people each, treat a ratio below 0.80 as a reason to look harder and to pool more data, not as a conclusion. Pooling across quarters, or across similar roles scored against the same sheet, often gets you to numbers that mean something.

What a New York City bias audit requires

Local Law 144 has been enforced by the city's Department of Consumer and Worker Protection since 5 July 2023. It applies when an automated employment decision tool is used to substantially help with hiring or promotion decisions for people in the city. The final rule, adopted in April 2023, requires:

  • An audit within the year before the tool is used, and annually after that.
  • An independent auditor, meaning someone not involved in using, developing or distributing the tool and with no employment or financial relationship with the employer that would compromise independence.
  • The selection rate for each sex and race or ethnicity category used on an EEO-1 report, and the impact ratio of each against the most selected category. For tools that score rather than pass or fail, the rule defines a scoring rate based on whether a person's score sits above the sample median. Intersectional categories are required.
  • The number of people the tool assessed who were left out because their sex or race and ethnicity was unknown.
  • A published summary of the results, and notice to candidates that the tool is in use.

The law does not set 0.80 as a pass mark; it requires the ratios to be computed and published. The published audits show what this looks like in practice. Pfizer's 2023 bias audit of HireVue, for example, reports impact ratios by sex, by race and ethnicity and by their intersections for the tool as used on Pfizer's own applicants, with the auditor and date on the cover. Reading two or three of these before you commission your own is the fastest way to see the format.

Handling the demographic data

An impact analysis needs to know applicants' sex and race or ethnicity, and the scoring process must not. Keep the two apart.

  • Collect demographic data on a voluntary form that is separate from the application, and say what it is for.
  • Store it where recruiters, hiring managers and the scoring tool cannot see it. The person who runs the analysis sees it; nobody who makes decisions does.
  • Match it to stage outcomes by a candidate identifier at analysis time, not before.
  • Report the share of applicants who did not answer. If a large share declined, the analysis is weaker, and the New York City rule requires that number in any case.

A scoring tool that never receives names, photos or demographic fields cannot score on them directly. It can still score on proxies, which is the reason the analysis exists. University of Washington researchers found in 2024 that large language models asked to rank otherwise identical resumes preferred white-associated names 85% of the time, which is a reminder that a tool reading free text has more information about a candidate than the fields you gave it.

What to do when a stage fails

A ratio below 0.80 at the screening stage means the definition of "qualified" that the stage uses selects groups at different rates. Turning the tool off and going back to recruiters skim-reading at 11pm does not remove that definition; it hides it. The useful work is in the inputs.

Look at the must-have list first. Harvard Business School and Accenture's 2021 survey of 2,275 executives found 88% believed qualified candidates were being screened out by criteria in the job description such as a four-year degree. A degree requirement, a years-of-experience band or one named tool can produce an impact ratio on its own. The job description guide covers how to decide which requirements are genuine must-haves.

Compare the scored stage with the human stage. If the AI score cut-off shows a ratio of 0.70 and the recruiter screen just after it shows 0.95, the scoring sheet is doing something the recruiters are not. Read the area scores for a sample of rejected resumes from the affected group and find which area moves. If it is education, revisit the degree weight. If it is experience, check whether years or employer names are carrying the score.

Check the cut-off itself. Moving a cut-off from 70 to 65 can change the ratio materially without changing the ranking. The cut-off is a policy choice, and it should be recorded as one.

Write down the change and re-run. The record of what you found, what you changed and the ratio afterwards is the evidence that you treated the tool as something you supervise. Under the EU AI Act's deployer obligations that record is close to mandatory; under the Uniform Guidelines it is what 1607.4(A) has asked for since 1978.

What not to do is adjust scores by group to bring the ratio up. That substitutes one legal problem for another in most jurisdictions, and it leaves the underlying scoring sheet unchanged.

Keep the records

The records to keep are the stage data, the scoring sheet in force at the time, the cut-off, the demographic analysis, and the decisions made in response. On retention: the EEOC requires personnel and employment records, including application records, to be kept for one year, and until final disposition if a charge is filed. Federal contractors have longer periods. The EU AI Act requires deployers to keep a high-risk system's logs for at least six months. The record-keeping guide covers the schedules in more detail.

A quarterly routine

Most teams can run this in an afternoon once the exports exist.

  1. Pull stage outcomes for every role closed in the quarter, with candidate identifiers.
  2. Join the voluntary demographic data.
  3. For each stage, compute selection rates and impact ratios by sex, by race and ethnicity, and by their intersections. Flag categories below 2% and ratios below 0.80.
  4. For each flag, run a significance test and note the group sizes.
  5. For each flag that is significant or persistent across quarters, write one paragraph: the stage, the likely input, the change made, the owner.
  6. File the analysis with the scoring sheets and cut-offs that were in force.
  7. If you are in scope for New York City's rule, send the stage data to your independent auditor on the annual cycle.

How HireRabbit.AI fits

HireRabbit.AI does not run your bias audit; an audit has to be independent. What it does is make the inputs to one visible. Every overall score is built from four area scores, for skills, experience, projects and education, so when a stage fails the check you can see which area moved. Must-have skills act as a floor rather than a hidden weight. Auto-reject only applies below a cut-off your team sets, in a batch a person confirms, and it is reversible, so the cut-off is a recorded choice rather than a default. Every move of an individual candidate carries a written reason. Those records are what an auditor, a court or your own quarterly review will ask for. See the security page for how access to resumes and scores is scoped.

Sources

Keep reading

Put this into practice.

Free during invite-only early access. All you need is a work email, and a person reviews every request.

Request access See the product