AI Interview Bias: Where It Comes From and How to Check a Tool
Where bias gets into AI interviews, from training data and accents to fluency scoring, documented examples, and how to test a tool before you trust its scores.
AI interview bias is when an AI interview scores or treats candidates differently for reasons that have nothing to do with the job: their accent, their speaking style, words associated with their gender, or how well a speech-recognition system hears them. It can enter through training data, proxies, transcription errors, inconsistent questions, scoring that rewards fluency over substance, and drift over time, and the only way to know whether a tool has it is to measure.
Human interviewers aren't the benchmark of fairness either. In Greenhouse's May 2026 survey of 1,200 US job seekers (a vendor survey), candidates reported nearly identical rates of perceived bias from AI and human interviewers: 36% felt age bias from both, and 27% flagged race or ethnicity bias from both. The useful question isn't whether AI or people are biased. It is which process lets you find the bias and fix it.
What is AI interview bias?
In measurement terms, a biased interview is one where the same evidence earns a different score depending on who gives it. Two candidates describe the same project with the same results; one has a regional accent, or speaks English as a second language, or pauses before answering, and gets a lower score. The difference comes from the tool, not the candidate.
Two things make AI bias different from a biased person. It is consistent: a model can repeat the same mistake across thousands of interviews, at whatever volume you run. And it is testable: you can feed a tool the same answer twice in different voices and compare the results, which you can't do with a hiring manager's instincts. The first is the risk. The second is the reason to measure instead of trusting a vendor's description.
What are some examples of AI hiring bias?
Five public cases show most of the ways it happens. Each is covered in more detail below.
| Published | What happened | Reported by |
|---|---|---|
| 2018 | Amazon scrapped an experimental résumé-rating tool that had learned to penalize résumés including the word "women's" | Reuters |
| 2020 | Five commercial speech-recognition systems averaged a word error rate of 0.35 for Black speakers and 0.19 for white speakers | PNAS |
| 2021 | A phone-interview tool rated answers read in German 6 out of 9 for English; another tool gave a gibberish transcript a 73% match | MIT Technology Review |
| 2021 | A video-assessment tool rated candidates differently when they changed their background or wore glasses | BR, via MIT Technology Review |
| 2024 | Three language models ranking résumés favored white-associated names 85% of the time and Black-associated names 9% | University of Washington |
Where can bias enter an AI interview?
Training data that learned old hiring decisions
A model trained on past hiring decisions learns the patterns in those decisions, including the unfair ones. The best-known case: in 2018, Reuters reported that Amazon had scrapped an experimental tool that rated candidates from one to five stars. It had been trained on résumés submitted to the company over a 10-year period, most of them from men, and it taught itself to penalize résumés that included the word "women's" and to downgrade graduates of two all-women's colleges. Amazon edited the programs to make them neutral to those terms, but, according to Reuters' sources, that gave no assurance the system wouldn't find other ways to sort candidates unfairly. The team was disbanded.
The lesson for interviews: ask what a model learned from. A tool trained to predict "who got hired here before" will reproduce who got hired here before.
Proxies: names, accents, speaking style, backgrounds
A model doesn't need to know a candidate's gender or race to treat groups differently. It only needs a signal that travels with them. In a 2024 University of Washington study, researchers ran more than three million comparisons of résumés against job listings through three large language models, changing only the names. White-associated names were favored 85% of the time and Black-associated names 9%; male-associated names 52% and female-associated names 11%. The models never favored Black male-associated names over white male-associated names.
In a spoken interview, the possible proxies include accent, dialect, pace and word choice. On video, they include what is behind the candidate. A 2021 investigation by the German public broadcaster BR, described by MIT Technology Review, found that one video-assessment tool rated candidates differently when they changed their video background or wore accessories such as glasses.
Speech-recognition errors on some accents and dialects
Every spoken AI interview starts with a transcript, and transcripts aren't equally accurate for everyone. A 2020 study in PNAS, Racial disparities in automated speech recognition, tested systems from Amazon, Apple, Google, IBM and Microsoft on 19.8 hours of interviews with 42 white and 73 Black speakers. The average word error rate was 0.35 for Black speakers and 0.19 for white speakers. Using an error rate above 0.5 as the threshold for an unusable transcript, 23% of audio snippets from Black speakers were unusable, against 1.6% from white speakers.
Speech recognition has improved since that study, but the principle holds: any score built on a transcript inherits the transcript's errors, and those errors don't fall evenly. A candidate whose best sentence was transcribed as nonsense has nothing for a scorer to credit.
Questions that change from candidate to candidate
If an AI improvises its questions, two candidates for the same job can face different questions of different difficulty, and their scores stop being comparable. Consistency is the foundation of fair interviewing. A 2022 re-analysis of selection research in the Journal of Applied Psychology (Sackett, Zhang, Berry and Lievens) ranked structured interviews as the top selection procedure for predicting job performance. An AI interview should keep that structure: the same core questions for every candidate for a role, with follow-ups to get the evidence.
Scoring that rewards fluency over substance
Some tools score how something is said rather than what is said. In the same MIT Technology Review piece, a reporter answered a phone-interview tool's questions by reading a German Wikipedia entry and was rated 6 out of 9 for English competency. A second tool turned the German into an English transcript that was gibberish and still gave her a 73% match for the job. A psychologist working with that company told the reporters the algorithm had drawn personality traits from her voice rather than from the content of her answers.
A score that moves without the content moving is measuring something else, and that something else, such as intonation, pace, confidence or polish, varies with first language, accent, disability and nerves. A candidate who stutters through a strong answer should not lose to a smooth speaker with a thin one.
Drift over time
Vendors update models. Prompts and rubrics get edited. Job descriptions change, and so does who applies. A bias check you ran in January describes January. Ask your vendor to tell you when the scoring model changes, and treat every change as a reason to re-run your checks.
| Where bias enters | What it looks like | The check that finds it |
|---|---|---|
| Training data | Scores that track who was hired before | Ask what the model learned from; compare pass rates |
| Proxies | Same content, different score by accent or background | Same answer in different voices |
| Speech recognition | Low scores where the transcript is wrong | Listen to recordings behind low scores |
| Inconsistent questions | Candidates asked different things | Compare the questions asked across interviews |
| Fluency scoring | Scores that move without the content moving | Ask for the evidence behind each score |
| Drift | Results change after an update | Re-run the checks after every change |
How to check an AI interview tool for bias
Treat this as a measurement exercise, the way you would check any instrument before trusting its readings. None of these steps needs special software.
1. Compare pass rates across groups, where you have the data
If your organization already collects voluntary demographic information, keep it apart from the people making decisions and analyze it in aggregate. For each group, divide the number of candidates your team moved forward after the AI interview by the number interviewed. Then divide each group's rate by the highest group's rate.
A worked example: 120 candidates in one group, 48 moved forward, a pass rate of 40%. In another group, 90 candidates, 27 moved forward, a pass rate of 30%. The ratio is 0.30 ÷ 0.40 = 0.75. A gap like that doesn't prove the tool is biased, because the decision was your team's and other factors play in, but it tells you where to run the next three checks.
Mind the sample size. With 20 candidates in a group, one person moves the rate by five percentage points. Pool several months before you read much into a gap.
2. Compare score distributions, not just averages
Look at each competency separately. If one competency, communication for example, splits by group while the others don't, look hard at whether it is rewarding accent or polish. Check how often the tool leaves a score blank for each group too. A tool that abstains more often for one group may be struggling to transcribe them.
3. Read the transcripts behind low scores
Pull the 20 lowest-scoring interviews from a recent month. For each one, listen to a few minutes of the recording alongside the transcript. Did the transcript get the words right? Does the low score reflect what the candidate said, or what the transcript lost?
4. Test the same answer in different voices
Write a script of strong answers to your interview's questions. Ask four to six colleagues with different accents, ages and speaking speeds to read it in test interviews. Run one voice twice first, to see how much scores move with no change at all. Then compare the rest. The content is identical, so any difference beyond that baseline comes from the voice.
5. Ask for the evidence behind every score
A score you can't trace to what the candidate said can't be checked. For a sample of scores, ask: what did the candidate say that earned this? A quote you can find in the recording is checkable. A personality trait or a "communication index" with nothing behind it is not. How AI interview scoring works covers what good evidence looks like.
6. Repeat after every change
Re-run the voice test and the transcript review after any model update, rubric change or new role, and look at pass rates and score distributions on a fixed schedule.
| Test | How to run it | What a problem looks like |
|---|---|---|
| Pass rates | Advanced ÷ interviewed per group, then ratio to the highest group | A persistent gap over several months |
| Score distributions | Each competency by group, plus blank-score rates | One competency splitting by group; more blanks for one group |
| Low-score review | Recording against transcript for 20 low scores | Transcription errors behind low scores |
| Voice test | Same script, four to six voices, one voice twice | Score differences beyond the repeat baseline |
| Evidence check | Ask what was said to earn a sample of scores | Scores with no quote behind them |
How to avoid AI bias: design choices that reduce the risk
No design makes a tool free of bias. Some choices make bias less likely and much easier to catch.
- The same structured questions for every candidate for a role. Comparable questions are the precondition for comparable scores.
- Anchored rubrics. Write down what a weak, an adequate and a strong answer looks like for each competency, so the scorer compares answers to a description, not to other candidates. The interview scorecard template has examples.
- Scores that must quote evidence. If every score has to point to the candidate's words, a reviewer can check it, and a score built on tone or polish has nowhere to hide.
- Abstaining when unsure. A blank score that says "not enough evidence" is more honest than a guess, and a guess is where bias slips in.
- No facial or emotion analysis. Expressions and backgrounds are exactly the kind of proxy described above. In 2021, HireVue said it had removed the facial analysis component from its screening assessments after its own research found that visual analysis no longer significantly added value, SHRM reported.
- A person decides. The tool produces evidence; a person reads it and makes the call, and can overrule a score that doesn't match the recording.
- Another route on request. Some candidates can't do a voice interview well for reasons unrelated to the job. Say whom to contact for a different format.
How DIO handles these risks
Here is how DIO, the AI interviewer Dio makes, lines up against that list, including what it doesn't cover.
Each role gets a structured interview plan that a recruiter reviews and edits before publishing. The screening round follows a fixed arc of the questions recruiters ask on a first call and rates every candidate on the same categories: skills and experience, motivation and interest, and communication. Each competency is rated 1 to 5 against anchors written for that role.
A competency score must quote the candidate's exact words, and the quote is checked against the transcript. Two different AI models grade each competency independently. When the evidence isn't there, or the two models disagree about whether it is, the score is left blank rather than guessed. A camera is never required. Knockout questions flag an answer for the recruiter's review and never end the interview, and a person on the team records every hiring decision.
The limit: a quote check confirms that the words are in the transcript. It can't confirm that the transcript heard the candidate correctly. The team gets the recording as well as the transcript, so a reviewer can check, and the voice test above is worth running on any tool, DIO included. Interviews are in English only today, so for roles where English fluency isn't part of the job, offer candidates another route.
Questions to ask a vendor about bias
- What does the score use: only what the candidate said, or also tone, pace, facial expression or background?
- Can I see the evidence behind every score, in the candidate's words?
- Which speech-recognition system transcribes interviews, and how has it been tested across accents?
- What happens when the model isn't sure? Does it guess, or leave the score blank?
- Are the questions the same for every candidate for a role?
- How will you tell us when the scoring model changes?
- Can we run our own test interviews with scripted answers?
- Who makes the decision to move a candidate forward or not?
A vendor that welcomes these questions is one you can measure. Treat a vendor that promises its tool is fair, without showing you how to check, with care. For the rest of the evaluation, see how to choose AI interview software, and for what an AI does better or worse than a person in the interview itself, AI interview vs human interview.