How AI Interview Scoring Works, and How to Check a Vendor's Scores
How AI interview scoring turns a transcript into scores, what goes wrong (invented quotes, inconsistent runs, polish over substance) and how to test a vendor.
AI interview scoring rates a candidate's interview answers against a rubric written for the role. The interview is transcribed, a language model compares what the candidate said with descriptions of weak and strong answers, and each competency gets a score backed by the candidate's own words. Good scoring shows that evidence, and leaves a score blank when the evidence isn't there.
The method is simple to describe and easy to get wrong. Below: how it works step by step, the failures to watch for, how DIO does it, and a test plan you can run on any vendor before you rely on its numbers.
What is AI interview scoring?
AI interview scoring, also called automated interview evaluation, is the step after the interview: turning a conversation into something a recruiter can review quickly and compare across candidates. The output is usually a score for each competency the role needs, an overall score, and the evidence behind each one. It is one part of how an AI interviewer works, and the part that most deserves scrutiny.
It is a review aid, not a decision. A good score tells you where to look and what the candidate said about it. It doesn't tell you whom to hire, and candidates care about that difference: in Greenhouse's May 2026 vendor survey of 1,200 US job seekers, 38% wanted to know that a human reviews the AI's evaluation before any decision is made, and 39% wanted a clear explanation of what the AI is measuring (Greenhouse, May 2026).
How does AI interview scoring work?
The process usually has five steps. The differences between vendors are in how carefully each step is done.
Step 1: A rubric with anchors
The rubric lists the competencies the role needs, usually a handful, each with anchors: short descriptions of what an answer at each level sounds like. Anchors are what make two scores comparable. Without them, a 4 means whatever the grader felt like that day, human or machine.
Here is one competency from a customer support role, on a 1–5 scale:
| Score | Handling an upset customer: what the answer sounds like |
|---|---|
| 1 | No example. A general claim: "I'm very patient with people." |
| 2 | A situation, but nothing about what the candidate did. |
| 3 | A real example and what they did, but the outcome is unclear, or they mostly followed a script. |
| 4 | A specific case: what the customer was upset about, the steps the candidate took, and how it ended. |
| 5 | Everything in a 4, plus what they changed afterwards so it wouldn't happen again. |
This is the same discipline as a scorecard for a structured interview, and the research on structured interviews applies. In a 2022 re-analysis in the Journal of Applied Psychology, Sackett, Zhang, Berry and Lievens found structured interviews to be the top-ranked selection procedure. Our interview scorecard template has more anchor examples you can adapt.
Step 2: A transcript
For a voice or video interview, speech-to-text turns the recording into a transcript, and in a transcript-based system the transcript is all the scoring model sees. This is what vendors mean by AI interview transcript analysis: everything downstream is analysis of the text. That makes transcript quality the first thing to check. Speech-to-text tends to stumble on names, numbers, specialist terms, some accents and noisy rooms, and a wrong number in a transcript can change the meaning of an answer. Being able to play the recording behind any line of the transcript makes those errors easy to catch.
Step 3: A language model grades each competency
A large language model reads the transcript with the rubric and a set of instructions: find what the candidate said that bears on this competency, compare it with the anchors, choose a level, and explain the choice. Grading one competency at a time helps keep the model on the evidence for that competency, rather than on its overall impression of the candidate, the same halo effect that pulls human interviewers off course.
Step 4: Evidence
Each score should come with the candidate's own words: the sentence or two that justify the level. The quote lets a recruiter check a score in ten seconds instead of replaying the whole interview. It is only worth something if it is real, so the system should check every quote against the transcript, word for word.
Step 5: Weights, an overall score, and a person
The competency scores are weighted into an overall score, with the weights reflecting what matters most in the role. The overall score is useful for sorting a list, but it is the least informative number on the page: two candidates can reach the same overall score with opposite strengths. Read the competency scores and their evidence, then decide. A person makes the decision.
What can go wrong with AI interview scoring?
| Failure | What it looks like | How to catch it |
|---|---|---|
| Scores without evidence | A number, perhaps a summary, but nothing the candidate actually said | Ask for the quote behind any score |
| An invented or blended quote | A "quote" that reads well but isn't in the transcript, or stitches two answers together | Search the transcript for the exact words |
| Inconsistent runs | The same interview scores a 2 one day and a 4 the next | Score the same interviews twice |
| Fluency over substance | Long, confident, general answers outscore short, specific ones | Compare a polished but empty answer with a plain, specific one |
| Signals outside the answers | Accent, pace, filler words or "energy" moving a score | Ask exactly what the model receives: it should be the transcript and the rubric |
| Transcription errors | A misheard number or name changes what an answer means | Spot-check transcripts against the recordings |
| A generic rubric | Identical competencies and anchors for a nurse and a sales rep | Read the anchors for your role before launch |
| A confident guess on thin evidence | A middling score on a competency the candidate barely discussed | Check what the tool does when evidence is missing |
Three of these deserve a closer look, because they are the hardest to spot from a scorecard alone.
Why do language models invent quotes?
A language model generates text that fits what it was asked for. Asked for a supporting quote, it will usually find a real one, but it can also tidy up a paraphrase and present it as a quote, or produce a sentence the candidate never said. The fix is mechanical rather than clever: compare every quote with the transcript, and don't accept a score whose words aren't there.
Why does the same interview score differently twice?
Language models are probabilistic, so two runs over the same transcript can differ, especially when the evidence sits between two anchor levels. A one-point difference on a borderline answer is normal. Swings of two points or more usually mean the anchors are too vague or the evidence too thin to score, and the honest output in that case is no score at all.
Why does fluency get rewarded?
Models, like people, can mistake length and polish for substance. Anchors that require specifics are the defense: a number, a named situation, what the candidate did and what happened. An answer without them shouldn't reach the top levels, however well it is delivered. If communication is a competency for the role, score it against anchors about clarity and structure, not accent or speed.
How does DIO score an interview?
DIO is the AI interviewer Dio makes. Its scoring answers several of the failures above directly:
- Each competency is rated 1 to 5 against anchors written for that role, and the competency scores are weighted into an overall score.
- A competency score must quote the candidate's exact words, and the quote is checked against the transcript.
- Two different AI models grade each competency independently.
- When the evidence isn't there, or the two models disagree about whether it is, the score is left blank rather than guessed.
- In a screening round, the categories stay the same from role to role: skills and experience, motivation and interest, and communication.
The team sees the scorecard and its quoted evidence alongside the recording, the transcript and the résumé. Candidates don't see a score, and a person on the team makes every decision. A blank on the scorecard isn't a malfunction: it marks where the interview didn't produce clear evidence, which is exactly where to listen, or what to ask in the next round.
How do you check any vendor's AI interview scores?
Don't take a vendor's word for its scoring, ours included. Run these tests during a pilot, on your own roles. None of them needs a data scientist.
| Test | How to run it | What a good result looks like |
|---|---|---|
| 1. Score the same interview twice | Ask the vendor to re-score five completed interviews from scratch, then compare competency by competency | The same score, or within one point, citing the same evidence |
| 2. Trace the evidence | Pick ten scores at random and find each quote in the transcript | Every quote appears word for word, and it supports the level given |
| 3. Thin evidence | Have a colleague take an interview giving vague answers: "I'm a team player, I always give it my best" | Low scores or blanks with a stated reason, not a confident middle score |
| 4. Polish vs substance | One colleague gives long, polished, general answers; another gives plain answers with specific examples and numbers | The specific answers score higher |
| 5. Compare with your recruiters | Two recruiters score 20–30 real interviews with the same rubric, without seeing the tool's scores | The tool agrees with your recruiters about as often as they agree with each other |
| 6. Check differences between groups | Compare score distributions across the groups you can measure | Gaps are measured, explained and acted on, not assumed away |
How to compare the tool with your recruiters
Test 5 matters most, and it is cheap. Take 20 to 30 completed interviews that span the range, not just the strongest candidates. Give two recruiters the rubric and the transcripts, have each score every competency on their own, and keep the tool's scores hidden until both are done. Then count, for each competency, how often the tool lands within one point of each recruiter, and how often the two recruiters land within one point of each other.
The recruiters' agreement with each other is your baseline. If they agree with each other far more often than either agrees with the tool, the tool isn't ready for your roles. If the tool agrees with them about as often as they agree with each other, read the interviews where everyone disagrees. They often point to a vague anchor, and fixing it improves your human interviews too.
What to ask while you test
- What exactly does the scoring model receive: the transcript and the rubric, or anything else?
- Can we read and edit the anchors for our roles before launch?
- What happens to a score when the quote isn't in the transcript?
- What does the scorecard show when there isn't enough evidence?
- How often are scores re-run, and does a re-run ever change a score a recruiter has already seen?
Differences between groups are a measurement question before they are anything else: you can't fix a gap you haven't measured. Our guide to AI interview bias covers where gaps come from and how to check a tool, and how to choose AI interview software has the full list of questions to put to every vendor.