Interview Calibration: Getting Interviewers to Score the Same Answer the Same Way
What interview calibration is, why interviewers score the same answer differently, and a one-hour calibration session you can run with anchors and examples.
Interview calibration is the work of getting every interviewer on a hiring team to give the same answer the same score. It has three parts: written anchors that say what a 1, a 3 and a 5 look like, a session where interviewers score the same sample answers and compare, and a regular check that scores stay close. A calibrated team's 4 means the same thing whoever gave it, which is what makes scores from different interviewers worth comparing at all.
This guide covers why scores drift apart, a one-hour calibration session you can run this week, how to write anchors that hold up, and how to tell whether it's working.
Why do interviewers score the same answer differently?
Give five interviewers the same recorded answer and ask for a score from 1 to 5, and you'll rarely get five matching numbers. The usual reasons:
- Different pictures of "good". One interviewer thinks a strong answer names a metric; another thinks it shows how the candidate worked with people. Both are scoring something real, just not the same thing.
- Leniency and severity. Some people rarely give a 5 ("there's always room to improve"); others rarely go below a 3 ("they seemed nice"). Their averages differ before any candidate walks in.
- The middle of the scale. Unsure interviewers drift to 3, which hides the difference between candidates.
- The halo effect. One strong impression, a confident manner or a well-known employer, lifts every rating that follows.
- Comparison with the last candidate. An average answer after a weak interview feels good; the same answer after a strong one feels weak.
- The debrief. When scores are shared aloud, the first and most senior voice pulls the others toward it, and the written scores change to match.
None of this is carelessness. It's how judgment works without a shared reference, and it's why the fix is a shared reference rather than an instruction to be fair.
What does a calibrated team look like?
You can see calibration in the numbers and in the conversation:
| Sign | Uncalibrated team | Calibrated team |
|---|---|---|
| Scores for the same answer | Spread across three or four points | Mostly within one point |
| Each interviewer's average | Some consistently high or low | Close to the group's |
| Notes behind a score | "Strong", "not a fit" | What the candidate said, matched to an anchor |
| Debrief | Arguing about whether someone was good | Discussing the evidence for the scores that differ |
Perfect agreement isn't the goal. Interviewers notice different things, and a disagreement that points to evidence someone else missed is the most useful moment in a debrief. The goal is that when scores differ, it's because of the answer, not because of who was listening.
How to run a one-hour calibration session
Bring everyone who interviews for the role, the scorecard with its anchors, and three to five real answers to the same question: recordings or transcripts from past interviews, used with the candidates' consent and with names removed, or answers you write to cover a weak, an average and a strong response.
| Time | Step |
|---|---|
| 5 minutes | Read the competency and its anchors aloud. Answer questions about wording only. |
| 15 minutes | Everyone listens to or reads each answer and scores it alone, writing one line of evidence. No talking. |
| 10 minutes | Reveal the scores for each answer at the same time, on a shared sheet or by holding up fingers. |
| 20 minutes | Discuss only the answers where scores differ by two or more points. Each person names the words in the answer that earned their score. |
| 10 minutes | Rewrite any anchor that caused the disagreement, and agree on how to score the case that confused people. |
Two rules make the session work. Score before anyone speaks, or the first opinion becomes everyone's. And argue from the answer, not the candidate: "she said she rebuilt the checklist herself" is a reason; "she seemed sharp" is not.
How do you write anchors that hold up?
Most disagreements trace back to anchors that describe a feeling instead of an answer. "Excellent ownership" means something different to each reader. An anchor that holds up describes what the candidate says or did:
| Score | Weak anchor | Anchor that holds up, for ownership |
|---|---|---|
| 1 | Poor ownership | Describes a problem but gives no example of acting on it, or says someone else should have |
| 3 | Some ownership | Gives an example of fixing a problem they were asked to fix, with their own part clear |
| 5 | Excellent ownership | Gives an example of spotting a problem nobody asked them to fix, acting on it, and following it through to a result |
Write anchors for 1, 3 and 5, and let 2 and 4 sit between them. Tie each anchor to evidence the interview can actually produce. And test new anchors on two or three sample answers before using them on candidates. Our interview scorecard template has anchors for common competencies, and the glossary explains rating anchors.
How do you know calibration is working?
Measure it with a sample, a few times a year. Have each interviewer score the same five answers independently, then look at two things.
How often scores agree. For each answer, check whether every interviewer's score is within one point of the others:
| Answer | Interviewer A | Interviewer B | Interviewer C | All within one point? |
|---|---|---|---|---|
| 1 | 2 | 2 | 3 | Yes |
| 2 | 4 | 3 | 4 | Yes |
| 3 | 3 | 5 | 2 | No |
| 4 | 5 | 4 | 5 | Yes |
| 5 | 1 | 2 | 1 | Yes |
Four of five answers agree within a point. Answer 3 is the one to discuss: three interviewers heard it three different ways, which often means the answer touched something the anchors don't cover.
Whether anyone is consistently high or low. Average each interviewer's scores across the sample. In this example A averages 3.0, B 3.2 and C 3.0, so nobody is far from the group. An interviewer who averages a point above everyone else across several sessions needs a conversation and some practice, not a lecture.
Recruiters who sit in every debrief can track this as they go. When one interviewer's 4s keep turning into the panel's 3s, that interviewer needs calibrating.
Calibration in the debrief
The debrief is where calibration either holds or collapses. Three habits protect it:
- Submit scores and evidence before the debrief, so nobody adjusts a score to match the room.
- Ask the most junior interviewer to speak first, so the first opinion isn't automatically the most senior one.
- Spend the time on disagreements. Where everyone gave a 4, move on. Where scores differ, ask each person for the words in the answer behind their score.
Our evidence-based hiring playbook covers the rest of a debrief built on evidence.
Does AI scoring need calibrating too?
Yes, the same way: an AI score is only useful if it agrees with what trained interviewers would give the same answer, for reasons you can check. Before trusting any tool's scores, have two or three of your interviewers score a sample of the same interviews against the same anchors, and compare. Our guide on how AI interview scoring works lists what to look for.
DIO, Dio's AI interviewer, builds a check into every score. Each competency is rated 1 to 5 against anchors written for the role, every rating must quote the candidate's exact words, which are checked against the transcript, and two different AI models grade each competency independently. When the evidence isn't there, or the two models disagree about whether it is, the score is left blank rather than guessed, and a person on your team makes every hiring decision. Watch how it works.