Evidence-Based Hiring: A Practical Playbook for Deciding on Evidence, Not Impressions

Evidence-based hiring means deciding who to hire from information that predicts job performance, gathered the same way for every candidate and written down, instead of from impressions. The idea is borrowed from evidence-based medicine: use the methods research shows work, and keep a record that someone else can check.

Most hiring teams already believe in this. The gap is in practice: the same team that agrees structured interviews are better still lets each interviewer ask what they like, scores on a feeling, and decides in a debrief where the most confident voice wins. This playbook closes that gap in seven steps, starting with what the research says is worth collecting.

What does research say predicts job performance?

Researchers measure how well a hiring method predicts performance with a validity coefficient: the correlation between scores at hiring and later job performance. Zero means the method tells you nothing; 1 would mean it predicts perfectly. Nothing comes close to 1, because performance depends on far more than anything you can see at hiring.

For more than two decades, the standard reference was Schmidt and Hunter's 1998 summary in Psychological Bulletin, which put general cognitive ability at the center. In 2022, Paul Sackett, Charlene Zhang, Christopher Berry and Filip Lievens reanalyzed the evidence in the Journal of Applied Psychology. They found that earlier estimates had overcorrected for a statistical artifact, which inflated the validity of most methods, and that once it was fixed, structured interviews, not cognitive ability, came out on top. As HumRRO summarized the findings:

Three lessons follow for anyone who hires:

1. Structure is the single biggest lever. The structured interview, the same questions for every candidate scored against agreed criteria, came out on top. The authors suggested it, rather than cognitive ability, might be the predictor others are measured against.
2. Job-specific evidence beats general signals. Four of the top five are about the actual job: what the candidate knows, has done and can do. That favors questions and tasks built from the role over proxies such as school names or years of experience.
3. Averages hide spread. Structured interviews had the highest mean but a wide range: some structured interviews predict much better than others. How well the questions match the job and how consistently they're scored decide which end you land on.

Why do interviews drift away from evidence?

Interviews are where hiring is most exposed to impressions, because two people talking is a natural setting for them:

• First impressions set in the opening minutes, and the rest of the interview gets spent confirming them.
• The halo effect lets one strong trait, a confident manner or a famous employer, lift every other rating.
• Similarity feels like fit. A candidate who shares your background, school or sense of humor seems easier to work with, which is not the same as being better at the job.
• Inconsistency makes comparison impossible. When each interviewer asks different questions, the candidates' answers can't be lined up against each other.
• Memory fades and reshapes. Notes written hours later record the conclusion you reached, not the answers that led to it.
• The debrief rewards confidence. The first and loudest opinion in the room anchors everyone else's.

None of this means interviewers are careless. These are ordinary features of human judgment, which is why the fix is a process rather than an instruction to try harder. Our post on AI interview bias covers the same problem from the side of the tools.

The evidence-based hiring playbook

1. Define the job before you meet anyone

Write down what the person must achieve in their first 90 days, and the three or four skills or experiences that make that possible. Separate the must-haves from the nice-to-haves with one test: would you turn down an otherwise excellent candidate without it? This is the work of the hiring manager intake meeting, and everything after it depends on getting it right.

2. Choose the evidence each requirement needs

Different requirements need different evidence. Match each one to a method:

Drop anything you can't connect to a requirement. A step that measures nothing the job needs adds time and noise.

3. Ask every candidate the same questions

Write the questions in advance, tie each one to a requirement, and ask them in the same order every time. Follow-up questions can vary, because they should chase whatever the candidate actually said, but the core set doesn't. Our guide to structured interviews covers the design, and the competency-based interview questions post has 60 ready to adapt.

4. Score against written anchors

For each question or competency, write down what a 1, a 3 and a 5 answer sound like before the first interview. Anchors turn "I liked her answer" into "she gave a specific example with her own actions and a measurable result, which is a 5 on our scale." They're also what lets two interviewers' scores mean the same thing. The interview scorecard template shows how to write them.

5. Record the evidence, not the conclusion

Every score should carry its reason: what the candidate said that earned it, in their words where possible. "Rebuilt the onboarding checklist; new hires reached full shifts in two weeks instead of four" is evidence anyone can check. "Strong operator" is a conclusion only the person who wrote it can defend. When there's no evidence for a competency, leave the score blank. A blank is honest; a guessed 3 looks exactly like an earned one.

6. Score independently, then debrief

Each interviewer submits scores and evidence before talking to anyone else. The debrief then starts from the written scores and spends its time on the disagreements, because that's where the information is. Ask the most junior interviewer to speak first, so the first opinion in the room isn't automatically the most senior one.

7. Check the process against what happens next

Evidence-based hiring ends with evidence about your own process. Every quarter or two, compare first-round scores with how hires actually performed at 90 days or six months. If a competency you score heavily doesn't separate strong performers from weaker ones, change the questions or its weight. Track how many candidates each stage passes, too: a stage that passes almost everyone isn't adding evidence.

A checklist you can run on any open role

• The 90-day outcomes and must-haves are written down and agreed with the hiring manager
• Each interview stage has a different, stated purpose
• Every candidate at a stage gets the same core questions
• Each question maps to a must-have
• Anchors for 1, 3 and 5 are written before the first interview
• Every score has a written reason, in the candidate's words where possible
• Scores are submitted before the debrief
• Hard requirements are flagged for review, not used to reject automatically
• Someone checks first-round scores against later performance

What evidence-based hiring isn't

• It isn't keyword matching. A résumé that mentions the right words is evidence that the candidate knows the words.
• It isn't one test that decides. Even the best methods have modest validity on their own. Several job-related sources of evidence beat one perfect-sounding one.
• It isn't banning judgment. Interviewers still judge, but they judge answers against anchors, and they write down why.
• It isn't a score nobody can explain. A number from any tool or person counts as evidence only if you can see what it rests on.

Where does AI fit in evidence-based hiring?

AI tools can collect and score evidence at a scale people can't, which makes the last point above the important one. An AI score is evidence only if it is traceable: you can see the question, the answer, the anchor it was scored against and the part of the answer that earned it. Ask any vendor for a scorecard from a real interview and check it against the transcript. Our guide on how AI interview scoring works lists what to check.

DIO, Dio's AI interviewer, was built around this. It holds a structured voice interview with each candidate and rates each competency 1 to 5 against anchors written for the role. Every score must quote the candidate's exact words, which are checked against the transcript, and two different AI models grade each competency independently. When the evidence isn't there, or the two models disagree about whether it is, the score is left blank rather than guessed, and a person on your team makes every hiring decision. Watch how it works.