How to Choose AI Interview Software: 15 Questions to Ask Every Vendor
15 questions to ask every AI interview software vendor, what a good answer sounds like, a two-week pilot plan, and the red flags that should end a demo.
To choose AI interview software, put the same 15 questions to every vendor, covering candidate experience, the evidence behind scores, human control, fairness testing, data, workflow, cost and the pilot, then test the answers on your own roles for two weeks before you sign. A demo shows what a product does on the vendor's best day; a pilot shows what it does with your candidates.
The questions are below, grouped, each with why it matters and what a good answer sounds like. After them come a two-week pilot plan and the red flags that should end a conversation. The checklist download at the top of this page has all 15 questions, with a column for each vendor's answers.
What does AI interview software do?
AI interview software holds the interview itself. It asks candidates structured questions by voice, video or chat, follows up on their answers, and gives the hiring team a transcript and scores to review. Most AI interview tools for recruiters are built for the first-round screen, though some also run later rounds. If you are new to the category, start with what an AI interviewer is, which covers the formats and how they work.
More HR teams are using AI. SHRM's 2025 Talent Trends survey of 2,040 HR professionals found that AI adoption in HR tasks climbed to 43% in 2025, from 26% in 2024 (SHRM, 2025). As adoption grows, so does the number of similar-sounding demos, and the reason to ask every vendor the same questions.
Candidate experience
1. What does the candidate see and need, from the invitation to the last question?
Why it matters. Every step is a chance to lose someone. In Greenhouse's May 2026 vendor survey of 1,200 US job seekers, 38% had walked away from a hiring process because it included an AI interview (Greenhouse, May 2026). An app to install, a camera requirement or an interview with no stated length all add friction.
A good answer sounds like: a live walk-through on a phone and on a laptop. How the link arrives, what the candidate is told, what they need and don't need, how long it takes, and what they see at the end. No app to install, and no camera unless the role genuinely calls for one.
2. How is the candidate told they are talking to an AI, and what role does a person play?
Why it matters. In the same Greenhouse survey, 70% of job seekers were never clearly told upfront that AI would be evaluating them, 21% found out only once the interview had started, and 46% wanted the option to request a human interview instead. Disclosure is the cheapest trust you will ever buy.
A good answer sounds like: "Before joining, the candidate sees that an AI runs the interview, that it is recorded, and that a person makes the decision, and they agree to it. Here is the exact wording." Ask to see the screen, not a description of it.
3. What can the candidate do during the interview?
Why it matters. Candidates are nervous, connections drop, and some questions get misheard. A candidate who can't ask for a repeat is being tested on their audio, not their experience.
A good answer sounds like: the candidate can ask for a question to be repeated, skip one, turn on captions and end the interview at any time, and the vendor can say plainly what happens if the connection drops. If candidates can ask their own questions, the tool answers only from information you approved, and says when it doesn't know.
Evidence and scoring
4. What exactly is scored, and what is never scored?
Why it matters. Anything that feeds a score other than the candidate's answers, such as voice, accent, facial expressions or background, is hard to tie to the job and hard to check.
A good answer sounds like: "We score what the candidate said, from the transcript, against a rubric written for your role. Nothing else feeds the score. Here are the rubric and its anchors." Then they show you.
5. Can you show me the evidence behind every score?
Why it matters. A score you can't check is an opinion. Language models can also paraphrase an answer and present it as a quote, so the evidence needs checking too.
A good answer sounds like: "Every competency score quotes the candidate's exact words, and each quote is checked against the transcript." In the demo, click any score, read the quote, and find it in the transcript yourself.
6. What happens when the evidence is thin, and how consistent are the scores?
Why it matters. The dangerous output is a confident guess: a middling score on a competency the candidate barely discussed. And because language models are probabilistic, the same interview can score differently on two runs.
A good answer sounds like: "When there isn't enough evidence, the score is left blank and the scorecard says so. Score the same interview twice and you get the same result, or within a point. Here is how we measure that." Our guide to how AI interview scoring works has a test plan for checking this yourself.
Human control
7. Does the software ever reject or advance a candidate on its own?
Why it matters. People don't want software making the call: in a Pew Research Center survey of 11,004 US adults in December 2022, Americans opposed AI making final hiring decisions by 71% to 7% (Pew, April 2023). Automatic rejection on a knockout answer also misfires: a misread question or a license under a different name is enough.
A good answer sounds like: "No. Knockout answers are flagged for a recruiter, and every decision is made and recorded by a person." If automatic rejection exists as a setting, ask whether it is on by default and who can switch it on.
8. Can we review and edit the questions, follow-ups and rubric before anything goes live?
Why it matters. It is your interview. A plan generated from a job description can misread what the role needs, and you are the one who knows.
A good answer sounds like: "Nothing reaches candidates until a recruiter has reviewed and published it. You can change the questions, what counts as a hard requirement, and what the rubric looks for."
Fairness testing
9. How have you tested whether scores differ between groups of candidates, and can we see the method and the results?
Why it matters. Bias in an AI interview, where it exists, shows up as a measurable gap: in scores, in transcription accuracy across accents, or in who reaches the next round. Candidates are already skeptical; Gartner's survey of 2,918 job candidates in the first quarter of 2025 found that just 26% trust AI to evaluate them fairly (Gartner, July 2025).
A good answer sounds like: "Here is what we measured, on what data, how large the differences were, what we changed as a result, and what we haven't tested yet." A vendor who says there is nothing to measure hasn't measured. Our guide to AI interview bias explains where gaps come from.
10. Can we run the same check on our own candidates?
Why it matters. A vendor's test data isn't your applicant pool. Your roles, your locations and your candidates are what count.
A good answer sounds like: "Yes. You can export every candidate's scores and outcomes, so your team can run the same comparison on your own data, during the pilot and afterwards."
Data and deletion
11. What do you keep, for how long, who can see it, and where is it stored and processed?
Why it matters. Recordings, transcripts and résumés are personal data, and your security team will want all four answers before you sign.
A good answer sounds like: specifics, in writing. Exactly what is kept, for how long, which people can see it, which country or region it is stored and processed in, and which other companies process it on the vendor's behalf.
12. When a candidate asks to have their interview deleted, what goes, and who can do it?
Why it matters. Candidates do ask, and a deletion that leaves the recording behind isn't one.
A good answer sounds like: "An admin can delete an interview at any time, and the recording, the transcript, the scorecard and the candidate's details go with it." Ask how long it takes for copies in backups to clear, and test a deletion during the pilot.
Workflow and exports
13. How do results get into our ATS and our process today, not on the roadmap?
Why it matters. Recruiters live in the applicant tracking system (ATS). A tool that makes them retype results, or depends on an integration that doesn't exist yet, won't get used.
A good answer sounds like: a clear split between what works today and what is planned. Today might be a live integration, or exports to CSV and Excel and a PDF summary for each candidate. Planned is fine as long as it is labeled as planned. Ask too how candidates get their links, and whether your team can sign in with the accounts it already uses.
Cost model
14. What exactly do we pay for, and what counts as a billable interview?
Why it matters. Pricing units differ between vendors: an interview started, an interview completed, a recruiter seat, a monthly platform fee, or a mix with minimum commitments. Drop-offs and repeat attempts can quietly inflate the bill.
A good answer sounds like: "This is the unit. This is what counts as one: a completed interview, not a started one. Here is the cost at your volume, and here is what happens if you use less than you planned." Ask for a worked example using last quarter's application numbers.
The pilot
15. Will you run a pilot on our roles, with our recruiters scoring alongside, before we sign a long contract?
Why it matters. Everything above is a claim until you see it on your own candidates. A pilot turns a sales conversation into evidence.
A good answer sounds like: "Yes. Pick one or two roles, and we'll set them up with you. You'll get the transcripts, scores and exports you need to compare with your recruiters' ratings, without a long commitment first."
How do you run a two-week pilot?
Pick one or two roles with steady applications, and write down before you start what would make you buy: completion rate, what candidates say, how often the tool agrees with your recruiters, and how long a recruiter takes to review an interview.
| Days | What to do | What you learn |
|---|---|---|
| 1–2 | Set up the role. Review and edit the questions, follow-ups, hard requirements and rubric | How much of the plan you had to fix, and how easy that was |
| 3–4 | Five colleagues take the interview: one strong, one vague, one polished but empty, one who asks lots of questions, one on a phone in a noisy room | Whether vague and polished answers are scored as such, and how the tool handles questions it can't answer |
| 5–10 | Send links to real candidates for the pilot role, with a person reviewing every interview and a way to reach a recruiter for anyone who prefers to | Completion, drop-off, candidate comments, review time |
| 8–12 | Two recruiters score 20 to 30 completed interviews with the same rubric, without seeing the tool's scores | How often the tool agrees with your recruiters, compared with how often they agree with each other |
| 11–13 | Delete a test interview end to end. Export results and move them into your ATS. Show a hiring manager the summary | Whether the data, deletion and workflow answers were true |
| 14 | Score the vendor against the measures you wrote down on day one | A decision based on your candidates, not a demo |
Run the same pilot with each vendor on your shortlist and you can compare them on evidence. There is no general answer to which AI interview tool is best: the best one for you is the one that wins this pilot on your roles. Our comparison of the best AI interview software is a place to build the shortlist.
What are the red flags?
- Scores without quotes, or quotes you can't find in the transcript.
- Claims that the tool is free of bias, with no measurements to show.
- Automatic rejection switched on by default, or no way to turn it off.
- Scoring voice, face, emotion or "personality" from how someone sounds or looks.
- Accuracy figures with no method: a percentage, but no answer to "measured how, and against what?"
- Roadmap features demoed as if they were live, especially integrations.
- No pilot, or a pilot on the vendor's sample data instead of your candidates.
- A vague answer on deletion, or a deletion that leaves the recording behind.
- A candidate flow you wouldn't want to go through yourself. Take the interview before your candidates do. Our guide to AI interview candidate experience covers what candidates notice.
How does DIO answer these questions?
We make DIO, the AI interviewer from Dio, so hold it to the same list. Briefly, from what the product does today:
- Candidates open a link the recruiter shares, add their name, email and résumé, agree to a recording and AI notice, check their microphone, and talk. A camera is never required, captions are available, and they can ask DIO to repeat or skip a question, or end the interview at any time. Most interviews take about 15 minutes, and they are in English only.
- Evidence: each competency is rated 1 to 5 against anchors written for the role. Every competency score must quote the candidate's exact words, checked against the transcript. Two different AI models grade each competency independently, and when the evidence isn't there, or the two disagree about whether it is, the score is left blank.
- Human control: the interview plan is drafted from your job description and reviewed by a recruiter before it is published. Knockout questions flag an answer for review and never end the interview. A person records every decision.
- Deletion: an admin, or whoever created the role, can delete an interview at any time, including when a candidate asks; the recording, transcript, scorecard and candidate details go with it.
- Workflow: ATS integrations are on the roadmap. Today, results leave by CSV or Excel export and a PDF summary for each interview, jobs can be imported from Bullhorn, and teams can sign in with Google or Microsoft.