How Accurate Is AI Resume Screening? How to Measure It Yourself

Nobody can tell you how accurate an AI resume screening tool is on your roles without testing it on your roles. Accuracy depends on what you ask it to find, the quality of the criteria, the format of the resumes and what you count as a correct answer. A figure from a vendor page, with no description of the test, tells you very little. You can run a useful test in an afternoon with thirty to fifty resumes and a spreadsheet, and the method below works for any tool, including ours.

For background on what these tools do and where they go wrong, start with AI resume screening: how it works and where it fails.

Why a quoted accuracy number is hard to use

A percentage only means something if you know four things.

• What it was compared against. Human reviewers disagree with each other, so "agrees with recruiters" depends on which recruiters.
• What counted as correct. Picking the person eventually hired is one standard. Matching a shortlist is another. They produce different numbers.
• Which resumes were used. Clean, single-column resumes for a common role are easier than scanned multi-column CVs for a niche one.
• Whether the test data was seen in training.

Without those, a number cannot be compared between tools or applied to your pipeline. Ask for the test description, and if there is none, treat the number as a slogan.

A test you can run this week

1. Pick one role with known outcomes

Choose a role you have already hired for, so you have a view of who was strong. Pull 30 to 50 applications: some who were hired or reached final rounds, some clear mismatches and a spread in the middle.

2. Score them blind, by hand, first

Have two reviewers score each resume yes, maybe or no against the written criteria, separately. Record where they disagree. That disagreement rate is your human baseline, and it is usually higher than people expect. A tool cannot be expected to agree with humans more than they agree with each other.

3. Run the tool on the same set, with the same criteria

Use the same written requirements. If the tool lets you set criteria, copy the scorecard exactly. If it only reads the job description, use the same one the reviewers had.

4. Compare, then read the disagreements

The numbers are secondary. What you want is the reasons. A tool that explains each score by pointing to lines in the resume lets you see in minutes whether it misread something. A tool that returns a bare score does not.

5. Test the awkward cases deliberately

Add a few resumes designed to cause trouble: a multi-column layout, a career changer, a candidate who describes the right skills in unusual words, a resume with a long gap, one with a non-English section. How the tool handles these matters more than how it handles the easy ones.

Pass marks worth setting

Set them before running the test, not after.

• No candidate your reviewers rated strong lands in the bottom group without a reason you accept.
• Disagreements with your reviewers are no more frequent than the reviewers' disagreements with each other.
• Every score can be traced to evidence in the resume.
• Resumes in common formats parse without lost sections.

What the test does not cover

A 50-resume test cannot prove the tool is fair across groups of candidates. That needs a separate analysis of outcomes at scale, and in some places it is a legal requirement. See testing for bias in AI hiring and the four-fifths rule for how that works. It also does not tell you how the tool behaves on roles very different from the one you tested, so repeat it when you add a new kind of job.

Using the result

Treat the output as a first pass that orders the pile, with a person reading everything near the top and anything flagged. The score should save you reading, not replace the read. Resume World is built that way: criteria you define, scores traceable to the resume, and a ranked list that a recruiter reviews. You can try that test on the free plan; see pricing and the guide to choosing screening software.

Measure more than agreement

Agreement with reviewers is one measure. Three others are worth tracking once the tool is live.

• Precision of the top band. Of the candidates the tool puts in the strong group, how many pass your phone screen? If that share is lower than for candidates you shortlisted by hand, the criteria or the scoring need work.
• Misses found by sampling. Each month, read 20 resumes from the bottom band. Count how many you would have advanced. A rising count is an early warning that the role changed and the criteria did not.
• Review time. Compare minutes spent per hire-ready shortlist, before and after. A tool that is accurate but leaves you re-reading everything has saved nothing.

Keep these numbers for yourself. They are the only figures about your pipeline that you can trust, because you measured them.

A note on test data

Use resumes from real past applicants only where your privacy notice and retention rules allow it, and strip names and contact details first. If you cannot use real resumes, build a set of 30 synthetic ones that include the awkward cases above. It is less realistic, but it still shows how the tool handles layout, wording and gaps. Retention rules are covered in GDPR and recruitment data and how long to keep candidate data.