The short answer
Test an AI recruiter using clearly fictional profiles for one specific vacancy. Establish what information each criterion needs and how an authorised reviewer handles uncertainty. Compare the same profiles and questions with an independent human assessment. Record disagreements and investigate their causes. This can help identify errors, but does not prove future quality, fair treatment or legal compliance.
Bring one vacancy. In 30 minutes, we discuss where your recruitment is getting stuck, which approach could fit and your recruitment fee.
Why a Single Vacancy Is the Right Test Environment
Rolling out an AI recruiter broadly without prior testing is a risk you can easily avoid. Start with one technical vacancy – ideally a role for which you have already formulated the assessment criteria precisely. This limits the scope whilst still testing real complexity: technical requirements, experience levels, and the nuances that belong to a specific team.
The question is not whether the AI works quickly. The question is whether it surfaces the right profiles and sets aside the right ones. That difference demands a structured approach – not gut instinct.
Build an Acceptance Protocol with Synthetic Profiles
Start with fictional candidate profiles that are not based on identifiable applicants. Include suitable, unsuitable and uncertain situations. The set’s size depends on the role and variation; no universal number makes a test reliable.
For each profile, record the available answers, open questions and expected assessment. Include reviewers’ own uncertainty. Do not retrospectively redefine the expected outcome to match the AI’s result.
Deliberately Test the Hard Cases
Distinguish unclear wording, genuinely conflicting statements and missing information. A qualification or salary level alone does not establish a contradiction.
Fictional example: a profile claims both responsibility for a team and exclusively hands-on duties in the same period. The expected next step is to clarify the tasks. Also test someone with relevant practical experience who uses different terminology. This helps check whether wording unintentionally drives the assessment.
Frozen Dataset, Independent Human Judgement
Retain the test set, questions, criteria and available system settings for each round. Have a recruiter and a subject-matter expert assess independently before seeing the AI result. Record not only agreement but also missed suitable profiles and profiles assessed too favourably.
Change one element deliberately and repeat the comparison. A small test set can miss rare errors. Also assess data processing, candidate information and the effects of automatic exclusion. A human making the final hire does not automatically make earlier exclusion human-led.
Discuss what you need to learn from a test
Establish which findings would require configuration changes, further investigation or postponing rollout. Do not confuse a successful demonstration with proven performance with real candidates.
This test approach is a recommended assessment method, not a description of a standard or free Red Rocket pilot product. Discuss your vacancy and what you need to understand about AI qualification. Include someone who can explain the actual work and criteria.
Frequently asked questions
How many profiles do you need?
There is no universal minimum for a reliable test. Cover important criteria and varied edge cases and state what the set does not cover.
Can you use real candidate profiles?
Prefer fictional profiles to start. Testing with real personal data requires a separate assessment of purpose, legal basis, access and protection. Do not copy files without that assessment.
What if the AI and the human assessor consistently disagree?
Then you have valuable information. Note the nature of the divergence: is it about how criteria are interpreted, about missing context in the profile, or about a system configuration? That analysis determines whether the assessment criteria need tightening, or whether something in the AI's configuration requires adjustment.
