When Selection Results Should Not Depend on Who Happens to Evaluate the Candidate

The Story

Andrea had just completed a demanding recruitment process.

Twenty candidates. Multiple assessment tools. Structured interviews.Scores collected.
A final ranking ready for presentation.

Everything was carefully organized in a briefcase.

Then something unexpected happened.

During a short stop on his way home, the briefcase was stolen.

Days passed.

Despite attempts to recover the material, Andrea eventually made a difficult decision: he contacted all candidates, apologized, and repeated the entire process the following week.

The same candidates. The same tools. The same interview structure.

But different assessors.

The original evaluators were no longer available.

A week later, the second assessment process was completed.

And at that exact moment, the police called.

The briefcase had been found.

Andrea now had two complete sets of results.

The question was simple:

Would the same candidates still appear at the top of the ranking?

Or would the outcome change dramatically simply because different people had conducted the assessment?

That comparison reveals whether a selection process is truly reliable.

What Does “Accurate & Reliable” Really Mean?

In everyday language, reliability is often taken for granted.

Most organizations assume their hiring processes are reliable.

Yet many selection decisions change depending on:

• who conducts the interview;
• when the assessment takes place;
• how questions are interpreted;
• which contextual factors influence the evaluation.

In organizational psychology, reliability has a precise meaning:

Under equivalent conditions, the result should remain substantially the same.

If the outcome changes dramatically because of the evaluator, the context, or the administration procedure, the problem is not the candidate.

The problem is the process.

🔬 Scientific Perspective

Reliability is not an automatic property of an assessment tool. It emerges from the way tools are designed, administered, scored, and interpreted.

Cognitive Ability Tests

For cognitive assessments, reliability depends heavily on standardization.

Instructions, timing, environmental conditions, and administration procedures should be as similar as possible across candidates.

Even highly predictive tests can lose reliability when administration conditions vary.

Personality Assessments

Personality questionnaires face a different challenge.

Candidates may consciously or unconsciously distort their responses.

Examples include:

• impression management;
• social desirability;
• self-deception;
• fake-good responding.

When these effects are not measured and controlled, candidate comparisons become unstable and potentially misleading.

For this reason, advanced assessment systems often include validity scales and statistical correction procedures designed to improve comparability.

Structured Interviews

Interviews can be reliable—but only under specific conditions.

Questions should be defined in advance.

Evaluation criteria should be shared.

Behaviourally Anchored Rating Scales (BARS) should be agreed upon before the interview rather than invented afterward.

Most importantly, reliability increases substantially when multiple assessors use common evaluation criteria developed during the construction of a Selection Plan.

🤖 Why Reliability Matters More Than Ever in the Age of AI

Artificial intelligence introduces a new challenge.

Candidates increasingly use AI to:

• prepare interview answers;
• simulate behavioural interviews;
• optimize written applications;
• rehearse responses to common recruitment questions.

This does not automatically reduce reliability.

But it increases the need for reliable methods.

When assessment criteria are vague, different evaluators may react differently to highly polished AI-assisted answers.

When evaluation standards are clear and structured, consistency improves.

AI therefore makes reliable processes more important—not less.

The VALID Perspective

Within the VALID framework, reliability is designed into the process from the beginning.

The objective is not simply to collect information.

The objective is to ensure that results remain stable enough to support meaningful decisions.

VALID promotes reliability through:

• structured interviews;
• shared evaluation criteria;
• Behaviourally Anchored Rating Scales;
• multiple assessors where appropriate;
• standardized procedures;
• correction mechanisms for response distortion;
• transparent scoring systems.

Reliability is not left to chance.

It is engineered into the Selection Plan.

👥 For HR Specialists

Questions Worth Asking

Consider your current hiring process.

• Would the same candidate receive a similar evaluation from another recruiter?
• Are interview questions standardized?
• Are assessors using the same evaluation criteria?
• Do your personality assessments address impression management?
• Are your cognitive tests administered under comparable conditions?

If the answer to these questions is unclear, reliability may be lower than expected.

And low reliability inevitably reduces the quality of hiring decisions.

Why Reliability Protects Everyone

For recruiters and HR professionals, reliability means being able to defend a decision.

To management.
To the organization.
And, when necessary, to candidates.

For candidates, reliability means not being evaluated by chance.

It means being assessed according to stable, transparent, and replicable criteria.

When a process is reliable:

• evaluations are less dependent on individual assessors;
• results become more comparable;
• hiring decisions appear fairer and more professional;
• trust in the process increases.

Reliability is therefore more than a technical requirement.

It is a guarantee of fairness.

🎯 Key Takeaway

The second letter of VALID reminds us of a simple principle:

If a process does not produce similar results under similar conditions, it is not a good process.

Even if it is well intentioned.

Even if it is widely used.

A = Accurate & Reliable.

Continue the VALID Series

Next article:

L — Logical & Numerical: Why Selection Decisions Need Numbers, Not Opinions