Evaluate consistency without claiming neutrality
Course overview · 4 min reading + 12 min practice, estimated
Principles and method
Compare how similar evidence is scored and how exceptions are handled. Calibration samples can reveal variation, but agreement does not prove that the underlying criterion is valid or fair. A panel can consistently apply a flawed rule. Review criteria as well as ratings, and involve appropriate expertise for discrimination or data-protection concerns. Track changes and their effects over time. Communicate what the controls do and what remains uncertain. Avoid marketing a human or AI process as unbiased simply because a checklist was completed.
Worked example
All assessors agree on rejecting candidates without a particular university background. High agreement does not justify the criterion. The review asks whether the requirement has a defensible relationship to the work.
Put it into practice
Write a consistency audit that examines both scoring agreement and criterion relevance.
Use fictional information and keep your work in your own notes.
Compare your approach: self-review guidance
Include a case where everyone agrees for the wrong reason. Explain how the audit tests the rule itself and what evidence would support changing it. Consistency is one quality dimension, not a complete fairness guarantee.
Sources and further reading
Original Academy teaching and fictional examples. These references provide context, not endorsement. Edition 2026.09; updated 2026-09-24.
- CIPD: Selection methods
Professional context for selection methods and their limitations.
- GOV.UK: Discrimination in recruitment
UK legal context. Obtain advice for your jurisdiction and circumstances.