In 1998, Paul Black and Dylan Wiliam of King's College London published a review of research on formative assessment that became one of the most cited papers in educational research. Titled "Inside the Black Box: Raising Standards through Classroom Assessment," it synthesised evidence from over 250 studies and concluded that improving the quality and consistency of assessment feedback was one of the most powerful levers available for improving student learning outcomes.
The paper was primarily concerned with teaching and learning, not admissions. But its underlying findings, about what makes assessment reliable, what makes feedback useful, and what happens when standards are applied inconsistently, translate directly to the admissions context.
What inconsistency actually costs
When different assessors apply different implicit standards to the same type of evidence, the decisions that result are systematically unfair. Not dramatically, obviously unfair, the kind of unfairness that anyone would immediately notice and object to. Subtly unfair in ways that accumulate across a large number of decisions and that would be impossible to detect without systematic analysis.
An assessor who reads a student's writing early in the intake season, when they are fresh and bringing their full attention to the task, will apply a somewhat different standard than the same assessor reading writing at the end of a long day in the middle of a busy admissions period. An assessor who has just read an exceptionally strong piece of writing will judge the next piece differently than if they had started from a moderate baseline.
These are known effects in assessment psychology. They are not the result of carelessness or bias in the pejorative sense, they are the predictable consequences of how human perception and judgement work. The only solution is structural: to create assessment processes where the criteria are defined, the application is automated where possible, and individual variation is constrained by a consistent framework.
The reliability evidence
Research on marking reliability in educational assessment consistently finds that holistic, impression-based marking has lower inter-rater reliability than criterion-referenced, analytical marking. A study by Lumley and McNamara (1995) examining writing assessment in a high-stakes context found that rater effects accounted for a substantial proportion of score variance, meaning that who marked the paper was contributing as much to the final score as what the student had actually written.
This finding has been replicated across different assessment contexts and age groups. The implication is not that human judgement is unreliable in principle, it is that human judgement without structured support is unreliable in ways that disadvantage applicants who happen to have their work reviewed by different assessors.
Consistency across time as well as across assessors
Inter-rater consistency, different people applying the same standard, gets most of the attention in assessment research. Intra-rater consistency, the same person applying the same standard across different points in time, is equally important and less often discussed.
An admissions process that runs from September to April has to maintain consistent standards across eight months. The criteria that defined a strong Grade 7 applicant in October must be the same criteria applied to a student assessed in March. If threshold settings can drift, if the implicit standard shifts as the applicant pool changes, or if early decisions are not anchored to the same reference points as later ones, the process is unfair in a way that is very difficult to detect from the outside.
Configured pass thresholds, defined per grade and per domain, and applied automatically and identically to every assessment processed, are the structural solution to this problem. They do not prevent professional judgement, they ensure that professional judgement is applied on top of a consistent foundation rather than in place of one.
What consistency means for defensibility
There is a governance dimension to consistency that sits alongside the fairness argument. A school that applies consistent, documented standards to every admissions decision is in a fundamentally stronger position if those decisions are ever questioned. "We applied the same threshold to every applicant" is a complete and defensible answer to a challenge about how a specific decision was made. "We used our professional judgement on each case" is not.
This matters increasingly as international schools operate in regulatory environments where accountability for admissions decisions is growing. Consistency is not bureaucracy, it is protection.
See how Evalent builds defensibility into every admissions decision.
See this in practice
Book a demo of Evalent Admissions
See how a report, your thresholds and the decision record fit together, and ask us anything about your own admissions process.