Why Automated Scoring Is Not the Same as Arbitrary Scoring

The objection is understandable. An experienced teacher reading a student's extended writing response brings genuine expertise to that task. They understand register, voice, argument structure, and the difference between a student who is struggling to express a sophisticated idea and one who simply does not have a sophisticated idea to express. Can an automated system do that?

The honest answer is: partly. Automated writing evaluation does some things very well and some things less well. The question is not whether it is perfect, nothing in assessment is, but whether it is more reliable and consistent than the available alternatives.

What automated scoring does reliably

Automated scoring systems trained on large datasets of human-scored writing samples have demonstrated strong reliability on measurable writing features: sentence-level accuracy, vocabulary range and sophistication, paragraph structure, text length and development, and coherence of argument. For these features, automated scoring is typically highly consistent, the same response will receive the same score every time, which is a property that human marking does not share.

Research comparing automated and human scoring of extended writing, including studies by Educational Testing Service on e-rater, and by ACT on their automated scoring systems, consistently finds that automated scores correlate with expert human scores at levels comparable to the correlation between two independent human markers. In other words, automated scoring disagrees with a human rater at roughly the same rate as one human rater disagrees with another.

What automated scoring does less well

Automated systems are less reliable at evaluating features that require contextual or cultural knowledge: humour, irony, culturally-specific references, unconventional structure that works, and the kind of genuine intellectual originality that breaks expected patterns in interesting rather than confused ways.

For school admissions at Grades 3-10, these limitations are less consequential than they would be in literary analysis or university admissions. The writing task in a structured admissions assessment is not asking for literary sophistication, it is asking for evidence of academic writing capability. Organisation, vocabulary, accuracy, and task engagement are the relevant criteria, and these are the features that automated evaluation handles well.

The comparison that matters

The relevant comparison is not automated scoring versus an ideal human expert with unlimited time and no fatigue. It is automated scoring versus the actual alternatives available in a school admissions context: a busy assessor reviewing 40 writing samples in an afternoon, with variable attention, variable standards, and the full range of cognitive biases that affect human judgement.

Against that baseline, automated scoring has a structural advantage. It applies the same criteria to every sample in the same way regardless of how many samples it has already processed. It does not mark more harshly after a sequence of strong samples. It does not give more benefit of the doubt to students whose written voice is similar to the assessor's own.

The appropriate use of automated evaluation in admissions is not to replace human judgement but to provide a consistent, rubric-based foundation that human judgment can then build on, reading the verbatim responses alongside the structured evaluation, and applying professional expertise to the interpretation rather than the initial scoring.

See how Evalent Admissions evaluates student writing.