Most discussions of fairness in admissions focus on bias: the systematic tendency to favour certain types of applicants for reasons unrelated to their academic potential. Bias is important and worth addressing. But in 2021, Daniel Kahneman, Olivier Sibony, and Cass Sunstein published "Noise: A Flaw in Human Judgment," which argued that a different and less-discussed problem may be equally damaging to decision quality.
Noise, as they define it, is variability in judgments that should be identical. Where bias is a systematic skew in one direction, noise is random scatter. Two judges sentencing the same defendant reach different conclusions. Two doctors assessing the same patient disagree on the diagnosis. Two insurance underwriters evaluating the same risk quote meaningfully different premiums. The same decision-maker on a different day may reach a different conclusion than they did this morning.
The insurance company study
One of the most striking demonstrations in the book involved a major insurance company. The company's executives estimated that two underwriters assessing the same case would differ in their quotes by about 10 per cent. When the researchers ran an experiment, asking 48 underwriters to price the same set of cases, they found the actual median difference was 56 per cent. The executives were off by a factor of more than five.
This is not an unusual finding. Kahneman and his colleagues reviewed studies across medicine, criminal justice, hiring, and professional evaluation, and found high levels of noise to be the rule rather than the exception wherever human judgment is exercised on complex cases without structured support.
Occasion noise: the same person on a different day
One of the more uncomfortable findings in the noise research is the magnitude of occasion noise: the variation in a single person's judgments across different points in time. Studies of experienced professionals across multiple domains have found that the same person assessing the same case on different occasions will often reach noticeably different conclusions, even without any new information.
In admissions terms, this means that a borderline application reviewed in the morning after a good night's sleep may receive a different judgment than the same application reviewed in the afternoon after a difficult day. The assessor is not being negligent. They are exhibiting a property that Kahneman and colleagues show is universal in human judgment under conditions of genuine uncertainty.
What reduces noise
Kahneman, Sibony and Sunstein identify structured, rule-based decision processes as the primary tool for reducing noise. When the criteria are defined in advance, when the evidence is gathered consistently, and when the judgment required is anchored to specific benchmarks rather than formed holistically, noise decreases substantially.
This is not an argument against professional judgment. It is an argument for creating a framework within which professional judgment is exercised. An admissions assessor who reads a structured report comparing a student's domain scores against configured thresholds, with specific written commentary on each domain, is working with less noise than one who forms an impression from a folder of unstructured documents.
The noise research provides one of the strongest intellectual foundations for the shift from informal, impression-based admissions to structured, criterion-referenced assessment. Not because human judgment is untrustworthy, but because the conditions under which it operates in admissions are exactly the conditions the research identifies as noise-generating.
Source: Kahneman, D., Sibony, O., and Sunstein, C. R. (2021). Noise: A Flaw in Human Judgment. William Collins.
See how Evalent reduces noise in admissions decision-making.
See this in practice
Book a demo of Evalent Admissions
See how a report, your thresholds and the decision record fit together, and ask us anything about your own admissions process.