Daniel Kahneman is notable for his work on biases in human judgement that affects everyday life for all of us. For example, courtroom judges exercise judgmental biases when one judge is more lenient than another. Business people make judgments whether a candidate is more qualified than another candidate. Bankers judge who will get a loan. Kahneman argues that judgmental biases are only part of the problem. The other is that judgments are often noisy. Ask ten different judges to suggest a sentence, and you'll get ten different answers.

Bias judgments

Imagine four different targets - each shot by eight guns. <ul><li>Target 1 has eight holes closest to the bullseye - showing low bias and low noise.</li><li>Target 2 has eight holes close together but above the bullseye - demonstrating high bias but low noise.</li><li>Target 3 has eight holes widely distributed around the bullseye, showing low bias and high noise.</li><li>Target 4 has eight holes widely distributes, showing high noise and bias. </li></ul>

Differentiating between bias and noise

<ul><li>Individual differences, such as the tendency to judge targets roughly or harshly.</li><li>Using vague or imprecise judgement scales.</li><li>Random variation in the order judges views evidence.</li></ul> How to reduce noise:<ul><li>Apply a uniform set of decision rules, leaving no room for subjectivity.</li><li>Different members of a team should judge different dimensions individually.</li><li>Judges should first write down their evaluation before any discussion.</li><li>Use ranking to make relative judgments instead of a scale where everyone can be rated the same.</li></ul>

Sources of noise in human judgement

<h3>Key points</h3><UL><LI>Judges, doctors, personnel committees, and stock portfolio managers make wildly different decisions based on the same evidence.</LI><LI>Kahneman, Sibony, and Sunstein make a strong case that such noisy judgments are common, and lead to as many bad decisions as do cognitive biases.</LI><LI>Noisy judgments are costly, unfair, and sometimes downright dangerous</LI><LI>Kahneman and colleagues offer handy suggestions for reducing costly noise in human judgment.</LI></UL>Daniel Kahneman won a Nobel Prize for his work on biases in human judgment. Those judgmental biases are not just relevant to intellectual nerds interested in dissecting cognitive processes. They matter in everyday life for all of us.If your nephew is convicted of drug possession while on vacation, he may confront a judge who has especially harsh attitudes about drugs and end up having the book thrown at him; or he might luck into an easygoing judge who lets him off with a warning.<div dir="ltr">
 article continues after advertisement
 </div>Courtroom judges aren't the only people who exercise judgmental biases. Businesspeople make judgments about whether a particular candidate is more qualified for a job than the other candidates. University faculty make judgments about accepting or rejecting graduate school applicants, bankers judge who will get a loan at what interest rate. Physicians judge how to diagnose a particular person’s symptoms and which course of treatment to prescribe. Kahneman became an academic rock star because he demonstrated several systematic biases in judgment, such as the <a href="https://www.psychologytoday.com/us/basics/heuristics" title="Psychology Today looks at availability heuristic" hreflang="en" dir="ltr">availability heuristic</a> (we often overestimate the prevalence of things that come readily to mind). But in a new book, he joins Olivier Sibony and Cass Sunstein to argue that judgmental biases, although they have received an immense amount of press, are only a part of the problem (Kahneman, Sibony, and Sunstein, 2021).The other huge part of the problem has been largely ignored – judgments are often very noisy. Ask ten different judges to suggest a sentence for your nephew, and you are likely to get widely divergent answers, as demonstrated in research by legal scholars. This is, of course, unfair.Ask ten different doctors to diagnose a patient with a complicated set of presenting symptoms, and you may well get widely divergent diagnoses. This is not so much unfair as potentially dangerous. Physicians often misdiagnose, which leads to mistreatment that can sometimes lead to death. <img alt="Original figure, based on Kahneman et al." height="384" src="https://cdn.psychologytoday.com/sites/default/files/styles/image-article_inline_full_caption/public/field_blog_entry_images/2021-08/targets.jpg?itok=3yW4kVBA" title="Original figure, based on Kahneman et al." width="684"><div class="caption">Distinguishing bias from noise</div><div>Source: Original figure, based on Kahneman et al.</div>Kahneman et al. presented a useful mnemonic device for distinguishing between <a href="https://www.psychologytoday.com/us/basics/bias" title="Psychology Today looks at bias" hreflang="en" dir="ltr">bias</a> and noise. Imagine four different targets that have each been shot at by eight different gunslingers. One target (see Figure, upper left) has eight holes closely clustered near the bullseye – this demonstrates low bias and low noise. A second target has eight holes clustered around ten inches to the upper right of the bullseye – this demonstrates high bias but low noise. A third has eight holes widely dispersed around the bullseye, showing low bias but high noise. The fourth target has eight widely distributed holes, all leaning to the upper right, demonstrating noise and bias. <div dir="ltr">
 article continues after advertisement
 </div>The authors noted several different sources of noise in human judgment –one problem is individual differences (in the tendency to judge targets harshly or benignly, for example). Another is the use of judgment scales that use vague and imprecise subjective descriptors. Still another is random variation in the order in which judges see different sources of evidence.Kahneman et al. also suggested several helpful hints about how to reduce noise. One is to have all judges apply a strict set of decision rules, with no room for subjective judgment (research has consistently demonstrated that computational formulas outperform even highly expert individual judges). Another is to have different members of a team judge different dimensions separately (in accepting graduate students, one person evaluates their research experience, another evaluates the quality of letters, another evaluates their grades and test scores, for example). Judges should not talk to one another during the evaluation process but record their conclusions before any discussion. Finally, the authors suggested making relative judgments and using ranking (in judging job applicants, or evaluating employees, for example, rank them all against one another rather than using a scale that allows everyone to be rated as “excellent”). Kahneman et al. presented several real-life cases demonstrating the prevalence and importance of noise and addressing some of the common objections to reducing noise. One of the most troubling cases involved judicial penalties for the same <a href="https://www.psychologytoday.com/us/basics/law-and-crime" title="Psychology Today looks at crime" hreflang="en" dir="ltr">crime</a>. Judges like to think that they are tailoring the <a href="https://www.psychologytoday.com/us/basics/punishment" title="Psychology Today looks at punishment" hreflang="en" dir="ltr">punishment</a> to each particular case.<div dir="ltr">
 article continues after advertisement
 </div>However, evidence suggests that different judges judging the same criminal case present wildly different sentences. And the same judge may be harsh or lenient depending on what time of the day they are making the decision.After the evidence about noise in sentencing became widely known, there was an attempt to apply fairer rules by developing standardized penalties for the same crime. Unfortunately, judges rebelled, the corrective laws were overturned, and the unfair variation in the distribution of penalties returned. Noise in human judgment is a serious problem, and it results in a great deal of inefficiency and unfairness. Hence, this ought to be required reading for anyone who ever makes judgments with implications for their future or for the lives of others, which is pretty much every one of us (in my judgment). References<div data-hide-large-count-items="false"><button type="button" data-show-large-count-items="false" dir="ltr">
 
 More
 
 </button></div><div data-hide-large-count-items="false"><button type="button" data-show-large-count-items="false">
 references
 </button></div><UL><LI><img src="https://cdn2.psychologytoday.com/theme-assets/icons/icon-share-round-facebook-md.svg" alt="Share on Facebook" width="36" height="36"></LI></UL>

A new book by Daniel Kahneman and colleagues on the silent problem of noise.

Understand yourself and those around you.

psychology

Psychology

personaldevelopment

Personal Development

problemsolving

Problem Solving

leadershipandmanagement

Leadership & Management

<ul><li>Judges, doctors, personnel committees, and stock portfolio managers make wildly different decisions based on the same evidence. </li><li>Kahneman, Sibony, and Sunstein make a strong case that such noisy judgments are common, and lead to as many bad decisions as do cognitive biases. </li><li>Noisy judgments are costly, unfair, and sometimes downright dangerous </li><li>Kahneman and colleagues offer handy suggestions for reducing costly noise in human judgment. </li> </ul>

Daniel Kahneman won a Nobel Prize for his work on biases in human judgment. Those judgmental biases are not just relevant to intellectual nerds interested in dissecting cognitive processes. They matter in everyday life for all of us.

If your nephew is convicted of drug possession while on vacation, he may confront a judge who has especially harsh attitudes about drugs and end up having the book thrown at him; or he might luck into an easygoing judge who lets him off with a warning.

Courtroom judges aren't the only people who exercise judgmental biases. Businesspeople make judgments about whether a particular candidate is more qualified for a job than the other candidates. University faculty make judgments about accepting or rejecting graduate school applicants, bankers judge who will get a loan at what interest rate. Physicians judge how to diagnose a particular person’s symptoms and which course of treatment to prescribe.

Kahneman became an academic rock star because he demonstrated several systematic biases in judgment, such as the <a dir="ltr" href="https://www.psychologytoday.com/us/basics/heuristics" hreflang="en" title="Psychology Today looks at availability heuristic">availability heuristic </a>(we often overestimate the prevalence of things that come readily to mind). But in a new book, he joins Olivier Sibony and Cass Sunstein to argue that judgmental biases, although they have received an immense amount of press, are only a part of the problem (Kahneman, Sibony, and Sunstein, 2021).

The other huge part of the problem has been largely ignored – judgments are often very noisy. Ask ten different judges to suggest a sentence for your nephew, and you are likely to get widely divergent answers, as demonstrated in research by legal scholars. This is, of course, unfair.

Ask ten different doctors to diagnose a patient with a complicated set of presenting symptoms, and you may well get widely divergent diagnoses. This is not so much unfair as potentially dangerous. Physicians often misdiagnose, which leads to mistreatment that can sometimes lead to death.

Kahneman et al. presented a useful mnemonic device for distinguishing between <a dir="ltr" href="https://www.psychologytoday.com/us/basics/bias" hreflang="en" title="Psychology Today looks at bias">bias </a>and noise. Imagine four different targets that have each been shot at by eight different gunslingers. One target (see Figure, upper left) has eight holes closely clustered near the bullseye – this demonstrates low bias and low noise. A second target has eight holes clustered around ten inches to the upper right of the bullseye – this demonstrates high bias but low noise. A third has eight holes widely dispersed around the bullseye, showing low bias but high noise. The fourth target has eight widely distributed holes, all leaning to the upper right, demonstrating noise and bias.

The authors noted several different sources of noise in human judgment –one problem is individual differences (in the tendency to judge targets harshly or benignly, for example). Another is the use of judgment scales that use vague and imprecise subjective descriptors. Still another is random variation in the order in which judges see different sources of evidence.

Kahneman et al. also suggested several helpful hints about how to reduce noise. One is to have all judges apply a strict set of decision rules, with no room for subjective judgment (research has consistently demonstrated that computational formulas outperform even highly expert individual judges). Another is to have different members of a team judge different dimensions separately (in accepting graduate students, one person evaluates their research experience, another evaluates the quality of letters, another evaluates their grades and test scores, for example). Judges should not talk to one another during the evaluation process but record their conclusions before any discussion. Finally, the authors suggested making relative judgments and using ranking (in judging job applicants, or evaluating employees, for example, rank them all against one another rather than using a scale that allows everyone to be rated as “excellent”).

Kahneman et al. presented several real-life cases demonstrating the prevalence and importance of noise and addressing some of the common objections to reducing noise. One of the most troubling cases involved judicial penalties for the same <a dir="ltr" href="https://www.psychologytoday.com/us/basics/law-and-crime" hreflang="en" title="Psychology Today looks at crime">crime </a>. Judges like to think that they are tailoring the <a dir="ltr" href="https://www.psychologytoday.com/us/basics/punishment" hreflang="en" title="Psychology Today looks at punishment">punishment </a>to each particular case.

However, evidence suggests that different judges judging the same criminal case present wildly different sentences. And the same judge may be harsh or lenient depending on what time of the day they are making the decision.

After the evidence about noise in sentencing became widely known, there was an attempt to apply fairer rules by developing standardized penalties for the same crime. Unfortunately, judges rebelled, the corrective laws were overturned, and the unfair variation in the distribution of penalties returned.

Noise in human judgment is a serious problem, and it results in a great deal of inefficiency and unfairness. Hence, this ought to be required reading for anyone who ever makes judgments with implications for their future or for the lives of others, which is pretty much every one of us (in my judgment).

Have We Been Ignoring the Biggest Flaw in Human Judgment?

Bias judgments

Differentiating between bias and noise

Sources of noise in human judgement

CURATED BY

Related collections

More like this