A Conversation with Daniel Kahneman About “Noise” - By Evan Nesterak - Behavioral Scientist

You can do a noise edit if you have several employees performing an interchangeable function, like different federal judges, doctors,  or different underwriters in an insurance company.

In a noise audit, people are given a realistic problem that they could encounter on their job. Then look at the variability of judgments. If the judgements are variable, then errors are variable.

If you realise that there's more variability than you expected, you have several possibilities:

• If the judgement is relatively simple, consider if you actually need human judgment or if you can replace judgment with some rule or algorithm. For example, the Apgar score, how to decide whether infants are healthy, is a rule.
• In more complex cases, try to discipline judgment with "decision hygiene". It is a disciplined process where people making judgments follow the same thought process and likely reach similar conclusions.

Decision hygiene is intended to reduce noise, but it is also applicable to single decisions where noise is invisible.

When facing a decision with multiple options, treat options like job candidates. Break up the problem and evaluate various aspects of the option, just like you would break up various traits of candidates. Do that while keeping the specific judgements:

• fact-based as much as possible
• as independent of each other as possible

Wherever there is judgment, there will be noise. When trying to reduce noise, you can expect some pushback as people perceive what is happening as limiting their options and as a bureaucratic infringement on their role.

Employees who are affected by this have to see this as something that helps them do their job, rather than something that replaces them or constrains them too much. 

The purpose is not to eliminate intuition, but to delay it - to engage in a disciplined thought process before they form an intuition or global judgment.

Reducing noise is primarily for organizations that make judgments and decisions, and it wants to do that with one voice. Noise is unwanted variability.

In many other situations, diversity is actually interesting and valuable, for example, film reviewers and creative processes. There are also situations where a team with diverse expertise make partial judgments that you don't want to be identical. You want them to reflect different characteristics of the problem.