Sources of noise in human judgement

• Individual differences, such as the tendency to judge targets roughly or harshly.
• Using vague or imprecise judgement scales.
• Random variation in the order judges views evidence.

How to reduce noise:

• Apply a uniform set of decision rules, leaving no room for subjectivity.
• Different members of a team should judge different dimensions individually.
• Judges should first write down their evaluation before any discussion.
• Use ranking to make relative judgments instead of a scale where everyone can be rated the same.