Decades of research on human judgment and decision making have demonstrated the presence of cognitive biases. This literature has led to a negative view of our judgmental capacities, a view that Krueger (1998) laments. However, the road to a more positive perspective first requires a clearer picture of the extent of bias and of methods for combating it. Rather than continue to debate the existence of biases, we should strive to understand their prevalence and magnitude, catalog them by source, and address them through corrective procedures. Some of Krueger's suggestions appear highly relevant for these goals.
2. Researchers in the area of judgment and decision making have generated impressive lists of specific biases or cognitive errors observed in human judges. The limitation of null hypothesis significance testing (NHST) in this field, as in many others, is that it makes a fairly unimportant, categorical decision regarding the presence or absence of a phenomenon while failing to provide an important, quantitative estimate of its magnitude. Krueger's suggestions fail to address this limitation.
3. Krueger's examples do not support the need to modify conventional statistical practices. In the case of the false consensus effect, classic studies purporting to demonstrate this bias have been criticized for using an overnarrow definition of rationality, or "correct" judgments (Dawes 1989). Subsequent research has addressed this problem not through novel statistical practices, but through a reformulation of the normative standard of comparison (Krueger & Clement 1994; Krueger & Zeigler 1993). The same holds true in the case of enhancement bias, where one solution to the potential problem posed by a skewed distribution of abilities is to have participants make judgments in terms of percentiles (e.g., Svenson 1981). Krueger persuasively argues the need to critically evaluate the normative standard against which judgments are compared, but this does not imply that statistical changes would be useful.
4. For example, the Bayesian alternative espoused by Krueger, given its ability to retain a well-established empirical trend in the face of a single deviant observation, serves to promote a categorical decision model concerned solely with the presence or absence of bias. Other, more conventional practices, such as estimating effect sizes and combining them through meta-analysis, provide richer and, as discussed in section III, more pertinent information.
5. Krueger portrays the detection of subtle biases as a threat to positive views of human judgment, but the real threat stems from those who interpret all magnitudes of bias as equally problematic and impugning of our judgmental abilities. A virtue of the scientific method is that large-scale experiments with sensitive instruments allow us to examine subtle phenomena. It is emphatically not the case, for example, that "the more perceivers there are, the more it will seem that they are biased" (Krueger, 1998, paragraph 18). This may be a common overextension of a statistically significant though-subtle effect, but the statistics alone do not suggest it. The more perceivers there are, the more sensitive is the experiment in detecting a subtle bias, which should of course be interpreted in this manner if the bias is small in magnitude. The detection of all biases, regardless of their potency, provides the most accurate depiction of reality and indicates where corrective procedures might be beneficial.
6. Finally, Krueger suggests the examination of individual differences as another way to shield positive views of social judgment from negative results. However, it would be more worthwhile to use measures of individual differences to assess the prevalence of a bias. If a particular group deviates from appropriate standards of rationality, at least some of its individual responses must deviate in this way. Along with an estimate of effect magnitude, an estimate of the prevalence of a specific bias among a particular group or sample would be worth cataloguing in order to inform the development of corrective procedures.
7. We must understand more than just the magnitude and prevalence of biases if we hope to develop successful corrective procedures. Arkes (1991) has introduced a typology of biases that organizes them according to source: strategy-based, association-based, or psychophysically-based. Interventions are likely to differentially influence biases stemming from different sources. For example, encouraging individuals to explicitly consider alternative scenarios may reduce association-based errors, whereas simply offering incentives for improved judgment may have no effect on these biases. Therefore, identifying the source of error is a critical first step in approaching the task of debiasing.
8. Who among us believes that our judgments are truly infallible, conforming to all relevant rational prescriptions? Surely nobody would care to defend this position. Likewise, who among us believes that our judgments are wholly irrational, deviating markedly from normative guidelines? Again, nobody would take this view. Clearly, then, the critical issue concerns the nature and extent of the various biases that influence our judgment. Krueger hints at this issue when he discusses an idiographic approach that examines both accuracy and bias. As noted above, the merit of this approach is not that it denotes the presence or absence of bias, but that it permits estimation of the prevalence of a bias. Armed both with knowledge of how problematic a bias is and with information about its source, researchers can begin to explore methods of counteracting various judgmental biases. Presumably, research would initially target biases that are particularly problematic, commonly observed, and attributable to known sources. Debiasing research could then spread to more subtle biases whose sources are less clear.
9. Hammond (1996) makes a distinction that seems highly relevant to discussions about the presence of bias and reduction of biases. He articulates two independent views of accurate judgment: coherence and correspondence. Judgments that are logically consistent with one another and conform to rational standards are coherent, and judgments that are in empirical agreement with external reality are correspondent. Krueger (1998) equates correspondence with accuracy and poor coherence with bias (paragraph 19), but this mapping is incomplete. Poor correspondence also indicates bias, and good coherence indicates accuracy. Examination of both the coherence and correspondence of judgments would reveal the degree of accuracy and bias in each; these are not all-or-nothing phenomena.
10. The development of corrective procedures, arguably the most critical application of research in judgment and decision making, must consider both the coherence and the correspondence of judgments. Krueger proposes an excellent way to study these characteristics: sample not only judges, but also judgment items. By repeatedly testing individuals, one can test the coherence of judgments according to rational standards as well as comparing the correspondence of these judgments to some external criterion. It is essential to note the degree of accuracy and bias on both counts, and to bear in mind that coherence does not necessarily lead to correspondence (and vice versa).
11. As an example of this research strategy, Ruscio (1998) investigated the accuracy and confidence of clinical predictions -- evaluated in terms of both coherence and correspondence -- by systematically manipulating two experimental factors (social accountability and cue profiles), assessing one subject variable (need for cognition), and having judges make a series of predictions and confidence estimates. Results shed light on the judgment process, individual differences, the prevalence and magnitude of several biases (some very large and others more subtle), and potential methods for reducing these biases. Krueger is absolutely right that multiple theories and methods will help to advance the field. However, the particular problems to which he devotes attention, such as conventional statistical practices, are not necessarily the ones that present the greatest impediments to progress at this time.
Arkes, H. R. (1991). The costs and benefits of judgment errors: Implications for debiasing. Psychological Bulletin 110: 486-498.
Dawes, R. M. (1988). Rational choice in an uncertain world. Harcourt-Brace-Jovanovich.
Dawes, R. M. (1989). Statistical criteria for a truly false consensus effect. Journal of Experimental Social Psychology 25: 1-17.
Hammond, K. R. (1996). Human judgment and social policy: Irreducible uncertainty, inevitable error, unavoidable injustice. Oxford University Press.
Krueger, J. (1998). The bet on bias: A foregone conclusion? PSYCOLOQUY 9(46) http://www.cogsci.soton.ac.uk/cgi/psyc/newpsy?9.46 ftp://ftp.princeton.edu/pub/harnad/Psycoloquy/1998.volume.9/psyc.98.9.46.social-bias.1.krueger
Krueger, J., & Clement, R. W. (1994). The truly false consensus effect: An ineradicable and egocentric bias in social perception. Journal of Personality and Social Psychology 67: 596-610.
Krueger, J., & Zeigler, J. S. (1993). Social categorization and the truly false consensus effect. Journal of Personality and Social Psychology 65: 670-680.
Plous, S. (1993). Psychology of judgment and decision making. Temple University Press.
Ruscio, J. (1998). The role of complex thought in clinical prediction: Social accountability and the need for cognition. Manuscript under review.
Svenson, O. (1981). Are we all less risky and more skillful than our fellow drivers? Acta Psychologica 47: 143-148.