Koehler (1993) calls for a much-needed re-analysis of the base-rate fallacy. In this commentary, I discuss the importance of a re-analysis of the work on base rates in the applied field accounting. Base rate data appear to be an important input for accounting tasks.
1. In accounting, academic researchers frequently use findings from psychology to better understand the dynamics of judgment and decision making by professional accountants. One such instance concerns the so-called base-rate fallacy. According to professional accounting standards, base rates should be used by accountants in many of their judgments and decisions. However, when accounting researchers replicated the results of Kahneman and Tversky (1973) in accounting contexts, they concluded that base rates were ignored. As Koehler (1993) had indicated, such premature conclusions persisted despite substantial research suggesting that professional accountants do actually use base rates. This persisting effect was unfortunate, because it led academic accounting researchers to misfocus their research efforts during the 1980's.
2. Koehler's critical analysis of the research on the base-rate fallacy is important because it will help researchers in applied domains, such as accounting, re-analyze previously generated conclusions about the base-rate fallacy (as well as conclusions about heuristics and biases in general). Within the last several years, academic accounting researchers have begun to realize that contextual features in accounting might have a unique influence on the judgment and decision making behavior of accountants (Smith and Kida, 1991). Koehler's target article provides a much needed account of why such contextual features should be considered. Indeed, his paper provides a foundation for future research which re-examines the use of base rates in applied domains, such as accounting.
3. As noted by Smith and Kida (1991), early studies in accounting, using subjects asked to perform highly abstract and unfamiliar accounting tasks, had concluded that base rates were not used. For example, Johnson (1983) asked undergraduate accounting majors to make judgments as to the probability that each of 12 firms went bankrupt. Subjects were given the base rate of bankruptcy for the industry to which the firm belonged plus individuating information for each of the 12 firms. Consistent with Kahneman and Tversky's (1973) findings, base rates did not have a significant effect on the subjects' probability judgments.
4. In contrast, other experiments, which required experienced professional accountants to undertake familiar, job-related tasks, suggested a different picture. These studies generally indicate that even though the judgments of experienced accountants performing familiar tasks are not normative, they are more normative than the judgments of inexperienced student subjects. Even more important, however, experienced accountants do not ignore base rates:
5. In a widely cited accounting article published in 1981, Joyce and Biddle asked professional auditors to estimate the probability of management fraud when a key manager's personality profile matched a master profile of fraudulent managers. The results show a significant main effect for the base-rate manipulation. Although the auditor subjects underweighted base-rate information, they did use it in their analysis.
6. Kida (1984) also reported that professional auditors do use base rate data. Specifically, in an experiment using an audit case that corresponded to the professional responsibilities of the audit-partner subject group, he was able to show that 79% of the audit partners did use base rates. Moreover, his analysis indicates that attention to base rates was enhanced when they were given a causal interpretation.
7. More recently, Libby (1985) shows that when auditors are evaluating unusual financial trends, they initially generate the most frequently occurring potential causes of those trends. These results provide indirect evidence that auditors attend to the base rates of financial statement errors. Butt (1988) provides converging evidence by noting that auditors are able to learn base rates of financial-statement errors most easily by directly experiencing those errors.
8. This research in accounting using experienced subjects performing familiar tasks calls into question the generalizability of the base-rate fallacy. It is noteworthy that this research supports Koehler's assertion that the extant literature does not support the widely held belief that people ignore base rates. Indeed, the accounting research suggests that perhaps the experience of decision makers and their familiarity with the task influence their use of base rate. That researchers need to match (on an ex ante basis) the decision maker with the decision task is a suggestion that appears consistent with Koehler's position: Researchers examining the effects of base rates must examine base-rate usage in realistic decision problems and environments.
9. In response to the uncertainty regarding why some studies report that individuals use base rates while others report that they ignore them, Koehler calls for an empirical program that examines real-world base-rate usage and embraces more flexible standards of acceptable performance. This is an excellent suggestion, for several reasons.
10. First, there are many features found in the accounting environment that are not present in many of the psychology studies which examine base-rate use. For example, the accountant's substantial domain-specific experience (Libby and Frederick, 1990) and aversion to compromising effectiveness (Beck, Solomon, and Tomassini, 1985), time pressure (McDaniel 1990), and accountability to others for work completed (Johnson and Kaplan, 1991) are all features which distinguish accounting from other domains. Although not conducted within the base-rate arena, prior research has shown that such contextual features can cause accountants' judgments and decisions to be dramatically different from those of non-accountants.
11. To illustrate this, consider the work on confidence. Whereas most of the psychological literature reports that people tend to be overconfident in their judgments and decisions (Einhorn, 1980), accounting research indicates that accountants tend to be underconfident in their judgments and decisions (Tomassini, Solomon, Romney, and Krogstad, 1982). This underconfidence effect appears to be a manifestation of accountants' tendencies toward conservatism -- a distinctive characteristic of the incentives in the accounting environment. As suggested by Koehler, additional research is necessary to determine the extent to which unique features of the accounting domain influence the use of base rates by accountants.
12. Second, unlike many of the domains studied in the prior psychological research, there is often no timely revelation of an outcome referent or external criterion which can be used to gauge correctness in accounting (Davis and Solomon, 1989). For example, in judging the inherent risk of an audit situation, considering the reliability of a company's system of internal controls, or planning the audit approach, there really is no correct answer. As a consequence of this lack of a correctness measure in accounting, researchers have turned to other measures of "good" performance. In particular, measures of judgment/decision justifiability and defensibility have become important for judging effectiveness in accounting (see ENDNOTE #1).
13. Such a focus away from "accuracy" is consistent with the basic premise of Koehler's analysis. He suggests that we relax our notion of what constitutes an appropriate response to accommodate the unique characteristics of different judgment/decision contexts. Doing so will allow us to expand our notion of what constitutes acceptable behavior in terms of reliance on base rates.
14. Even in those accounting situations where there is a correct answer, relying on the Bayesian model to determine the "correct" course of action is not always optimal. This point was noted by Koehler. To illustrate this in the accounting domain, consider the following example: The base rate of fraud in the United States is quite low. If, during the course of a financial-statement audit, an auditor were to uncover some evidence suggesting that fraud might be present, most audit professionals would indicate that the "best" course of action would be to investigate the potential fraud vigorously. Relying solely on a Bayesian posterior probability (with the base rate of fraud used as a prior probability) as a guide for subsequent behavior would probably fail be optimal in the auditing domain, in light of the extremely high cost to the auditor of failing to investigate and discover a fraud.
15. In summary, the time has come to challenge "matters of established fact." Koehler's target article sounds a warning that should be heeded by all: Science cannot be built with complacency. Indeed, as scientists, we must endeavor to pursue knowledge--even at the expense of exposing our ignorance. And ignorance may well be the appropriate descriptor in the case of the base rate fallacy.
#1. For example, when performing the audit, an auditor may decide to test and rely heavily on a company's system of internal controls. As a result of this decision, the auditor would need to perform a relatively small number of detailed tests of the company's year-end financial statements. Alternatively (and equally acceptable), an auditor may decide not to test, and to rely instead on controls. This approach would require an increased number of detailed tests of the year-end financial statements. As long as the auditors justify and defend their approach in light of the circumstances, however, either of these two approaches should be equally defensible to outsiders (e.g., other auditors, the courts, etc.).
Beck, P., Solomon, I., & Tomassini, L. (1985) Subjective prior probability distributions and audit risk. Journal of Accounting Research 23:37-56.
Butt, J. (1988). Frequency judgments in an auditing-related task. Journal of Accounting Research 26:315-330.
Davis, J. & Solomon, I. (1989) Experience, expertise, and expert-performance research in public accounting. Journal of Accounting Literature 8:150-164.
Einhorn, H. (1980) Overconfidence in judgment. In R. Shweder & D. Fiske (eds.) New directions for methodology of social and behavioral science: Fallible judgment in behavioral research. San Francisco: Jossey-Bass, 1-16.
Johnson, W. (1983) "Representativeness" in judgmental predictions of corporate bankruptcy. The Accounting Review 58:78-97.
Johnson, V. & Kaplan, S. (1991) Experimental evidence on the effects of accountability on auditor judgments. Auditing: A Journal of Practice and Theory (Supplement) 10:96-107.
Joyce, E. & Biddle, G. (1981) Are auditors' judgments sufficiently regressive? Journal of Accounting Research 19:323-349.
Kahneman, D. & Tversky, A. (1973) On the psychology of prediction. Psychological Review 80:237-251.
Kida, T. (1984) The effect of causality and specificity on data use. Journal of Accounting Research 22:145-152.
Koehler, Jonathan J. (1993) The Base Rate Fallacy Myth. PSYCOLOQUY 4(49) base-rate.1
Libby, R. (1985) Availability and the generation of hypotheses in analytical review. Journal of Accounting Research 23:648-667.
Libby, R., & Frederick, D. (1990). Experience and the ability to explain audit findings. Journal of Accounting Research 28:348-367.
McDaniel, L. (1990) The effects of time pressure and audit program structure on audit performance. Journal of Accounting Research 28:267-287.
Smith, J., & Kida, T. (1991) Heuristics and biases: Expertise and task realism in auditing. Psychological Bulletin 3:472-489.
Tomassini, L., Solomon, I., Romney, M., & Krogstad, J. (1982) Calibration of auditors' probabilistic judgments: Some empirical evidence. Organizational Behavior and Human Performance 30:391-406.