We have been oversold on the base rate fallacy in probabilistic judgment. First, few tasks map unambiguously into the simple, narrow framework that is held up as the standard of good decision making. Second, the literature does not support the widely held belief that people ignore base rates. Third, we know very little about how the ambiguous, unreliable and unstable base rates of the real world are and should be used by decision makers who have complex performance goals. It is suggested that the existing research paradigm be replaced by an empirical program that examines real world base rate usage and embraces more flexible standards of acceptable performance.
1.1. Psychologists, particularly behavioral decision theorists, routinely use Bayes' theorem as a normative model for aggregating base rates [see ENDNOTE #1] with other probabilistic information (Fischhoff & Beyth-Marom, 1983; Meehl & Rosen, 1955; Slovic & Lichtenstein, 1971; von Winterfeldt & Edwards, 1986) [see ENDNOTE #2]. While an increased attention to base rates and Bayesian methods will sometimes improve predictive accuracy, there is no single, clear normative standard for using base rate information in most realistic decision situations. This is not because Bayesian methods lead to unresolvable paradoxes (Jonakait, 1983) or are logically flawed (Brilmayer & Kornhauser, 1978; Bergman and Moore, 1991), as some have intimated. Instead, it is because base rates do not map into the Bayesian framework in most real world problems.
1.2. The failure to appreciate this point has led to a vast oversale of the so-called "base rate fallacy" in the probabilistic judgment literature. According to this fallacy, people routinely ignore base rates and it is an error to do so. The normative portion of the fallacy is widely regarded as a self-evident implication of Bayes' Theorem. In most experiments, base rates are equated with prior probabilities, and deviations between subjects' judgments and the Bayesian posterior probability are used to measure the extent to which the base rate fallacy has been committed. The descriptive portion of the fallacy, it is said, has been demonstrated so often that one commentator has concluded that "The genuineness, the robustness, and the generality of the base-rate fallacy are matters of established fact" (Bar-Hillel, 1980, p. 215; see also Bar-Hillel, 1983; Borgida & Brekke, 1981).
1.3. The present target article offers a different perspective on the base rate fallacy. It is contended that both the normative and the descriptive components of the base rate fallacy have been exaggerated. Indeed, it is far from clear that people misuse base rates in any important contexts.
1.4. At the normative level, we must bear in mind that there is an important difference between identifying a theoretically sound normative rule for aggregating probabilistic information of a certain sort (i.e., prior odds and likelihood ratios) and fairly applying that rule to a broad class of base rate tasks. Applicability of the rule depends critically on how well the task or, more precisely, the decision maker's representation of the task, meets the assumptions of the rule. Where key assumptions are violated or unchecked, the superiority of the normative rule reduces to an untested empirical claim. No studies have addressed this issue.
1.5. At the empirical level, there is little to support widely repeated claims that people ignore base rates (see e.g., Christensen-Szalanski & Bushyhead, 1981, p. 931; Evans & Bradshaw, 1986, p. 16; Ginossar & Trope, 1987, p. 464; Nisbett & Borgida, 1975, p. 935; Pollard & Evans, 1983, p. 124). A fair review of the literature -- i.e., one that does not start and end with the 1970s "classics" -- suggests that base rates almost always do influence judgments, and that their influence is often quite substantial.
2.1. Should people make greater use of base rates than they presently do? The answer is far from clear. Part of the reason for the uncertainty is that the ecological validity of the base rate literature is low. Subjects are asked hypothetical questions about unfamiliar and unrealistic situations. They are presented with a single base rate that they are expected to treat as perfectly reliable. Failure to do so is recorded as an error. But such "errors" tell us little about whether people should make greater use of base rates in the kinds of situations they confront in their daily lives.
2.2. A second reason we can't say whether people should make greater use of base rates is that researchers have not given the normative component of the base rate fallacy the attention it deserves. Too often, base rate researchers make simplifying assumptions to invoke a normative standard, but at costs that deny the appropriateness or generality of their analyses. Consider the following assumptions:
2.2.1. "Subjects' prior beliefs are precisely represented by the single base rate statistic provided by the experimenter": This assumption allows researchers to use Bayes' theorem to determine whether their subjects rely on base rates (i.e., priors) to an appropriate extent. But why must a base rate be equated with a subject's prior belief? Subjects' prior beliefs may be informed by base rates, but the two need not be identical. One will almost always have additional information that does, and by all rights should, affect one's prior beliefs. Even under conditions where it is hard to imagine what such additional information might be, a study showed that 36% of subjects held priors that differed from supplied base rates (Beyth-Marom and Fischhoff, 1983, exp. 5).
2.2.2. "The context within which base rate problems are presented and solved does not and should not influence their solutions": Studies by Schwarz, Strack, Hilton, & Naderer (1991), however, have shown that subjects pay close attention to conversational norms when trying to solve base-rate/Bayesian problems. Schwarz et al. (1991) reported that people relied more on individuating information when the instructions defined the task as a psychological problem rather than as a statistical problem. Subjects apparently used the psychological context of the problem as a cue that the forthcoming individuating information would be relevant and diagnostic. It is interesting to note that subjects who attached additional weight to the individuating information in the psychology context might be regarded as good Bayesians: their predictions were based on all available information, including information derived from previous experience with similar contexts.
2.2.3. "Subjects understand and accept that the individuating information in base rate problems is a random sample from an unambiguous reference class": In the paradigmatic lawyer-engineer problem subjects are told that the personality descriptions of 5 people were chosen at random from a pool of lawyers and engineers. The descriptions were not randomly selected, however, and some subjects may have suspected as much once they were exposed to their stereotypical content. When Gigerenzer, Hell, & Blank (1988; experiment 1) ran a version of the lawyer-engineer problem in which subjects performed and observed the random sampling for themselves, the influence of base rates was much stronger than it was when random sampling was only verbally asserted. Similarly, reassurances about the representativeness of base rates in causal attribution studies promoted greater use of these data (Hansen & Donoghue, 1977; Wells & Harvey, 1977).
2.2.4. "Subjects' prior beliefs are and should be made independently of their assessments of the diagnostic strength of individuating information." Birnbaum (1983) pointed out that it is usually unrealistic to assume that likelihood ratios are independent of either base rates or prior probabilities. Birnbaum reviewed empirical investigations in the signal detection literature which show that, for human witnesses, the ratio of the hit rate to the false alarm rate (i.e., the likelihood ratio) depends on signal probabilities (i.e., base rates). That is, the accuracy of likelihood information derived from observer reports changes as the observer's knowledge of the base rates changes. Illustrating his point with frequent references to the taxi cab problem [see ENDNOTE #3], Birnbaum explained that a witness who is aware that there are many more Green cabs than Blue cabs is probably predisposed to see Green cabs in ambiguous situations. In light of this dependence, the actual probability that a cab in an accident was Blue given that a witness says so may be much closer to the median and modal responses given by untrained subjects (80%) than to the solution presented by base rate investigators (41%). Indeed, Birnbaum showed that one normatively appropriate response for witnesses who are aware of the 85% base rate for Green cabs and who wish to minimize errors is 82%.
2.5 In short, mechanical applications of Bayes' theorem will not help us measure a "base rate fallacy" when key assumptions of the model are either unchecked or grossly violated, or when features of the task and task environment that may appropriately influence subjects' responses are ignored.
3.1. It is frequently claimed that people ignore base rates and that the literature supports this conclusion. This claim is a considerable exaggeration at best. Although many studies have concluded that base rates are accorded relatively less weight than individuating, case-specific information in particular situations, few have shown that base rates are ignored.
3.2. Some of the confusion may be attributable to the unfortunate use of the term "ignore" by some investigators to describe data which suggested only that subjects attach relatively less weight to base rates than to descriptive, individuating information. In their classic lawyer-engineer experiment, Kahneman and Tversky (1973) reported a small but statistically significant main effect for the base rate (p < .01). Nevertheless, this study is widely cited as a convincing demonstration that base rates are "ignored" (e.g., Fagley, 1988; Nisbett & Borgida, 1975). But even if one regards small base rate effects as evidence that base rates are ignored, additional work on the lawyer-engineer problem suggests that the conclusions others have drawn from this study should be viewed with caution.
3.3. There have been numerous attempts to replicate the Kahneman and Tversky (1973) lawyer-engineer results. Table 1 presents the posterior probability estimates given by subjects in seven lawyer-engineer experiments that were conducted and reported in a comparable manner. These experiments were identical or nearly identical to Kahneman & Tversky's (1973) experiment and examined the impact of both diagnostic and nondiagnostic individuating information [see ENDNOTE #4]. If we set aside the difficulties of interpreting base rate study data for the time being, the top panel of Table 1 suggests that base rates uniformly influence subjects' judgments in the presence of diagnostic individuating information. Differences between high and low base rate groups ranged from 2% to 30%, with an average near 10%.
TABLE 1. BASE RATE USAGE IN THE LAWYER-ENGINEER PROBLEM
DIAGNOSTIC INDIVIDUATING INFORMATION
Study 1 2 3 4a 4b 4c 5 6a 6b 7a 7b 7c 7d 7e
High BR (.70) 55 79 70 04 100 100 57 35 80 62 61 36 81 38 posteriors (%)
Low BR (.30) 50 71 68 00 96 70 43 22 72 46 45 34 71 25 posteriors (%)
Difference 05 08 02 04 04 30 14 13 08 16 16 02 10 13
NONDIAGNOSTIC INDIVIDUATING INFORMATION
Study 1 2 3 4 5 6 7
High BR (.70) 50 54 59 73 70 65 61 posteriors (%)
Low BR (.30) 50 36 31 38 30 45 60 posteriors (%)
Difference 00 18 28 35 40 20 01
Note 1: Cell values are means in studies 1, 2, 4, 6, and 7; medians in study 5, and estimated medians (from Fig. 1 and Table 1 in original study) in study 3.
Note 2: The studies included are: 1. Kahneman & Tversky (1973); 2. Wells & Harvey (1978); 3. Ginossar & Trope (1980); 4. Fischhoff & Bar-Hillel (1984); 5. Hamilton (1984); 6. Ginossar & Trope (1987); and 7. Gigerenzer, Hell & Blank (1988).
Note 3: Fischhoff and Bar-Hillel (1984) used 10 diagnostic and 10 nondiagnostic "profiles." Due to limited space, only the results of the profiles that were identical to those used by Kahneman and Tversky (1973) are reproduced in Table 1. The results for the profiles used in Ginossar and Trope (1987) and in Gigerenzer et al. (1988) are listed separately. The results for the remaining studies are averaged across profiles as reported in the original studies.
3.4. The bottom panel of Table 1 does not present a consistent picture regarding whether or how base rates influenced judgments in the presence of nondiagnostic individuating information. In Kahneman and Tversky (1973) and Gigerenzer et al. (1988), judgments for high and low base rate groups were virtually indistinguishable, suggesting that base rates had little impact on final judgments. However, the remaining four studies revealed strong base rate effects. Indeed, a doctoral dissertation by Hamilton (1984) concluded that subjects in the nondiagnostic individuating information conditions based their judgments exclusively on the base rates provided.
3.5. This set of data from the lawyer-engineer problem does not provide strong support for the conclusion that base rates are universally ignored or even "largely ignored" (Kahneman and Tversky, 1973, p. 242). Data from other types of base rate studies contradict this conclusion as well. For example, it has been shown that base rates influence social judgments (Hewstone et al., 1988; Manis, Dovalina, Avis, & Cardoze, 1980; Rasinski, Crocker & Hastie, 1985), personality judgments (Iennarella & Kaplan, 1988), moral judgments (McGraw, 1987), auditing judgments (Hackenbrack, Nelson, & Amer, 1992), medical judgments (Weber, Bockenholt, Hilton and Wallace, in press), sports judgments (Gigerenzer et al., 1988), among others. These results are not surprising. A moment's reflection suggests that base rates are commonly used in daily life, even when other sources of information are available. Baseball managers routinely "play the percentages" by choosing left-handed batters to face right-handed pitchers and vice versa; police officers stop and detain suspected criminals, in part, on the basis of background characteristics; voters mistrust the political promises of even their most favored politicians. In each instance, base rate probabilities are considered and given substantial -- if not determinative -- weight.
4.1. So how did the base rate neglect myth emerge and sustain itself in the academic literature? Two explanations come to mind. The first relies on a Kuhnian account of scientific belief; the second is a heuristic explanation. Kuhn (1962/1970) is well-known for his views about the nature of scientific progress and paradigm shift. Kuhn stressed that a simple and powerful theory can withstand empirical challenge when the challenging data are not accompanied by a simple, general theory of their own. The base rate neglect thesis sprang from the heuristics and biases paradigm that dominated research on judgment and decision making in the 1970s and 1980s. This paradigm, developed largely by Daniel Kahneman and Amos Tversky, was extremely critical of people's intuitive judgments about probabilistic events, claiming that people make such judgments via simple error-prone heuristics. One well-known heuristic, representativeness, suggests that people's judgments about the probability of category membership depend on how similar the features of the target are to the essential features of the category (Kahneman & Tversky, 1972). Thus, judgments that Viki is an accountant depend upon how similar Viki's interests, background, talents, etc., are to those ordinarily associated with accountants.
4.2. As evidence in support of this heuristic mounted, base rate neglect became an easy sell. If people use the representativeness heuristic, and if base rates are typically less representative of a category's central features than individuating information, then it follows that people will ignore base rates. Empirical support for this phenomenon soon appeared (Kahneman & Tversky, 1973) and the "base rate fallacy" (Bar-Hillel, 1980) became a favorite instantiation of the heuristics and biases paradigm. When subsequent research failed to support the theory, the data were ignored and the theory persisted; the underlying principle was too attractive to abandon on account of data. The result was a simplification and misinterpretation of the body of literature by observers, researchers and reviewers alike.
4.3. Ironically, psychologists' misperception of the base rate literature may also be attributed to heuristic thinking. In order to make sense of a morass of complex and sometimes conflicting data, scientists may search for (or create) simple conclusions to represent a given body of research. The consequences of such simplification are potentially large. In a recent paper on the misrepresentation of behaviorism, Todd and Morris (1992) document this phenomenon and note that simple and general statements about a literature can become more authoritative than either the existing data or the claims made about the data by the original authors. The construction and acceptance of the Hawthorne effect similarly illustrates the point. Although a series of studies at the Hawthorne electrical plant that began in the late 1920s are widely cited in authoritative texts and reviews as demonstrating that workers' productivity increased regardless of the type of change made in their work environments, this interpretation is a gross simplification and distortion of the actual findings (for sociological analyses, see Adair, 1984; Gillespie, 1991; Jones, 1992).
5.1. If we are to increase our understanding of how people should and do use base rates, an ecologically sound program of research is required. The current research paradigm, which relies heavily on the use of controlled stimuli and artificial task environments, says too little about the use of the imperfectly reliable base rates that the natural ecology presents. In the end, it may matter little how much attention people pay to base rates in problems of the lawyer-engineer type. In the real world, those who appreciate the unreliability of certain types of base rates may make better decisions than those who do not. If we are serious about wanting to help decision makers make the best possible use of base rates, the next generation of studies must examine base rate usage in more realistic decision problems and environments.
5.2. Such a research program might begin by speculating about when judgmental accuracy is and is not likely to be impaired by a relative inattention to base rates. Inattention to base rates would seem likely to impede accuracy when the base rates conflict with other sources of information and are high in relative diagnosticity. In medical diagnoses, for example, inattention to reliable low base rates could lead to extensive overdiagnosis and excessive treatment. Consider that the general base rate for hypothyroidism is less than 1 in 1000 among young adult males (De Keyser & Van Herle, 1985), although the primary symptoms of this disease -- dermatological problems, depression, and fatigue -- are quite common. A diagnostician who disregards the base rate and relies solely on the individuating symptomatology and resultant likelihood ratios will surely overdiagnose this disease.
5.3. There may be a class of situations in which a relative inattention to base rates will not impede judgmental accuracy. Specifically, where there is a good deal of redundant information, a relative inattention to or underweighting of base rates may not matter. Most real world situations are rich with information. Furthermore, unlike most problems that appear in the existing base rate literature, base rates often do not run contrary to the bulk of other sources of information. For example, suppose you know that the summer evening jazz festivals in the city are usually very crowded; perhaps the base rate for a very large crowd is 80%. It is unlikely that subsequently received information relevant to an estimate of crowd size will contradict this base rate. Traffic in the immediate vicinity of the festival will probably be heavy, not light; more police officers will be assigned to this area rather than fewer; the lines at nearby restaurants will probably be longer, not shorter. In cases such as this, where much of the available information is consistent, failure to incorporate base rates is unlikely to hinder predictive accuracy.
6.1. Although decision makers are generally motivated to make accurate judgments, there are many situations in which accuracy is not the only basis for evaluating performance. For example, the American legal system is greatly concerned with a variety of fairness and due process issues, some of which interfere with judgmental accuracy (Nesson, 1985; Tribe, 1971). Indeed, certain types of highly probative evidence are routinely excluded in court (e.g., illegally obtained confessions) because their admission undermines other judicial values. It is often argued that admitting base rates violates the legal norm of individualized justice and should therefore be excluded as well [see ENDNOTE #5].
6.2. In other cases, cost of error considerations may persuade decision makers to make judgments that they believe are inaccurate. The American legal system requires that criminal guilt be proved "beyond a reasonable doubt." This subjective standard is designed to minimize erroneous convictions. A cost of this systemic value is that juries will often return "not guilty" verdicts in cases where they believe the defendant is guilty (albeit not beyond a reasonable doubt).
6.3. The point is that judgments about the performance of real world decision makers will often need to take factors other than judgmental accuracy into account. Moreover, even where accuracy is the primary goal, prescriptive recommendations might also take into account such costs as the time, mental effort, and money that may be required to improve accuracy. This means that we will need to relax our notion of what constitutes an appropriate response in base rate problems given to us in the natural ecology (cf. Bar-Hillel, 1983). Many problems will have multiple solutions. Indeed, whenever the assumptions, goals and values of decision makers vary, people exposed to identical information may arrive at different solutions, none of which are necessarily erroneous.
6.4. An empirical research program that embraces person- and situation-specific performance criteria can provide richer insights and more useful prescriptive recommendations than the existing research program. In time, we may determine when people ought to pay more or less attention to the imperfect base rates of the natural ecology in order to best achieve their goals. At this time, prescriptive guidelines should be developed and marketed to the relevant decision making communities.
#1. A base rate may be defined as the relative frequency with which an event occurs or an attribute is present in a population (Ginossar & Trope, 1987; Hinsz, Tindale, Nagao, Davis, & Robertson, 1988; Lanning, 1987). Hence the base rate for six-figure annual incomes might be 95% among Major League Baseball players in the United States, but less than 1% among professional guitar players.
#2. Bayes' theorem follows directly from the multiplicative rule of probability which holds that the joint probability of two events, H and E, equals the product of the conditional probability of one of the events given the second event, plus the probability of the second event. In mathematical notation: P(H&E) = P(H|E)P(E) P(H&E) = P(E|H)P(H) Therefore: P(H|E) = P(E|H)P(H) / P(E) where P(E) = P(E|H)P(H)+P(E|-H)P(-H) for binary hypotheses.
Odds form: P(H|E) / P(-H|E) = [P(E|H)P(H) / P(E)] / [P(E|-H)P(-H) / P(E)] = P(E|H)P(H) / P(E|-H)P(-H) = [P(H) / P(-H)] X [P(E|H) / P(E|-H)]
The letters H and E stand for Hypothesis and Evidence respectively. P(H) and P(-H) refer to the probabilities of the truth and falsity of a hypothesis H prior to the collection of additional evidence. P(H) and P(-H) are "prior probabilities" and their ratio is the "prior odds." P(E|H) and P(E|-H) represent the information value of the evidence if the hypothesis is true and false respectively; their ratio is the "likelihood ratio." P(H|E) and P(-H|E) are the probability that the hypothesis is true and false in light of the evidence; their ratio is the "posterior odds," which represents the combination of the prior odds and likelihood ratio.
#3. The taxi cab problem, as described by Tversky and Kahneman (1980, p. 62) is as follows: "A cab was involved in a hit-and-run accident at night. Two cab companies, the Green and the Blue, operate in the city. You are given the following data: (i) 85% of the cabs in the city are Green and 15% are Blue. (ii) A witness identified the cab as a Blue cab. The court tested his ability to identify cabs under the appropriate visibility conditions. When presented with a sample of cabs (half of which were Blue and half of which were Green) the witness made correct identifications in 80% of the cases and erred in 20% of the cases. Question: What is the probability that the cab involved in the accident was Blue rather than Green?"
#4. Other lawyer-engineer studies not described in Table 1 for reasons of noncomparability include Borgida and Nisbett (1977), Davidson and Hirtle (1990), Gabrenya and Arkin (1979), and Schwarz, Strack, Hilton, and Naderer (1991).
#5. Debate about what role base rates and other types of probabilistic evidence should play in court has raged for years. See Brilmayer & Kornhauser, 1978; Cohen, 1981; Finkelstein & Fairley, 1970; Kaye, 1979, 1981; Koehler, 1992, 1993; Koehler & Shaviro, 1990; Nesson, 1985; Saks & Kidd, 1980-1; Shaviro, 1989; Tribe, 1971. Related papers can also be found in "Probability," 1986; "Debate," 1991; and "Decision," 1991.
REFERENCES:
Adair, J. G. (1984) The Hawthorne Effect: A reconsideration of the methodological artifact. Journal of Applied Psychology 69:334-345.
Bar-Hillel, M. (1980) The base-rate fallacy in probability judgments. Acta Psychologica 44:211-233.
Bar-Hillel, M. (1983) The base rate fallacy controversy. In: Decision making under uncertainty (pp. 39-61), ed. R. W. Scholz. Elsevier Science.
Bergman, P. & Moore, A. (1991) Mistrial by likelihood ratio: Bayesian analysis meets the F-word. Cardozo Law Review 13:589-619.
Beyth-Marom, R. & Fischhoff (1983) Diagnosticity and pseudodiagnosticity. Journal of Personality and Social Psychology 45:1185-1195.
Birnbaum, M. H. (1983) Base rates in Bayesian inference: Signal detection analysis of the cab problem. American Journal of Psychology 96:85-93.
Borgida, E. & Brekke, N. (1981) The base rate fallacy in attribution and prediction. In: New directions in attribution research, (Vol. 3, pp. 63-95), eds. J. H. Harvey, W. J. Ickes, & R. F. Kidd. Erlbaum.
Borgida, E. & Nisbett, R. (1977) The differential impact of abstract versus concrete information on decisions. Journal of Applied Social Psychology 7:258-271.
Brilmayer, L. & Kornhauser, L. (1978) Review: Quantitative methods and legal decisions. University of Chicago Law Review 46:116-153.
Christensen-Szalanski, J. J. J. & Bushyhead, J. B. (1981) Physicians' use of probabilistic information in a real clinical setting. Journal of Experimental Psychology: Human Perception and Performance 7:928-935.
Cohen, L. J. (1981) Subjective probability and the paradox of the gatecrasher. Arizona State Law Journal 1981:627-633.
Davidson, D. & Hirtle, S. C. (1990) Effects of nondiscrepant and discrepant information on the use of base rates. American Journal of Psychology 103:343-357.
De Keyser, L. & Van Herle, A. (1985) Hypothyroidism. Current diagnosis (7th ed.) (pp. 840-843), eds. R. B. Conn. W. B. Saunders Co. Debate on statistics and evidentiary theory (1991) Tulane Law Review 65, 66. Decision and inference in litigation (1991) Cardozo Law Review 13.
Evans, J. St. B. T. & Bradshaw, H. (1986) Estimating sample-size requirements in research design: A study of intuitive statistical judgment. Current Psychological Research and Reviews 5:10-19.
Fagley, N. S. (1988) Judgmental heuristics: Implications for the decision making of school psychologists. School Psychology Review 17:311-321.
Finkelstein, M. & Fairly, W. (1970) A Bayesian approach to identification evidence. Harvard Law Review 83:489-517.
Fischhoff B. & Bar-Hillel, M. (1984) Diagnosticity and the base- rate effect. Memory and Cognition 12:402-410.
Fischhoff, B., & Beyth-Marom, R. (1983) Hypothesis evaluation from a Bayesian perspective. Psychological Review 90:239-260.
Gabrenya, W. K. & Arkin, R. M. (1979) Motivation, heuristics, and the psychology of prediction. Motivation and Emotion 3:1-17.
Gigerenzer, G., Hell, W. Blank, H. (1988) Presentation and content: The use of base rates as a continuous variable. Journal of Experimental Psychology: Human Perception and Performance 14:513-525.
Gillespie, R. (1991) Manufacturing knowledge: A history of the Hawthorne experiments. Cambridge University Press.
Ginossar, Z. & Trope, Y. (1980) The effects of base rates and individuating information on judgments about another person. Journal of Experimental Social Psychology 16:228- 242.
Ginossar, Z. & Trope, Y. (1987) Problem solving in judgment under uncertainty. Journal of Personality and Social Psychology 52:464-473.
Hackenbrack, K., Nelson, M. & Amer, T. (1992) The influence of event base rate on the interpretation of probability phrases in accounting settings. Unpublished manuscript.
Hamilton, M. M. (1984) An examination of processing factors affecting the availability of consumer testimonial information in memory. Unpublished doctoral dissertation, Johns Hopkins University, Baltimore, MD.
Hansen, R. D. & Donoghue, J. M. (1977) The power of consensus: Information derived from one's own and other's behavior. Journal of Personality and Social Psychology, 35:294-302.
Hewstone, M., Benn, W., & Wilson, A. (1988) Bias in the use of base rates: Racial prejudice in decision making. European Journal of Psychology 18:161-176.
Hinsz, V. B., Tindale, R. S., Nagao, D. H., Davis, J. H., & Robertson, B. A. (1988) The influence of the accuracy of individuating information on the use of base rate information in probability judgment. Journal of Experimental Social Psychology 24: 127-145.
Iennarella, R. S. & Kaplan, M. F. (1988) Acceptance of personality descriptions by subjects: individual and base rate truthfulness. Journal of Social and Clinical Psychology 6:388-397.
Jonakait, R. N. (1983) When blood is their argument: Probabilities in criminal cases, genetic markers, and, once again, Bayes Theorem. University of Illinois Law Review 1983:369-421.
Jones, S. R. G. (1992). Was there a Hawthorne effect? American Journal of Sociology 98:451-468.
Kahneman, D. & Tversky, A. (1972) Subjective probability: A judgment of representativeness. Cognitive Psychology 3:430-453.
Kahneman, D. & Tversky, A. (1973) On the psychology of prediction. Psychological Review 80:237-251.
Kaye, D. H. (1979) The paradox of the gatecrasher and other stories. Arizona State University Law Journal 101:101- 109.
Kaye, D. H. (1981) Paradoxes, gedanken experiments and the burden of proof: A response to Dr. Cohen's reply. Arizona State Law Journal 1981:635- 645.
Koehler, J. J. (1993) The normative status of base rates at trial. Individual and group decision making, ed. N. J. Castellan. Erlbaum.
Koehler, J. J. (1992) Probabilities in the courtroom: An evaluation of the objections and policies. Handbook of psychology and law, eds. D. K. Kagehiro, & W. S. Laufer. Springer-Verlag.
Koehler, J. J. & Shaviro, D. N. (1990) Veridical verdicts: Increasing verdict accuracy through the use of overtly probabilistic evidence and methods. Cornell Law Review, 75:247-279.
Kuhn, T. (1962/1970) The structure of scientific revolutions (2nd ed.). University of Chicago Press.
Lanning, K. (1987) Some reasons for distinguishing between 'non- normative response' and 'irrational decision.' The Journal of Psychology 12:109-117.
Manis, M., Dovalina, I., Avis, N. E., & Cardoze, S. (1980) Base rates can affect individual predictions. Journal of Personality and Social Psychology 38:231-248.
McGraw, K. M. (1987) Outcome valence and base rates: The effects on moral judgments Social Cognition 5:58-75.
Meehl, P. E. & Rosen, A. (1955) Antecedent probability and the efficiency of psychometric signs, patterns, or cutting scores. Psychological Bulletin 52:194-216.
Nesson, C. (1985) The evidence or the event? On judicial proof and the acceptability of verdicts. Harvard Law Review 98:1357-1392.
Nisbett, R. E. & Borgida, E. (1975) Attribution and the psychology of prediction. Journal of Personality and Social Psychology 32:932-943.
Pollard, P. & Evans, J. St. B. T. (1983) The role of 'representativeness' in statistical inference: A critical appraisal. In Thinking and reasoning: Psychological approaches (pp. 107-134), ed. J. St. B. T. Evans. Routledge and Kegan Paul. Probability and inference in the law of evidence [Symposium] (1986) Boston University Law Review 66.
Rasinski, K. A., Crocker, J. & Hastie, R. (1985) Another look at sex stereotypes and social judgments: An analysis of the social perceiver's use of subjective probabilities. Journal of Personality and Social Psychology 49:317-326.
Saks, M. J. & Kidd, R. F. (1980-1) Human information processing and adjudication: Trial by heuristics. Law and Society Review 15:123-160.
Schwarz, N., Strack, F., Hilton, D., & Naderer, G. (1991) Base rates, representativeness, and the logic of conversation: The contextual relevance of 'irrelevant' information. Social Cognition 9:67-83.
Shaviro, D. (1989) Statistical-probability evidence and the appearance of justice. Harvard Law Review 103:530-554.
Slovic, P. & Lichtenstein, S. (1971) Comparison of Bayesian and regression approaches to the study of information processing in judgment. Organizational Behavior and Human Performance 6:649-743.
Todd, J. T. & Morris, E. K. (1992) Case histories in the great power of steady misrepresentation. American Psychologist 47:1441-1453.
Tribe, L. H. (1971) Trial by mathematics: Precision and ritual in the legal process. Harvard Law Review 84:1329-1393.
Tversky, A. & Kahneman, D. (1980) Causal schemas in judgments under uncertainty. Progress in social psychology (pp. 49-72), ed. M. Fishbein. Erlbaum.
von Winterfeldt, D. & Edwards, W. (1986) Decision analysis and behavioral research. Cambridge University Press.
Weber, E. U., Bockenholt, U., Hilton, D. J. & Wallace, B. (in press) Determinants of diagnostic hypothesis generation: Effects of information, base rates and experience. Journal of Experimental Psychology: Learning, Memory and Cognition.
Wells, G. L. & Harvey, J. H. (1977) Do people use consensus information in making causal attributions? Personality and Social Psychology 35:279-293.
Wells, G. L. & Harvey, J. H. (1978) Naive attributors' attributions and predictions: What is informative and when is an effect an effect? Journal of Personality and Social Psychology 36:483-490.