Craig R. M. McKenzie (1994) Base Rates Versus Prior Beliefs in Bayesian Inference. Psycoloquy: 5(01) Base Rate (6)

Commentary on Koehler on Base-Rate

Department of Psychology

University of Chicago

Center for Decision Research

1101 E. 58th Street

Chicago, IL 60637

fac_mckenzie@gsbvax.uchicago.edu

The distinction between prior beliefs and base rates is often blurred in the literature on judgment and decision making and can account for some apparently contradictory conclusions regarding behavior in Bayesian tasks.

1. Bayes' theorem can be used in at least two distinct ways to study behavior. One way is to insert objective values into the theorem in order to see whether people's responses correspond to the environment, that is, maximize objective accuracy. A second way is to insert subjective values into the theorem to see whether people's responses are consistent, that is, maximize subjective accuracy. In terms of the prior probabilities in Bayes' theorem, inserting base rates (i.e., the relative frequency of an event) is appropriate in the first case, and inserting people's prior beliefs (i.e., the subjective probability of an event prior to receiving new information) is appropriate in the second.

2. However, as Koehler (1993) mentions, base rates are often equated with the prior probabilities in Bayes' theorem--despite the fact that Bayes' theorem is usually associated with subjective probability (see also Gigerenzer, in press). This commentary discusses the implications of Koehler's distinction between base rates and prior beliefs for interpreting findings in the judgment and decision making literature.

3. In their seminal paper, Kahneman and Tversky (1973) concluded that subjects ignore or underweight prior probabilities when presented with likelihood (or case-specific) information. But what did they mean by "prior probabilities?" Although they fail to distinguish between base rates and prior beliefs, I believe that they were implicitly interested in subjects' prior beliefs. I claim this on the basis of their experimental methodologies.

4. One method used by Kahneman and Tversky (1973) was to ask one group of subjects to estimate the base rates of the events in question and to ask another group to give probability estimates based on only likelihood information. If Kahneman and Tversky were interested only in whether or not people use base rates when provided with likelihood information, why did they ask a separate group of subjects for estimates of base rates? Why not simply use the objective relative frequencies as the prior probabilities?

5. A second method Kahneman and Tversky used was to provide subjects with only base-rate information for predicting some events, and with both base-rate and likelihood information for predicting other events. If Kahneman and Tversky were interested only in whether or not people use base rates when provided with likelihood information, why did they check to see whether subjects used the base rate for the base-rate-only questions?

6. The answer to these questions is that Kahneman and Tversky (1973) were implicitly interested in studying prior beliefs, not base rates per se. It was important to Kahneman and Tversky's point that, for example, subjects used the base rates when presented with the base-rate-only questions. Base rates were meant only as proxies for subjects' prior beliefs.

7. Does this mean, then, that Kahneman and Tversky (1973) provide evidence that subjects ignore their prior beliefs? I don't think so. I believe that the assumptions behind both methods for discovering whether people ignore or underweight their prior beliefs are wrong (or at least not always right). The first method equates base rates (albeit subjective) with prior beliefs. It is not clear that the likelihood group ever held the estimated base rates as prior beliefs. For that matter, it's not even clear that the base-rate group's estimates would be equivalent to their prior beliefs. For example, though I may believe that there are 80% Blue cabs in the city, my prior belief that a cab involved in an accident was Blue may be higher or lower than 80% if I think that Blue cab drivers are more or less reckless than others. Even in a within-subjects experiment, it cannot be assumed that estimated base rates correspond to prior beliefs.

8. The second method assumes that the task involving both base-rate and likelihood information is identical to the base-rate-only task (in which subjects tend to use the base rate), with the added step of processing the likelihood information. However, there is no evidence that I know of showing that subjects, who are simultaneously shown both likelihood and base-rate information and then appear to ignore the base rate, ever instantiate the base rate as a prior belief. Under such conditions, subjects often consider the base rate irrelevant (Lyon & Slovic, 1976). Here, then, base rates are clearly not prior beliefs.

9. What was needed was a two-stage task that first provided subjects with base-rate information and asked for a prior probability, then provided them with likelihood information and asked for a posterior probability. To the best of my knowledge, it was 10 years after the first studies showing base-rate neglect before subjects were presented with such a task (Beyth-Marom & Fischhoff, 1983). As Koehler (1993) states, more than a third of the subjects did not report the base rate after the first stage. Base rates are not equivalent to prior beliefs.

10. Interestingly, studies subsequent to Kahneman and Tversky (1973) switched from viewing base rates as a means of examining prior beliefs to a concern for base rates per se (e.g., Bar-Hillel, 1980; Lyon & Slovic, 1976; Tversky & Kahneman, 1982). However, there was never any mention of the shift in focus; it was as though the later studies were simply extensions of the first. This, I think, is further evidence of the failure to distinguish between base rates and prior beliefs. Base rates became synonymous with the prior probabilities in Bayes' theorem, and ignoring or underweighting them was tantamount to being "non- Bayesian" (Bar-Hillel, 1980, p. 225).

11. So, what happens when people's prior beliefs rather than base rates are equated with the prior probabilities in Bayes' theorem? The results are strikingly different. Consider the dominant paradigm for studying Bayesian thinking before the heuristics-and-biases paradigm, namely, the book-bag-and-poker-chip experiments of the 1960s. These tasks were often belief-updating tasks; subjects updated probability estimates for the same hypothesis multiple times as new information was received. Here, the posterior probability given for response n becomes the prior probability for response n+1. After the first response, then, the prior probability is given by the subject to the experimenter rather than by the experimenter to the subject; it is therefore a prior belief. The typical finding (Edwards, 1968) was that people were "conservative," that is, they did not move far enough away from their prior beliefs--exactly the opposite of what is found from studies supplying subjects with base rates.

12. Indeed, in one form of Hogarth and Einhorn's (1992) belief-updating model, response n is the reference point for encoding incoming evidence for response n+1 (see also Anderson, 1981). That is, new evidence is evaluated in terms of whether it is stronger or weaker than the current position (i.e., prior belief), and confidence is shifted in that direction [see ENDNOTE #1]. Far from being ignored, prior beliefs play a focal role in such models. The distinction between base rates and prior beliefs is an important one.

13. The distinction between base rates and prior beliefs has often been blurred in the judgment and decision-making literature, leading to contradictory conclusions regarding Bayesian behavior. Studies equating base rates with the prior probabilities in Bayes' theorem often conclude that subjects ignore or underweight the priors, while studies equating prior beliefs with the prior probabilities often conclude that people weight priors too heavily. Both interpretations of the prior probabilities in Bayes' theorem are interesting and important for studying behavior. But performance in one task may be completely independent of performance in the other, and neither can claim to be The Bayesian Task.

#1. An interesting implication of such "averaging" of the prior and likelihood information is that, when presented with strong support for a hypothesis followed by weak support for the same hypothesis, confidence decreases, though Bayes' theorem prescribes an increase (Hogarth & Einhorn, 1992). Nonetheless, the averaging strategy always supports the correct hypothesis and, when examined over a variety of conditions, it leads to responses that are virtually perfectly correlated with the normative Bayesian response (McKenzie, in press).

Anderson, N.H. (1981). Foundations of Information Integration Theory. New York: Academic Press.

Bar-Hillel, M. (1980). The Base-rate Fallacy in Probability Judgments. Acta Psychologica, 44, 211-233.

Beyth-Marom, R., & Fischhoff, B. (1983). Diagnosticity and Pseudo-diagnosticity. Journal of Personality and Social Psychology, 45, 1185-1195.

Edwards, W. (1968). Conservatism in Human Information Processing. In B. Kleinmuntz (Ed.), Formal Representation of Human Judgment (pp. 17-52). New York: Wiley.

Gigerenzer, G. (in press). Why the Distinction Between Single-event Probabilities and Frequencies is Important for Psychology (and vice versa). In G. Wright & P. Ayton (Eds.), Subjective Probability. New York: Wiley.

Hogarth, R.M., & Einhorn, H.J. (1992). Order Effects in Belief- updating: The Belief-adjustment Model. Cognitive Psychology, 24, 1-55.

Kahneman, D., & Tversky, A. (1973). On the Psychology of Prediction. Psychological Review, 80, 237-251.

Koehler. J.J. (1993). The Base Rate Fallacy Myth. PSYCOLOQUY 4(49) base-rate.1.koehler.

Lyon, D., & Slovic, P. (1976). Dominance of Accuracy Information and Neglect of Base Rates in Probability Estimation. Acta Psychologica, 40, 287-298.

McKenzie, C.R.M. (in press). The Accuracy of Intuitive Judgment Strategies: Covariation Assessment and Bayesian Inference. Cognitive Psychology.

Tversky, A., & Kahneman, D. (1982). Evidential Impact of Base Rates. In D. Kahneman, P. Slovic, & A. Tversky (Eds.), Judgment under Uncertainty: Heuristics and Biases (pp. 153-160). Cambridge: Cambridge University Press.