Most tasks used to demonstrate the base rate fallacy are ambiguous about a crucial assumption required for properly applying Bayes' rule. This results in a verbal confusion between likelihood and posterior probability. Experiments that disambiguate the texts presented to subjects have a large effect on the use of the base rate. Some comments on the ecological validity of tasks are offered.
1. The question of the ecological validity of tasks in unfamiliar and unrealistic situations seems to me quite secondary to the problem of the effective communication of the requisite assumptions for Bayesian reasoning. It has been shown (indirectly) that manipulating procedural variables can convey the nature of the relationship between different pieces of information (Christensen-Szalanski and Beach, 1982; Ginossar and Trope, 1987) as well as the random nature of the information itself (Gigerenzer, Hell and Blank, 1988). Both of these are crucial factors in making probabilistic judgments. The use or neglect of the base rate are greatly influenced by the communication (and comprehension) of such factors.
2. According to the "confusion" hypothesis (Braine, Connel, Freitag and O'Brien, 1990; Cohen, 1981; Eddy, 1982; Dawes, 1986; Hamm and Miller, 1988), subjects confuse the conditional probability of the accuracy of a specific item of information (i.e., the probability of observing a particular datum given that a hypothesis is true: P(D|H)) with the inverse conditional probability P(H|D) (i.e., the probability that a hypothesis is true given that a particular datum has been observed). This confusion is attributed (by these authors) to people's (inherent) inability to understand the difference between these two conditional probabilities. However, subjects are capable of distinguishing these two types of probability in many other contexts (Bar-Hillel, 1990; Thuring and Jungermann, 1990). One could therefore argue that this confusion does not depend so much on a natural tendency to err, but rather on an ambiguous transmission of information produced by the structure of the problem text (see Lindley, 1985). If this is a specific difficulty, a change in the text which did not alter the nature of conditional probability should not influence the final evaluation of the subjects.
3. Demonstrations of the role of text structure in problem solving (Mosconi, 1990), and analyses of the pragmatic rules of natural language (Grice, 1975; Sperber and Wilson, 1986) of the texts used in one influential research paradigm (the textbook-problem paradigm, Bar-Hillel, 1983), suggest that the so-called "base rate fallacy" may arise from the fact that one of the relationships between the data is not sufficiently clear. Specifically, subjects may not perceive the two pieces of information as conditionally independent -- a crucial assumption for proper Bayesian analysis (Birnbaum, 1983). If subjects interpreted the diagnostic information [P(D|H)] as the posterior probability [P(H|D)], then they would be entitled to assume that the base rate had already been taken into account.
4. It has been shown (Macchi, 1994) that the diagnostic (or specific) information is mistaken for a posterior probability as a consequence of: (i) how one formulates the question posed to subjects (when it refers only to the specific information) and (ii) its specific information content (when it is interpretable as if it were already the result of its combination with the prior probability). Two problems were used in the experiments: the Suicide Problem (Bar-Hillel, 1980; Tversky and Kahneman, 1980) and the Cab Problem (Kahneman and Tversky, 1973). The base rate fallacy decreased or disappeared when these textual elements were modified -- regardless of the (unchanged) specificity and causality of the information. By the same token, it was possible to generate the "bias" by introducing those textual elements into problems referring to "causal" information which do not usually demonstrate the fallacy (Tversky and Kahneman, 1980).
5. On the basis of a pragmatic analysis of the text-problems used in the literature, differences between problems which induced the bias and those which did not have been identified. These differences were not linked to the usual heuristic account but had to do with the pragmatic structure of the texts. Although this explanation is not general (other types of problems, e.g., those concerning the "social judgment paradigm," Bar-Hillel, 1983; Hilton and Fein, 1989; Tversky and Kahneman, 1980, deserve separate consideration with respect to pragmatic factors ), it does explain the paradigm considered here.
6. We now turn to the problem posed by Koehler (1993) concerning the adequacy of the paradigmatic tasks used to study the use of base rate. A question is prompted by our pragmatic analysis: which formulation is more natural (spontaneously used in the "real world"), the one used in the original tasks (i.e., the original "Cab" text-problem) or one that takes into consideration the pragmatic aspects of how one communicates the problem?
7. It would be hard to say which of these texts is more similar to problems encountered in real life. Traditional studies do not give clear indications on this issue. Nonetheless, we speculate that even if the first type of formulation (i.e., the original "Cab" text-problem) was more "natural," in the "real world" the context -- the way the information is experienced -- will usually disambiguate its meaning. In most cases, the independence of the information would still be clearly transmitted (see Christensen-Szalanski and Beach's 1982 study, where there was progressive acquisition of the data in a medical context and proper use of base rates). This suggests that using a task in an experimental setting requires us to take into account the loss of information arising from the abstractness of the question relative to the natural context in which it was produced: care should be taken to compensate for the lost information through clarity of phrasing.
8. To return to the question raised by Koehler about the ecological validity of research tasks, it does not seem to me that the familiarity of the content (Gigerenzer, Hell and Blank, 1988) or the generic realism of the situations are crucial, given that with the traditional problems (often criticized on these grounds), base rates can be brought into use by modifying pragmatic variables -- leaving completely unaltered the putative role of the heuristics traditionally considered responsible for the bias.
9. It seems to me that ecological validity is established by demonstrating effective communication of the assumptions necessary to justify Bayesian reasoning (e.g., the independence of the data). This is, in any case, an ineluctable premise for the study of probabilistic reasoning.
Bar-Hillel, M. (1980). The Base Rate Fallacy in Probability Judgements. Acta Psychologica, 44, 211-233.
Bar-Hillel, M. (1983). The Base Rate Fallacy Controversy. In R.W. Scholtz (ed.), Decision Making Under Uncertainty. Elsevier Science Publishers, North Holland.
Bar-Hillel, M. (1990). Back to Base Rates. In R.M. Hogarth (ed.) Insights in Decision Making. Chicago, IL: University of Chicago Press.
Birnbaum, M.H. (1983). Base Rates in Bayesian Inference: Signal Detection Analysis of the Cab Problem. American Journal of Psychology, 96, 85-94.
Braine, M.D.S., Connel, J., Freitag, J. and O'Brien, D.P. (1990). Is the Base-Rate Fallacy an Instance of Asserting the Consequent? In K.L. Gilhooly, M.T.G. Leane, R.H. Logie and G. Erdos (eds.), Lines of Thinking, vol. 1, John Wiley & Sons Ltd.
Christensen-Szalanski, J.J.J. and Beach, L.R. (1982). Experience and the Base-Rate Fallacy. Organizational Behavior and Human Performance, 29, 270-278.
Cohen, L.J. (1981). Can Human Irrationality Be Experimentally Demonstrated? Behavioral and Brain Sciences, 4, 317-331.
Dawes, R.M. (1986). Representative Thinking in Clinical Judgment. Clinical Psychology Review, 6, 425-441.
Eddy, D.M. (1982). Probabilistic Reasoning in Clinical Medicine: Problems and Opportunities. In D. Kahneman, P. Slovic and A. Tversky (eds.), Judgment Under Uncertainty: Heuristics and Biases. Cambridge University Press.
Gigerenzer, G., Hell, W. and Blank, H. (1988). Presentation and Content: The Use of Base Rates as a Continuous Variable. Journal of Experimental Psychology: Human Perceptions and Performance, 14(3), 513-525.
Ginossar, Z. and Trope, Y. (1987). Problem Solving in Judgment Under Uncertainty, Journal of Personality and Social Psychology, 52, 464-474.
Grice, H.P. (1975). Logic and Conversation. In P. Cole and J.L. Morgan (eds.), Syntax and Semantics: vol. 3. Speech Acts. New York: Academic Press.
Hamm, R.M. and Miller, M.A. (1988). Interpretation of Condition Probabilities in Probabilistic Inference Word Problems. (Publication No. 88-15). Boulder, CO: Institute of Cognitive Science, University of Colorado.
Hilton, J.L. and Fein, S. (1989). The Role of Diagnosticity in Stereotype-Based Judgments. Journal of Personality and Social Psychology, 57, 201-211.
Kahneman, D. and Tversky, A. (1973). On the Psychology of Prediction. Psychological Review, 80, 237-251.
Koehler, J.J. (1993). The Base Rate Fallacy Myth. PSYCOLOQUY, 4(49) base-rate.1.koehler.
Lindley, D.V. (1985). Making Decisions. John Wiley and Sons Ltd.
Macchi, L. (1994, in press). Pragmatic Aspects of the Base Rate Fallacy. The Quarterly Journal of Experimental Psychology.
Mosconi, G. (1990). Discorso e Pensiero. Il Mulino, Bologna.
Sperber, D. and Wilson, D. (1986). Relevance: Communication and Cognition, Basil Blakwell, Oxford.
Thuring, M. and Jungermann, H. (1990). The Conjunction Fallacy: Causality Versus Event Probability. Journal of Behavioural Decision Making, 3, 61-74.
Tversky, A. and Kahneman, D. (1980). Causal Schemata in Judgments Under Uncertainty. In M. Fishbein (ed.), Progress in Social Psychology, vol. 1, Hillsdale, NJ, Erlbaum, 49-72.