15.1.1 The Mathematical Long-Run and Short-Run Functions

We solve the Standard Search Problem (Problem 1) for the general case of n equally likely drawers and prior probability L₀ that the letter is in the desk. The solution applies as well to the isomorphic Wait Problem (Problem 2). If the letter is in the desk, the conditional probability of not finding it when searching the first i drawers is $\frac{n-i}{n}$; if the letter is out of the desk, not finding it in the first i drawers is a certainty. Let’s denote the respective long-term and short-term posterior probabilities we wish to find by L_i = P (letter is in desk | letter was not in first i drawers), S_i = P (letter is in next drawer | letter was not in first i drawers). Clearly, S₀ = L₀⁄n, and $S_i=\frac{L_i}{n-i}$. By Bayes’ rule:

$$L_i=\frac{\frac{n-i}{n}\cdot L_0}{\frac{n-i}{n}\cdot L_0+(1-L_0)}$$

A few algebraic manipulations yield:

• (15.1)

  $$L_i=\frac{(n-i)\cdot L_0}{n-i\cdot L_0}i=0,1,2,...,n-1,n$$

• (15.2)

  $$S_i=\frac{L_0}{n-i\cdot L_0}i=0,1,2,...,n-1$$

Formulas (15.1) and (15.2) describe the hope functions for the long run (L_i) and the short run (S_i), given i initial failures.

Table 15.1 presents the specific forms which L_i and S_i assume in the case of Problems 1 and 2, along with the answers to the questions posed in the Problems. The numbers in Table 15.1, as well as formulas (15.1) and (15.2), indicate the long-term hope function, L_i, decreases as i grows, whereas the short-term hope function, S_i, increases with i, until L_n−1 = S_n−1.

[pg359]

15.1.2 Further Explorations

A number of issues surface as we extend our formal analysis to the examples cited earlier. Suppose L₀ = 1, as in the case of Sherlock’s absolute confidence that the pearl is hidden in one of the busts (Example 1). If indeed there is no doubt whatsoever about the existence of the target object in one of the available locations, no initial sequence of failures, long as it may be, will shatter that (long-term) certainty. L_i will equal 1 for all values of i. The short-term probability of success in the next unit (location or time slot) will equal the inverse of the number of remaining units and will thus rise to 1 when only one unit remains (ie. for i = n−1). The results for the case of initial certainty may also be obtained from formulas (15.1) and (15.2) by substituting 1 for L₀. These formulas are, in fact, valid for the entire range of possible values of L₀, including the end points 1 and 0.

15.2.1 Directional and Numerical Assessments

We first analyzed the data ordinally. Ignoring the exact values in the numerical versions, we sorted responses into 3 main types: strictly increasing, strictly decreasing, or a constant function. A 4^th category (other) included functions which changed directions or were weakly monotonic. (The undergraduate and senior high-school students’ responses were pooled since the pattern of responses of the 2 groups were very similar.) Table 15.3 shows the 2-dimensional distribution, pooled across story types, of the 61 subjects according to the kind of L and S functions which they produced.

The results in Table 15.3 show that a majority of the subjects (35) intuitively sensed the decline of the L function. The modal group of subjects (27) produced an increasing S function. Yet, only about one fifth of the subjects (12) generated the correct combination of a decreasing L and an increasing S function. It is noteworthy that in a pilot study with 42 undergraduate law students at the Hebrew University of Jerusalem, about one fifth (8) produced the correct combination. The pilot study used different but isomorphic stories [pg364] (searching an escaped prisoner in successive houses, and waiting for a forgetful professor to come to an appointment).

15.2.2 Principal Assumptions Underlying Solution Strategies

Solution strategies are suggested by the pattern of subjects’ numerical responses and the explanations they provided. In examining these, a few heuristics appear to us to be guiding a substantial number of responses. In particular, in many cases assumptions of constancy underlie the choice of the 3 answers.

Suppose one assumes that the given L₀ of 0.80 in Problem 1 (desk) stays unchanged despite failing to find the letter in the first i drawers. That assumption, which we label constant L, entails an identical response of 0.80 to all questions of DL and an increasing triplet of answers to DS—(1) 0.114 (ie. 0.80/7), (2) 0.20, (3) 0.80 (see the correct set of answers in Table 15.1). One may, however, assume that the probability of success per drawer (unit) stays unchanged. We label that assumption constant S. It entails an identical response of 0.10 (ie. S₀) to all the questions in DS and a decreasing triplet of answers to DL—(1) 0.70, (2) 0.40, (3) 0.10 (cf. Table 15.1). The corresponding predictions of responses to BL and BS under the 2 constancy assumptions can be easily obtained.

The responses of 12 subjects to the 2 forms were compatible with the constant L assumption. 6 subjects assumed constant S across both forms, [pg365] and another 13 assumed constant L in answering one form and constant S in the other. Among the remaining 30 subjects, 19 assumed constancy in only one of the forms (6 constant L, and 13 constant S). Overall, of the 122 forms answered by 61 subjects, 81 (ie. 66.4%) were based on constancy assumptions: 43 constant L, and 38 constant S.

The heuristic of adhering to one constant parameter of the setup (whether L₀ or S₀) reduces the complexity of the hope problems. But it may also reflect subjects’ conception of probability as an unchanging propensity of the situation at hand. Kahneman & Tversky1982 draw a distinction between 2 loci to which uncertainty can be attributed: the external world or our state of knowledge. Real-world systems are frequently perceived as having dispositions to produce different events, and the probabilities of these events are judged by assessing the strength of these dispositions. The propensity of the desk (or drawer) to produce the missing letter (or, for that matter, of the transportation system to produce the bus) may have been considered a fixed parameter of the setup by many of our subjects. This would explain why they refused to update that parameter in light of the accumulating search results. They did not interpret the question as addressing their state of knowledge, and were consequently impervious to the effect of new evidence.

Subjects often explicitly expressed the idea that constant probability was a characteristic disposition of the chance setup. The following statements were made by subjects who responded invariably with an answer of 80% to all the questions in numerical DL: “The probability that the letter is in the desk is 80%, and that’s it!” A deliberate attempt to ignore the information about successive failures (as if the subject is wary of falling prey to the gambler’s fallacy) is notable in another subject’s words: “Like the lottery, no matter how many times you play or what number you use, you have the same probability in winning. So each desk has an 80% chance of having the letter.” Similar insistence on the irrelevance of the reported outcomes is found in: “Finding empty drawers doesn’t change probability that letter is in desk”, and “The letter is equally likely (80%) to be in any of the drawers—so the fact that x number of drawers was checked does not lower the probability.”

The constancy of the long-run hope for the arrival of the bus (Problem 2) was justified by “I figure that the exact time between 11:30 and 12:00 (11:35, 11:45, 11:55) doesn’t really matter—since 60% of the buses operate this late I think there is still a 60% chance that a bus will come.” However, the same subject assumed constancy per unit when asked about short-term probabilities: “Since there are 8 drawers and the letter, if it is any of the drawers, is equally likely to be in any of the drawers, the probability that the letter is in any one drawer is 10%. This doesn’t change if the letter is not in one or more of the other drawers [italics added].” Had this discussion taken place in class, the teacher could have asked at that point, “and what if the letter is found in the _i_th drawer, would you still think the probability doesn’t change?”

[This kind of error could be considered a version of E.T. Jaynes’s mind projection fallacy (LW wiki); see Jaynes’s 1989_35ya paper “Probability Theory as Logic” or “Clearing up Mysteries—The Original Goal”. –Editor]

[pg366]

Assuming constant S when answering numerical BL, results in a decreasing _L function ((1) 50%, (2) 30%, (3) 10%). This was typically justified by answers such as: “I estimated that since there was 60% chance that the bus was still running…the chance of it arriving decreased by 10% as each 5 minute (of 6) passed.” Another subject’s explanation repeats the same rationale for DL: “There is 80% chance of letter in desk and 20% not. Checking one drawer with an unsuccessful try drops your chances of it being there by 10%, to 70%, and so on.”

In terms of the issue of “Evidential Impact of Base Rates” (the title of a paper by Tversky & Kahneman, 1982⁷), an assumption of constant L represents an extreme point of “conservatism” on the continuum of use versus neglect of base-rate data. In fact, constant L is the reverse of the “base rate fallacy” according to which subjects typically ignore the base rate and consider only the specific evidence about the case in hand (as in Tversky & Kahneman’s well-known cab problem). The constant S assumption, although resulting in exaggerated decrease of the L function, keeps the base-rate unit unchanged instead of duly increasing it in light of the evidence. In this sense, constant S is conservative as well.

Our impression is that subjects’ conservatism, as revealed by the prevalence of the constancy assumptions, is a consequence of their external attribution of uncertainty (Kahneman & Tversky, 1982_42ya). The parameters L₀ and/or S₀ are apparently perceived as properties that belong to the desk, like color, size and texture. Subjects think of these parameters in terms of “the probabilities of the desk”, whereas the Bayesian view would imply expressions like “my probability of the target event”. Thus, subjects fail to incorporate the additional knowledge they acquire when given successive search results.

15.2.3 Other Strategies

Several subjects denied the presence of chance altogether and acted as if it were certain that the letter was in the desk (the bus is going to come), and others embraced the historic position of equal ignorance and responded with “fifty-fifty”, in apparent disregard of the givens of the problem.

Eleven subjects relied on assumption of certainty in response to one of the problems they answered. Another 3 subjects assumed certainty in both problems. Most of the certainty-based responses were made by subjects assuming either constant L or constant S. Thus, for example, assuming initial certainty and constant S when responding to numerical BL means that S₀ = 17% = 1⁄6, and that S₀ is subtracted from the L function (starting with L₀ = 100%) for every 5-minute interval in which the bus does not arrive. This results in: (1) 83%, (2) 50%, (3) 17%. One subject justified this triplet as follows: “I made a time table of 30 minutes. I take a fraction of how much time has elapsed, then divide by 100%, giving the answer.” Note that this [pg367] subject was not disturbed by the fact that probability of 83% following a 5-minute wait for the bus was higher than the given initial probability of 60%.

The double assumption of initial certainty and constant L means that L₀ = 100% stays unchanged. Thus, when answering numerical DS, these 100 percents are divided each time in equal shares among the remaining drawers, resulting in (1) 14% = 1⁄7, (2) 25%, (3) 100%. We quote one subject’s elaborate justification of the above triplet: “If you didn’t eliminate drawers and randomly pointed to any drawer there would be still 1⁄8 probability because there is replacement. But here we don’t have any replacement and each drawer is equally likely of containing the letter, so however many drawers you have it’s 1⁄N probability.”

No less surprising than the responses that converted the initial probability of 80% (or 60%) into certainty were those that assumed total ignorance and concluded therefore that the target probability should be one half. 10 subjects appeared to invoke the maxim of “insufficient reason” assigning equal probabilities to the 2 possible outcomes. Consider the explanation of a subject who gave a constant 50% answer to all 3 BS questions: “Because since you don’t have any idea what time the bus arrives and you don’t even know if the bus is coming, then it is equally likely to arrive at any time.” Another subject, who responded similarly, wrote: “It doesn’t matter that the bus didn’t arrive in the last 5 minutes. There is always a 50% chance it will come an a 50% chance it will not come.” A uniform 50% response to the 3 BL questions was explained by: “There is no ↑ in probability it will come because there’s only 5 min left—there’s still a 50:50 chance it will either come or its doesn’t.” One subject’s “ingenious” reasoning with respect to BL resulted in: (1) 41.6%, (2) 25%, (C) 8.3%. His telegraphic-style explanation ran as follows:

6 5 min intervals from 11:30–12:00

- said it was =ly likely at 11:30 (50%)

50%:6 = 8.3

each 5 min interval decreases probability by 8.3%

We see here an interesting combination of the equal-ignorance and constant-S heuristics.

The human tendency to remove chance from our considerations has been observed in various judgmental contexts (several examples are reviewed by Falk & Konold, 1992⁸). The same is true for people’s inclination to assume equal likelihood once uncertainty is acknowledged. The tendency to identify randomness with equiprobability and thus assign equal chances to the available options has been widely documented in empirical investigations (eg. Konold et al 1991⁹; Shimojo & Ichikawa, 1989). The primacy of the equiprobability intuition has been described in studies of the historical [pg368] development of probability theory. Uniformity was the first presumption on which probability calculations were based (Gigerenzer et al 1989¹⁰; Hacking, 1975¹¹, Chapter 14). Converging evidence thus testifies to the genuine power of the intuitive bent toward symmetry (Falk, 1992; Zabell, 1988).

Paradoxically, subjects’ assumptions of certainty and of equal ignorance, although diametrically at odds with each other, might be viewed as the 2 Janus-faces of the same orientation. Konold (1989_35ya) has referred to that orientation as the outcome approach. People reasoning via the outcome approach tend to interpret a request for a probability of some event as a request to predict whether or not that event will occur on the next trial. Contrary to current scientific thinking, these reasoners do not view probability as a measure of one’s uncertainty, nor as answering the question about the relative frequency of occurrence of the target event in many repeated trials. According to Konold’s (1989_35ya) description, outcome-oriented subjects translate probability values into yes/no decisions, transforming their probability evaluations into certainty. Thus, a probability of 20% means “it won’t happen”, a probability of 80% means “it will happen”. When they sense a total lack of knowledge about the outcome, however, they express it by the 50:50 numerical probability, which means “it either will happen or won’t happen—don’t know which”. Konold found in several studies that a certain subgroup of the subjects (not necessarily a majority) was fairly consistent in responding according to this outcome-oriented perspective.

Although we cannot predict whether an outcome-oriented subject would convert the probabilities given in our problems into certainty or into equal ignorance, it stands to reason that the former would occur more often when the probabilities are close to 100% (or to zero) and the latter when the probabilities are close to 50%. Our data show roughly this pattern. Of the 17 forms which elicited certainty-based responses, 10 were desk problems (L₀ = 80%) and 7 bus problems (L₀ = 60%). In contrast, of the 10 equally likely answers, 3 were given in response to the desk story and 7 to the bus. These include 2 subjects who responded by certainty to the desk and by equal ignorance to the bus. Overall, the conjecture that the outcome approach has played some role in answering the hope problems is weakly supported. It remains a possibility that should be further explored.

15.2.4 Toward a Solution

It was somewhat surprising that we did not find among the explanations of the S problems an explication of the conflict between the diminishing long-term hope and the increasing immediate hope implied by the fewer remaining units. One subject who produced a constant S function in response to directional BS [pg369] did describe another conflict: “The probability that the bus will arrive in any given time slot is the same. Although my intuition would urge me to expect to see the bus more (meaning—I would assume the probability would be greater) as time elapsed, I believe that the ‘laws’ of probability would have it otherwise. But—as I think about it more, this could be argued against, saying that the probability changes as each unknown 5-minute segment became known.” Several subjects, who produced a decreasing L function in response to directional versions (without giving numbers), gave a correct Bayesian-like explanation (eg. “Well, if it is not in a drawer, then it could fall in the 20% zone and the more drawers you open without it being in there the lower the probability that it’s in there”).

Only one subject (No. 62), a pre-college student enrolled at the Hebrew University of Jerusalem, responded correctly to both problems, in this case to numerical BL and directional DS.

BL:

• At 11:30 the probability of the bus arriving by midnight was 6⁄10, and of not: 4⁄10

• At 11:35 the probability of the bus arriving by midnight was 5⁄9, and of not: 4⁄9

• At 11:45 the probability of the bus arriving by midnight was 3⁄7, and of not: 4⁄7

• At 11:55 the probability of the bus arriving by midnight was 1⁄5, and of not: 4⁄5

DS:

• At the beginning of the search the letter could be in one of 10 “locations”: 8 drawers and 2 “others”. The 2 “others” stay in constant amount, whereas the number of drawers keeps decreasing. Therefore, the chance of finding the letter in the first drawer was only 10% (ie. 1⁄10), in the second 1⁄9, in the fifth 1⁄6 and in the 8th 1⁄3.

These considerations yielded the same results (for each i) as the Bayesian computations. Note, however, that whenever several units are eliminated, the posterior probability distribution over the remaining units (including the imaginary “other” locations) stays uniform. That is why this subject’s reasoning matched the Bayesian results. The same method would not work if applied to problems like that of the 3 prisoners, or Monty’s notorious TV game “Let’s make a deal” (Falk, 1992_32ya).

Inspired by that subject’s method of solution, we devised a simplified version of the desk problem. The main change in the modified version was the addition of a concrete representation of the sample space that includes the 2 “locations” out of the desk.

[pg370]

Figure 15.2

A pilot test was run at the University of Massachusetts, Amherst with 13 subjects (including graduate and postgraduate students). Each subject responded to only one form: 6 to RDL and 7 to RDS. 3 of the responses to RDL and 6 of the responses to RDS were perfectly correct. Of the other [pg371] 3 RDLs, subjects gave 2 constant L responses and 1 constant S response. The 7^th RDS answer assumed constant L.

Based on the results of this pretest, we conducted larger-scale surveys. Our aim was both to confirm the indications that the revised versions facilitate reaching the correct solution and to test whether subjects who succeed in solving Problem 1R would transfer the solution principle to the Standard Wait Problem (Problem 2) as originally phrased.

Fifty-four subjects—34 undergraduate psychology students from the University of Massachusetts, Amherst and 20 senior high-school students from Massachusetts—participated in the first survey. Each subject was asked to answer 2 problems: either RDL and numerical BL, or RDS and numerical BS. The revised desk problem was always given first; 26 subjects received 2 L versions and 28 received 2 S versions.

Eighteen of the 54 revised forms were answered correctly (9 RDLs, and 9 RDSs). Compared with no correct answers to numerical DL and DS in the original group of 61 subjects, the rise to 33.3% correct represents a “dramatic” improvement. The 36 incorrect responses to the revised forms included 14 based on constancy assumptions (12 constant S and 2 constant L), 2 based on certainty and based on equal ignorance (ie. 50:50).

None of the 54 bus problems were answered correctly, indicating no transfer of the solution strategy by those 18 subjects who have just solved a search problem (desk). Incorrect responses included 26 constancy-based answers (20 constant S and 6 constant L), 10 certainty and 5 equal ignorance.

Correct answers to the revised desk problem were often accompanied by lucid explanations of the underlying reasoning. here is one example given in response to RDL: “The probability that I gave is the number of unlocked drawers remaining (unsearched) divided by the total number of drawers remaining (unlocked+locked).”

Similar to the explanations of incorrect answers to Problem 1, a constant 80% answer to RDL was justified by: “The overall probability doesn’t change no matter how many drawers are searched”, and, as maintained by another subject: “regardless of whether I looked in them or not.” Some subjects who responded 80% throughout seemed to work hard not to be swayed by the given results: “It’s like the boy/girl baby problem, even if you get BBBBBBBBBG the probability still remains at chance—50:50.” Constant S responses to RDL were justified by, “I was almost fooled, but upon further thought I decided that as drawers are searched and found empty the statistics of the problem do not change. Same as if weather person says 50% chance of rain & it rains. Then is probability of rain 50% or 100%? It’s still 50%.” One subject, who gave 50% answers to RDS, explained: “The possibility of finding the letter was 50%, just like yes or no.”

In a second survey, each subject received RDL and RDS, with order of presentation counterbalanced, and a bus problem of the same range (L or S) [pg372] as the second of the 2 revised problems. The 109 subjects were undergraduate students of psychology or graduate students of education at the Hebrew University of Jerusalem. 53 got RDS, RDL, BL, in that order, and 56 got RDL, RDS, BS. Because of the extra length of this assignment subjects were not asked to explain their reasoning.

Correct responses were given to 48 of the 109 RDLs (44%), and to 73 out of 109 RDSs (67%), which is 56% correct overall. Every subject who correctly solved RDL correctly solved RDS as well, but not vice versa, suggesting that the revised short-term problem is more transparent. This makes sense if one notes that answering the S versions involves adjustment of only the denominator (the total number of remaining units) since the numerator is always one, whereas answering the L versions requires adjustment of both numerator and denominator. Assumptions of constancy, certainty, and equal ignorance were observed among the incorrectly answered forms. However, the absence of supporting explanations prevented a determination of subject’s underlying reasoning.

No single correct triplet of answers was given to any of the 109 bus problems. This was true despite the high rate of correct solutions of the immediately preceding revised desk problems. In particular, 68 of 109 subjects solved their second revised desk problem but not the equivalent bus problem of the same range. In conclusion, while the revised desk problems elicited more than half correct solutions, transfer of the method of solution to the bus problem failed to occur.*

15.4.1 Why Didn’t the Transfer Work?

We were puzzled by the failure of all the subjects who had solved the revised desk problem to transfer the solution’s rationale to the bus problem. However, on second thought, and as a result of post-experimental discussions with some of the subjects, we have one possible reason for this failure of transfer.

The solution in the revised version was suggested by extending the dimension along which the search was carried out: 2 units (drawers) were added so the subjects could visualize the whole sample space and see the reason for the a priori L₀ of 80%. As they eliminated drawers, they could see the remaining “favorable” (unlocked) and “unfavorable” (locked) units of the changing space. When facing the bus problem, however, one cannot apply the same trick without changing the nature of the story. Extending the units of wait beyond midnight would not help to see the reason why L₀ is 60%. That prior reflects the fact that 60% of the bus routes operate this late, and 40% do not. A revision, equivalent to that of the desk problem, would have the bus [pg374] certain to arrive sometime between 11:30 and 12:20, with equal probabilities for all the 10 5-minute intervals. The tourists, however, decide to wait until either the bus arrives or midnight, whichever happens first. Viewing the original bus problem as isomorphic to the revised desk problem was apparently too much to expect of subjects in an experimental situation.

The locked-drawers device can easily be applied to Dr Fischer’s bomb situation (Example 2). Without loss of generality, we can change the story so that there are 12 Christmas crackers: one contains a bomb for sure, 11 contain checks. Only 6, however, are at the guests’ disposal for this party. The other 6 are kept for the next party. It is now easy to see that L₀ is 50% and to assess the L and S probabilities of pulling the bomb throughout the game’s progress. We didn’t pose this problem to our subjects. Our guess, however, is that transfer from the revised desk problem to this particular problem would have been within reach of some subjects.

15.4.2 Possible Extensions

Several subjects who viewed the search/wait process as analogous to coin flipping, incidentally raised an interesting equation: what if the search were to be conducted with replacement? Suppose the man who comes home in the darkness with 2 bunches of keys (Example 3) is drunk. He would not be able to remove keys that have failed to unlock the door (see Feller, 1957_67ya, page 46). Or, imagine that an absent-minded professor is looking for a misplaced letter through the drawers of her desk (Problem 1) while her mind is deeply engaged in some other problem, thus forgetting instantaneously which drawers have been already searched. A “with replacement search” thus describes the case in which a key may be tried again after being found not to work or a drawer may be searched again after being found empty.

Does it make sense to think of waiting for the bus “with replacement”? Ennis (1985_39ya)¹² describes a situation perfectly suited for our case. He imagines

waiting for a bus on a route which offers a “15-minute service”. Because of heavy traffic, the buses do not arrive at exactly 15-minute intervals but randomly. The operators (Poisson Motor Services) do, however, provide a service which averages out at 15 minutes between buses. (page 27)

We only need to change 15 to 30 minutes, add the qualification that the chances are 60% that the bus is running that late, and Poisson Motor Services provide us a wait problem with replacement.

The computation of the long-term (L’) and the short-term (S’) hope functions for sampling with replacement requires a minor adjustment of [pg375] formulas (15.1) and (15.2). One easily obtains, for the case of sampling with replacement:

• (15.1’)

  $$L_i^{\prime }=\frac{(n-1{)}^i\cdot L_0}{(n-1{)}^i\cdot L_0+n^i\cdot (1-L_0)}$$

• (15.2’)

  S’_i = 1⁄n × L’_i

In both formulas i = 0, 1, 2, …, n − 1, n, …

In contrast to the case of sampling without replacement, where the function L_i decreases with i and S_i increases (Figure 15.1), in the case of sampling with replacement, both L’_i and S’_i decrease. The rate of their decline, however, is slower than that of L_i. Figure 15.3 presents the course of the functions L’_i and S’_i compared with that of L_i and S_i, for the desk problem. In the limit, as i grows indefinitely, both L’_i and S’_i tend to zero. This means that despair creeps in justifiably in an extended fruitless search (wait) with replacement. In a without-replacement search, the rising S function may boost our morale to some degree. it is probably the short-range increase in hope that keeps most of us going.