As is true of every short description, this is a little over-simplified. People are risk-averse and fundamentally uncertain, so their beliefs about the true probability won’t directly translate into the percentage/price they will buy at, and one can’t even average out and say ‘this is what the market believes the probability is’. See economist Rajiv Sethi’s “On the Interpretation of Prediction Market Data” & “From Order Books to Belief Distributions”; for more rigor, see Wolfers & Zitzewitz’s paper, “Interpreting Prediction Market Prices as Probabilities”↩︎

Immanuel Kant, Critique of Pure Reason (A824/B852)↩︎

Or negative-sum, when you consider the costs of running the prediction market and the various fees that might be assessed on participants—the house needs a cut. In some circumstances, prediction markets can be positive-sum for traders: if some party benefits from the information and will subsidize it to encourage trading. For example, when companies run internal prediction markets they tend to subsidize the markets.

Public prediction market subsidies are much rarer—the only instance I know of is Peter McCluskey subsidizing 2008_16ya Intrade markets (announcement). As far as he could tell in November 2008_16ya, his subsidies did not do much. I emailed him May 2012_12ya, and he said:

I was somewhat disappointed with the results.

I don’t expect a small number of subsidized markets is enough to accomplish much. I suspect it would require many donors (or a billionaire donor) to create the markets needed for me to consider them successful. I see no hint that my efforts encouraged anyone else to subsidize such markets.

See also the LessWrong articles on the topic.↩︎

Rowe & Wright2001, reviewing studies of the Delphi method:

When one restricts the exchange of information among panelists so severely and denies them the chance to explain the rationales behind their estimates, it is no surprise that feedback loses its potency (indeed, the statistical information may encourage the sort of group pressures that Delphi was designed to pre-empt). We (Rowe and Wright1996) compared a simple iteration condition (with no feedback) to a condition involving the feedback of statistical information (means and medians) and to a condition involving the feedback of reasons (with no averages) and found that the greatest degree of improvement in accuracy over rounds occurred in the “reasons” condition. Furthermore, we found that, although subjects were less inclined to change their forecasts as a result of receiving reasons feedback than they were if they received either “statistical” feedback or no feedback at all, when “reasons” condition subjects did change their forecasts they tended to change towards more accurate responses. Although panelists tended to make greater changes to their forecasts under the “iteration” and “statistical” conditions than those under the ‘reasons’ condition, these changes did not tend to be toward more accurate predictions. This suggests that informational influence is a less compelling force for opinion change than normative influence, but that it is a more effective force. Best (1974_50ya) has also provided some evidence that feedback of reasons (in addition to averages) can lead to more accurate judgments than feedback of averages (eg. medians) alone.

It may be a stretch to generalize this to a single person predicting on their own, though many tools involve groups or you could view predicting as a Delphi method involving temporally separated selves. (If multiple selves works for Ainslie in explaining addiction, why not predicting?)↩︎

See “Eliminating the hindsight bias”, Arkes 1988_36ya.↩︎

Wikipedia’s criticism section remarks that “Kelly betting leads to highly volatile short-term outcomes which many people find unpleasant, even if they believe they will do well in the end.”↩︎

On 2008-01-27, the IEM sent out an email which accidentally listed all recipients in the CC; the listed emails totaled 292 emails. Given that many of these traders (like myself) are surely inactive or infrequent, and only a fraction will be active at a given time, this means the 10 or so markets are thinly inhabited.↩︎

The problem is that if a contract is at 10%, and you buy 10 contracts, then if the contract actually pays off, you have to come up with 100% to pay the other people their winnings. Intrade, to guarantee them payment, will make you pay the full 10%, and then freeze the 90% in your account.↩︎

This section first appeared on LessWrong.com as “2011 Intrade fee changes, or, Intrade considered no longer useful for LessWrongers” and includes some discussion.↩︎

When I submitted my withdrawal request for my balance, I received an email offering to instead set my account to ‘inactive’ status such that I could not trade but would not be charged the fee; if I wanted to trade, I would simply be charged that month’s $5. I declined the offer, but I couldn’t help wonder—why didn’t they simply set all accounts to ‘inactive’ and then let people opt in to the new fee structure? Or at least set ‘inactive’ all accounts which have not engaged in any transactions within X months?

Regardless, here are my probabilities for Intrade ending in the next few years:

• Intrade will close/merge/be sold by 2012: 5%

• Intrade will close/merge/be sold by 2013: 8%

• Intrade will close/merge/be sold by 2015: 18%

• Intrade will not be open for business in 2020: 35%

In March 2013_11ya (relevant events post-dating my predictions include the US CFTC attacking Intrade), Intrade announced it was shutting down trading and liquidating all positions. I probably was far too optimistic.↩︎

I made $0.31 on DEM.2012, $3.65 on REP.2012, and $1.40 on 2012.REP.NOM.PALIN for a total profit of $5.36.↩︎

An aside: there’s not much point in accumulating more than, say, 1000 bitcoins. It’s generally believed that Bitcoin’s ultimate fate will be victory or failure—it’d be very strange if Bitcoin leveled off as a stable permanent alternative currency for only part of the Internet. In such a situation, the difference between 1000 bitcoins and 1500 bitcoins is like the difference to Bill Gates between $60 billion and $65 billion; it matters in some abstract sense, but not even a tiny fraction as much as the difference between $1 and $100 million. Money is logarithmic in utility, as the saying goes.↩︎

Alex Tabarrok, “A Bet is a Tax on Bullshit”↩︎

The famous neurotransmitter dopamine is intimately involved with feelings of happiness and pleasure (which is why dopamine is affected by most addictions or addictive drugs). It also is involved in learning—make an error and no dopamine for you; “Midbrain Dopamine Neurons Encode a Quantitative Reward Prediction Error Signal” (Bayer & Glimcher 2005_19ya, Neuron):

The midbrain dopamine neurons are hypothesized to provide a physiological correlate of the reward prediction error signal required by current models of reinforcement learning. We examined the activity of single dopamine neurons during a task in which subjects learned by trial and error when to make an eye movement for a juice reward. We found that these neurons encoded the difference between the current reward and a weighted average of previous rewards, a reward prediction error, but only for outcomes that were better than expected. Thus, the firing rate of midbrain dopamine neurons is quantitatively predicted by theoretical descriptions of the reward prediction error signal used in reinforcement learning models for circumstances in which this signal has a positive value. We also found that the dopamine system continued to compute the reward prediction error even when the behavioral policy of the animal was only weakly influenced by this computation.

See also “The Psychology and Neuroscience of Curiosity”, Kidd & Hayden2015.↩︎

For example, no one has actually taken up Kevin’s offer to wager on the outcome to Amanda Knox’s appeal, while there are dozens of specific probabilities given in an earlier survey.↩︎

Unfortunately they don’t give any population statistics so it’s hard for me to interpret my results:

Your calibration score is -3. Calibration is defined as the difference between the percentage average confidence rating and the percentage of correct answers. A score of zero is perfect calibration. Positive numbers indicate overconfidence and can go up to 100. Negative numbers represent under-confidence and can go down to -100.

Your discrimination score is 4.48. Discrimination is defined as the difference between the percentage average confidence rating for the correct items and the percentage average confidence rating for the incorrect items. Higher positive numbers indicate greater discrimination and are better scores.

Specifically, prediction #1007. In its preface to the results page, GJP told us:

Question 1007_1,017ya (the “lethal confrontation” question) illustrates this point. Many of our best forecasters got ‘burned’ on this question because a Chinese fishing captain killed a South Korean Coast Guard officer late in the forecasting window—an outcome that the tournament’s sponsors deemed to satisfy the criteria for resolving the question as ‘yes’, but one that had little geopolitical significance (it did not signify a more assertive Chinese naval policy). These forecasters had followed our advice (or their own common sense) by lowering their estimated likelihood of a lethal confrontation as time elapsed and made their betting decisions based on this assumption.

For example, in the YourMorals.org tests dealing with calibration/bias, I usually do well above average, even for LessWrongers; see:

• “an experimental investigation of how people evaluate research evidence that either supports or opposes their pre-existing beliefs”

• “Over-claiming Technique”

• “Balanced Inventory of Desirable Responding”

• “Marlowe-Crowne Social Desirability Scale”

• “This scale is designed to measure the better-than-average effect, which is also known as the illusory superiority bias.”

The 2001_23ya anthology of reviews and papers, Principles of Forecasting, is invaluable, although many of the papers are highly technical. Excerpts from Dylan Evans’s Risk Intelligence (in the Wall Street Journal) may be more readable:

Psychologists have tended to assume that such biases are universal and virtually impossible to avoid. But certain groups of people-such as meteorologists and professional gamblers-have managed to overcome these biases and are thus able to estimate probabilities much more accurately than the rest of us. Are they doing something the rest of us can learn? Can we improve our risk intelligence?

Sarah Lichtenstein, an expert in the field of decision science, points to several characteristics of groups that exhibit high intelligence with respect to risk. First, they tend to be comfortable assigning numerical probabilities to possible outcomes. Starting in 1965_59ya, for instance, U.S. National Weather Service forecasters have been required to say not just whether or not it will rain the next day, but how likely they think it is in percentage terms. Sure enough, when researchers measured the risk intelligence of American forecasters a decade later, they found that it ranked among the highest ever recorded, according to a study in the Journal of the Royal Statistical Society.

It helps, too, if the group makes predictions only on a narrow range of topics. The question for weather forecasters, for example, is always roughly the same: Will it rain or not? Doctors, on the other hand, must consider all sorts of different questions: Is this rib broken? Is this growth malignant? Will this drug cocktail work? Studies have found that doctors score rather poorly on tests of risk intelligence.

Finally, groups with high risk intelligence tend to get prompt and well-defined feedback, which increases the chance that they will incorporate new information into their understanding. For weather forecasters, it either rains or it doesn’t. For battlefield commanders, targets are either disabled or not. For doctors, on the other hand, patients may not come back, or they may be referred elsewhere. Diagnoses may remain uncertain.

…Royal Dutch Shell introduced just such a program in the 1970s. Senior executives had noticed that when newly hired geologists predicted oil strikes at four out of 10 new wells, only one or two actually produced. This overconfidence cost Royal Dutch Shell millions of dollars. In the training program, the company gave geologists details of previous explorations and asked them for numerical estimates of the chances of finding oil. The inexperienced geologists were then given feedback on the number of oil strikes that had actually been made. By the end of the program, their estimates roughly matched the actual number of oil strikes.

…Just by becoming aware of our tendency to be overconfident or underconfident in our estimates, we can go a long way toward correcting for our most common errors. Doctors, for instance, could provide numerical estimates of probability when making diagnoses and then get data about which ones turned out to be right. As for the rest of us, we could estimate the likelihood of various events in a given week, record our estimates in numerical terms, review them the next week and thus measure our risk intelligence in everyday life. A similar technique is used by many successful gamblers: They keep accurate and detailed records of their earnings and their losses and regularly review their strategies in order to learn from their mistakes.

Modified version of Eliezer Yudkowsky’s parody of the Fate/Stay Night chant.↩︎

Long-shot bias is the overvaluing of events in the 0-5% range or so; it plagues even heavily traded markets on Intrade. Ron Paul and Michele Bachmann are 2 cases in point—they are covered by the heavily-traded US Presidential contracts, yet they are priced too high, and this has been noted by many:

• https://fskrealityguide.blogspot.com/2008_16ya/02/defect-of-intrade.html

• https://www.lesswrong.com/posts/4Rrkdz9eRL5HtH6cc/arbitrage-of-prediction-markets

• https://freakonomics.com/2007_17ya/05/what-do-you-have-to-say-about-ron-paul/

• https://web.archive.org/web/20111011185020/http://www.regruntled.com/2008_16ya/10/21/selling-delusion-short/

• https://web.archive.org/web/20111011185055/http://www.regruntled.com/2008_16ya/11/06/intrade-retrospective/

Beyond blog posts, a 2004 Wolfers & Zitzewitz paper finds their presence (see also Rothschild2011):

In fact, the price differences implied a (small) arbitrage opportunity that persisted for most of summer 2003_21ya and has reappeared in 2004_20ya. Similar patterns existed for Tradesports securities on other financial variables like crude oil, gold prices and exchange rates. This finding is consistent with the long-shot bias being more pronounced on smaller-scale exchanges.

This is apparently due in part to the short-term pressure on prediction market traders; Robin Hanson says:

“Intrade and IEM don’t usually pay interest on deposits, so for long term bets you can win the bet and still lose overall. The obvious solution is for them to pay such interest, but then they’d lose a hidden tax many customers don’t notice.”

Another reason to use a free-form site like PB.com—you can (and I have) made predictions about decades or centuries into the far future without worrying about how to earn returns of thousands of percent.↩︎

Going through Intrade to copy over predictions to PB.com, I was struck by how non-liquid markets could be left at hilarious prices, prices that make no rational sense since they can’t even represent someone hedging against that outcome because so few shares have been sold; example contracts include:

1. US attacking North Korea

2. China attacking Taiwan

3. Japan acquiring nuclear weapons

Paul Graham, “It’s Charisma, Stupid”↩︎

“In Front of Your Nose”, George Orwell 1946_78ya:

To see what is in front of one’s nose needs a constant struggle. One thing that helps toward it is to keep a diary, or, at any rate, to keep some kind of record of one’s opinions about important events. Otherwise, when some particularly absurd belief is exploded by events, one may simply forget that one ever held it. Political predictions are usually wrong. But even when one makes a correct one, to discover why one was right can be very illuminating. In general, one is only right when either wish or fear coincides with reality. If one recognizes this, one cannot, of course, get rid of one’s subjective feelings, but one can to some extent insulate them from one’s thinking and make predictions cold-bloodedly, by the book of arithmetic. In private life most people are fairly realistic. When one is making out one’s weekly budget, two and two invariably make four. Politics, on the other hand, is a sort of sub-atomic or non-Euclidean word where it is quite easy for the part to be greater than the whole or for two objects to be in the same place simultaneously. Hence the contradictions and absurdities I have chronicled above, all finally traceable to a secret belief that one’s political opinions, unlike the weekly budget, will not have to be tested against solid reality.

Eliezer Yudkowsky, chapter 20, Harry Potter and the Methods of Rationality:

…while I suppose it is barely possible that perfectly good people exist even though I have never met one, it is nonetheless improbable that someone would be beaten for fifteen minutes and then stand up and feel a great surge of kindly forgiveness for his attackers. On the other hand it is less improbable that a young child would imagine this as the role to play in order to convince his teacher and classmates that he is not the next Dark Lord.

The import of an act lies not in what that act resembles on the surface, Mr. Potter, but in the states of mind which make that act more or less probable.

“When falsification is the only path to truth”; abstract:

Can people consistently attempt to falsify, that is, search for refuting evidence, when testing the truth of hypotheses? Experimental evidence indicates that people tend to search for confirming evidence. We report two novel experiments that show that people can consistently falsify when it is the only helpful strategy. Experiment 1 showed that participants readily falsified somebody else’s hypothesis. Their task was to test a hypothesis belonging to an ‘imaginary participant’ and they knew it was a low quality hypothesis. Experiment 2 showed that participants were able to falsify a low quality hypothesis belonging to an imaginary participant more readily than their own low quality hypothesis. The results have important implications for theories of hypothesis testing and human rationality.

One line of thought in evolutionary psychology is that our minds are not evolved for truth-seeking per se, but rather are split between heuristics and effective procedures like that, and argumentation to try to deceive & persuade others; eg. “Why do humans reason? Arguments for an argumentative theory” (Mercier & Sperber 2011_13ya). This ties in well with why we are better at falsifying the theories of others—you don’t convince anyone by falsifying your own theories, but you do by falsifying others’ theories.↩︎

“Can self-prediction overcome barriers to hepatitis B vaccination? A randomized controlled trial”:

Half of participants were assigned randomly to a “self-prediction” intervention, asking them to predict their future acceptance of HBV vaccination. The main outcome measure was subsequent vaccination behavior. Other measures included perceived barriers to HBV vaccination, measured prior to the intervention. Results: There was a [statistically-]significant interaction between the intervention and vaccination barriers, indicating the effect of the intervention differed depending on perceived vaccination barriers. Among high-barriers patients, the intervention [statistically-]significantly increased vaccination acceptance. Among low-barriers patients, the intervention did not influence vaccination acceptance. Conclusions: The self-prediction intervention [statistically-]significantly increased vaccination acceptance among “high-barriers” patients, who typically have very low vaccination rates.

Rowe & Wright 2001_23ya:

• In phrasing questions, use clear and succinct definitions and avoid emotive terms.

  How a question is worded can lead to [substantial] response biases. By changing words or emphasis, one can induce respondents to give dramatically different answers to a question. For example, Hauser (1975_49ya) describes a 1940_84ya survey in which 96% of people answered yes to the question “do you believe in freedom of speech?” and yet only 22% answered yes to the question “do you believe in freedom of speech to the extent of allowing radicals to hold meetings and express their views to the community?” The second question is consistent with the first; it simply entails a fuller definition of the concept of freedom of speech. One might therefore ask which of these answers more clearly reflects the views of the sample. Arguably, the more apt representation comes from the question that includes a clearer definition of the concept of interest, because this should ensure that the respondents are all answering the same question. Researchers on Delphi per se have shown little empirical interest in question wording. Salancik, Wenger & Heifer 1971_53ya provide the only example of which we are aware; they studied the effect of question length on initial panelist consensus and found that one could apparently obtain greater consensus by using questions that were neither “too short” nor “too long.” This is a generally accepted principle for wording items on surveys: they should be long enough to define the question adequately so that respondents do not interpret it differently, yet they should not be so long and complicated that they result in information overload, or so precisely define a problem that they demand a particular answer. Also, questions should not contain emotive words or phrases: the use of the term “radicals” in the second version of the freedom-of-speech question, with its potentially negative connotations, might lead to emotional rather than reasoned responses.

• Frame questions in a balanced manner.

  Tversky and Kahneman (1974_50ya, 1981_43ya) provide a second example of the way in which question framing may bias responses. They posed a hypothetical situation to subjects in which human lives would be lost: if subjects were to choose one option, a certain number of people would definitely die, but if they chose a second option, then there was a probability that more would die, but also a chance that less would die. Tversky and Kahneman found that the proportion of subjects choosing each of the two options changed when they phrased the options in terms of people surviving instead of in terms of dying (ie. subjects responded differently to an option worded “60% will survive” than to one worded “40% will die,” even though these are logically identical statements). The best way to phrase such questions might be to clearly state both death and survival rates (balanced), rather than leave half of the consequences implicit. Phrasing a question in terms of a single perspective, or numerical figure, may provide an anchor point as the focus of attention, so biasing responses.

It may help to read the dialogue/examples of “Dr. Malfoy” and “Dr. Potter” in chapter 22 of Eliezer Yudkowsky’s Harry Potter and the Methods of Rationality.↩︎

For example, the famous and replicated examples of doctors failing to correctly apply Bayes’ theorem to cancer rates is reduced when the percentages are translated into frequencies. Rowe & Wright 2001_23ya give this advice:

• When possible, give estimates of uncertainty as frequencies rather than probabilities or odds.

Many applications of Delphi require panelists to make either numerical estimates of the probability of an event happening in a specified time period, or to assess their confidence in the accuracy of their predictions. Researchers on behavioral decision making have examined the adequacy of such numerical judgments. Results from these findings, summarized by Goodwin & Wright 1998_26ya, show that sometimes judgments from direct assessments (what is the probability that…?) are inconsistent with those from indirect methods. In one example of an indirect method, subjects might be asked to imagine an urn filled with 1,000 colored balls (say, 400 red and 600 blue). They would then be asked to choose between betting on the event in question happening, or betting on a red ball being drawn from the urn (both bets offering the same reward). The ratio of red to blue balls would then be varied until a subject was indifferent between the two bets, at which point the required probability could be inferred. Indirect methods of eliciting subjective probabilities have the advantage that subjects do not have to verbalize numerical probabilities. Direct estimates of odds (such as 25 to 1, or 1,000 to 1), perhaps because they have no upper or lower limit, tend to be more extreme than direct estimates of probabilities (which must lie between zero and one). If probability estimates derived by different methods for the same event are inconsistent, which method should one take as the true index of degree of belief? One way to answer this question is to use a single method of assessment that provides the most consistent results in repeated trials. In other words, the subjective probabilities provided at different times by a single assessor for the same event should show a high degree of agreement, given that the assessor’s knowledge of the event is unchanged. Unfortunately, little research has been done on this important problem. Beach & Phillips 1967_57ya evaluated the results of several studies using direct estimation methods. Test-retest correlations were all above 0.88, except for one study using students assessing odds, where the reliability was 0.66.

Gigerenzer (1994_30ya) provided empirical evidence that the untrained mind is not equipped to reason about uncertainty using subjective probabilities but is able to reason successfully about uncertainty using frequencies. Consider a gambler betting on the spin of a roulette wheel. If the wheel has stopped on red for the last 10 spins, the gambler may feel subjectively that it has a greater probability of stopping on black on the next spin than on red. However, ask the same gambler the relative frequency of red to black on spins of the wheel and he or she may well answer 50-50. Since the roulette ball has no memory, it follows that for each spin of the wheel, the gambler should use the latter, relative frequency assessment (50-50) in betting. Kahneman & Lovallo 1993_31ya have argued that forecasters tend to see forecasting problems as unique when they should think of them as instances of a broader class of events. They claim that people’s natural tendency in thinking about a particular issue, such as the likely success of a new business venture, is to take an “inside” rather than an “outside” view. Forecasters tend to pay particular attention to the distinguishing features of the particular event to be forecast (eg. the personal characteristics of the entrepreneur) and reject analogies to other instances of the same general type as superficial. Kahneman and Lovallo cite a study by Cooper, Woo, and Dunkelberger (1988_36ya), which showed that 80% of entrepreneurs who were interviewed about their chances of business success described this as 70% or better, while the overall survival rate for new business is as low as 33 percent. Gigerenzer’s advice, in this context, would be to ask the individual entrepreneurs to estimate the proportion of new businesses that survive (as they might make accurate estimates of this relative frequency) and use this as an estimate of their own businesses surviving. Research has shown that such interventions to change the required response mode from subjective probability to relative frequency improve the predictive accuracy of elicited judgments. For example, Sniezek & Buckley 1991_33ya gave students a series of general knowledge questions with two alternative answers for each, one of which was correct. They asked students to select the answer they thought was correct and then estimate the probability that it was correct. Their results showed the same general overconfidence that Arkes (2001_23ya) discusses. However, when Sniezek and Buckley asked respondents to state how many of the questions they had answered correctly of the total number of questions, their frequency estimates were accurate. This was despite the fact that the same individuals were generally overconfident in their subjective probability assessments for individual questions. Goodwin & Wright 1998_26ya discuss the usefulness of distinguishing between single-event probabilities and frequencies. If a reference class of historic frequencies is not obvious, perhaps because the event to be forecast is truly unique, then the only way to assess the likelihood of the event is to use a subjective probability produced by judgmental heuristics. Such heuristics can lead to judgmental overconfidence, as Arkes (2001_23ya) documents.

From Principles of Forecasting:

Osberg & Shrauger1986 determined prediction accuracy by scoring an item as a hit if the respondents predicted the event definitely or probably would occur and it did, or if the respondent predicted that the event definitely or probably would not occur and it did not. Respondents who were instructed to focus on their own personal dispositions predicted [statistically-]significantly more of the 55 items correctly (74%) than did respondents in the control condition who did not receive instructions (69%). Respondents whose instructions were to focus on personal base rates had higher accuracy (72%) and respondents whose instructions were to focus on population base rates had lower accuracy (66%) than control respondents, although these differences were not statistically-significant.

From Richards Heuer’s Psychology of Intelligence Analysis:

Ideally, intelligence analysts should be able to recognize what relevant evidence is lacking and factor this into their calculations. They should also be able to estimate the potential impact of the missing data and to adjust confidence in their judgment accordingly. Unfortunately, this ideal does not appear to be the norm. Experiments suggest that “out of sight, out of mind” is a better description of the impact of gaps in the evidence.

This problem has been demonstrated using fault trees, which are schematic drawings showing all the things that might go wrong with any endeavor. Fault trees are often used to study the fallibility of complex systems such as a nuclear reactor or space capsule.

A fault tree showing all the reasons why a car might not start was shown to several groups of experienced mechanics.⁹⁶ The tree had seven major branches–insufficient battery charge, defective starting system, defective ignition system, defective fuel system, other engine problems, mischievous acts or vandalism, and all other problems–and a number of subcategories under each branch. One group was shown the full tree and asked to imagine 100 cases in which a car won’t start. Members of this group were then asked to estimate how many of the 100 cases were attributable to each of the seven major branches of the tree. A second group of mechanics was shown only an incomplete version of the tree: three major branches were omitted in order to test how sensitive the test subjects were to what was left out.

If the mechanics’ judgment had been fully sensitive to the missing information, then the number of cases of failure that would normally be attributed to the omitted branches should have been added to the “Other Problems” category. In practice, however, the “Other Problems” category was increased only half as much as it should have been. This indicated that the mechanics shown the incomplete tree were unable to fully recognize and incorporate into their judgments the fact that some of the causes for a car not starting were missing. When the same experiment was run with non-mechanics, the effect of the missing branches was much greater.

As compared with most questions of intelligence analysis, the “car won’t start” experiment involved rather simple analytical judgments based on information that was presented in a well-organized manner. That the presentation of relevant variables in the abbreviated fault tree was incomplete could and should have been recognized by the experienced mechanics selected as test subjects. Intelligence analysts often have similar problems. Missing data is normal in intelligence problems, but it is probably more difficult to recognize that important information is absent and to incorporate this fact into judgments on intelligence questions than in the more concrete “car won’t start” experiment.

Rowe & Wright 2001_23ya:

• Use coherence checks when eliciting estimates of probabilities.

Assessed probabilities are sometimes incoherent. One useful coherence check is to elicit from the forecaster not only the probability (or confidence) that an event will occur, but also the probability that it will not occur. The two probabilities should sum to one. A variant of this technique is to decompose the probability of the event not occurring into the occurrence of other possible events. If the events are mutually exclusive and exhaustive, then the addition rule can be applied, since the sum of the assessed probabilities should be one. Wright & Whalley 1983_41ya found that most untrained probability assessors followed the additivity axiom in simple two-outcome assessments involving the probabilities of an event happening and not happening. However, as the number of mutually exclusive and exhaustive events in a set increased, more forecasters became supra-additive, and to a greater extent, in that their assessed probabilities added up to more than one. Other coherence checks can be used when events are interdependent (Goodwin and Wright 1998_26ya; Wright, et al 1994_30ya).

There is a debate in the literature as to whether decomposing analytically complex assessments into analytically more simple marginal and conditional assessments of probability is worthwhile as a means of simplifying the assessment task. This debate is currently unresolved (Wright, Saunders and Ayton 1988_36ya; Wright et al 1994_30ya). Our view is that the best solution to problems of inconsistency and incoherence in probability assessment is for the pollster to show forecasters the results of such checks and then allow interactive resolution between them of departures from consistency and coherence. MacGregor (2001_23ya) concludes his review of decomposition approaches with similar advice.

When assessing probability distributions (eg. for the forecast range within which an uncertainty quality will lie), individuals tend to be overconfident in that they forecast too narrow a range. Some response modes fail to counteract this tendency. For example, if one asks a forecaster initially for the median value of the distribution (the value the forecaster perceives as having a 50% chance of being exceeded), this can act as an anchor. Tversky & Kahneman 1974_50ya were the first to show that people are unlikely to make sufficient adjustments from this anchor when assessing other values in the distribution. To counter this bias, Goodwin & Wright 1998_26ya describe the “probability method” for eliciting probability distributions, an assessment method that de-emphasizes the use of the median as a response anchor. McClelland & Bolger 1994_30ya discuss overconfidence in the assessment of probability distributions and point probabilities. Wright & Ayton 1994_30ya provide a general overview of psychological research on subjective probability. Arkes (2001_23ya) lists a number of principles to help forecasters to counteract overconfidence.

Darwin discusses this further in the context of brain preservation in his “Science Fiction, Double Feature, 2: Part 2”; see also my plastination vs cryonics essay.↩︎

You might get the opposite impression reading articles like this New York Times article, but consider the flip side of large percentage growth in philanthropy—they must be starting off from a small absolute base!↩︎

Charles Simonyi is actually the first person to come to mind when I think about ‘weird wealthy American technologist interested in old and long-term information who has already demonstrated philanthropy on a large scale’↩︎

Walker was the second, due to his library. Information on his net wealth isn’t too easy to come by; he had ~$2.91^$1.6₂₀₀₀ billion in 2000_24ya but fell to $422.29^$300₂₀₀₉ million and Business Insider claims “Although he never recovered financially, Walker had enough money—barely—to complete an expensive dream home in Connecticut, which includes a fantastic personal library (featured by Wired in 2008_16ya).”↩︎

See Thiel’s essay “The Optimistic Thought Experiment: In the long run, there are no good bets against globalization”↩︎

And I use ‘seem’ advisedly; it’s remarkable how selfish his donations all appear to be.↩︎

I initially couldn’t find anything on charitable giving by Jobs. Eventually I found a The Times interview with Jobs where the reporter says “Jobs had volunteered himself as an advisor to John Kerry’s unsuccessful campaign for the White House. He and his wife, Lauren, had given hundreds of thousands of dollars to Democratic causes over the last few years.” Ars Technica mentions a few others, but conflates Jobs with Apple. A large $150m donation speculated to be Jobs has been confirmed to not be from him, although rumors about a $50m donation to a hospital continue to circulate. (In 2004_20ya, Fortune estimated Jobs’s fortune at $2.1 billion.) And in general, absence of evidence is evidence of absence. While Bono praises Apple for its charity:

Through the sale of (RED) products, Apple has been (RED)’s largest contributor to the Global Fund to Fight AIDS, Tuberculosis and Malaria—giving tens of millions of dollars that have transformed the lives of more than two million Africans through H.I.V. testing, treatment and counseling. This is serious and significant. And Apple’s involvement has encouraged other companies to step up.

Isaacson’s 2011_13ya Steve Jobs biography (finished before he died and so includes nothing on Jobs’s will) does occasionally discuss Jobs’s few acts of philanthropy and offers a different version of the (RED) contributions:

He was not particularly philanthropic. He briefly set up a foundation, but he discovered that it was annoying to have to deal with the person he had hired to run it, who kept talking about “venture” philanthropy and how to “leverage” giving. Jobs became contemptuous of people who made a display of philanthropy or thinking they could reinvent it. Earlier he had quietly sent in a $5,000 check to help launch Larry Brilliant’s Seva Foundation to fight diseases of poverty, and he even agreed to join the board. But when Brilliant brought some board members, including Wavy Gravy and Jerry Garcia, to Apple right after its IPO to solicit a donation, Jobs was not forthcoming. He instead worked on finding ways that a donated Apple II and a VisiCalc program could make it easier for the foundation to do a survey it was planning on blindness in Nepal….His biggest personal gift was to his parents, Paul and Clara Jobs, to whom he gave about $750,000 worth of stock. They sold some to pay off the mortgage on their Los Altos home, and their son came over for the little celebration. “It was the first time in their lives they didn’t have a mortgage,” Jobs recalled. “They had a handful of their friends over for the party, and it was really nice.” Still, they didn’t consider buying a nicer house. “They weren’t interested in that,” Jobs said. “They had a life they were happy with.” Their only splurge was to take a Princess cruise each year. The one through the Panama Canal “was the big one for my dad,” according to Jobs, because it reminded him of when his Coast Guard ship went through on its way to San Francisco to be decommissioned…[Mona Simpson’s novel] depicts Jobs’s quiet generosity to, and purchase of a special car for, a brilliant friend who had degenerative bone disease, and it accurately describes many unflattering aspects of his relationship with Lisa, including his original denial of paternity…Bono got Jobs to do another deal with him in 2006_18ya, this one for his Product Red campaign that raised money and awareness to fight AIDS in Africa. Jobs was never much interested in philanthropy, but he agreed to do a special red iPod as part of Bono’s campaign. It was not a wholehearted commitment. He balked, for example, at using the campaign’s signature treatment of putting the name of the company in parentheses with the word “red” in superscript after it, as in (APPLE)^RED. “I don’t want Apple in parentheses,” Jobs insisted. Bono replied, “But Steve, that’s how we show unity for our cause.” The conversation got heated-to the F-you stage-before they agreed to sleep on it. Finally Jobs compromised, sort of. Bono could do what he wanted in his ads, but Jobs would never put Apple in parentheses on any of his products or in any of his stores. The iPod was labeled (PRODUCT)^RED, not (APPLE)^RED.

Inside Apple: How America’s Most Admired—and Secretive—Company Really Works (Lashinsky 2012_12ya) reportedly includes the following choice anecdote:

A highlight of the Top 100 for attendees was an extended Q&A between Jobs and his executives. One asked why Jobs himself wasn’t more philanthropic. He responded that he thought giving away money was a waste of time.

“The Job After Steve Jobs: Tim Cook and Apple; From the moment he became CEO of Apple, Tim Cook found himself in the shadow of his boss”

Cook’s second decision was to start a charity program, matching donations of up to $10,000, dollar for dollar annually. This too was widely embraced: The lack of an Apple corporate-matching program had long been a sore point for many employees. Jobs had considered matching programs particularly ineffective because the contributions would never amount to enough to make a difference. Some of his friends believed that Jobs would have taken up some causes once he had more time, but Jobs used to say that he was contributing to society more meaningfully by building a good company and creating jobs. Cook believed firmly in charity. “My objective—one day—is to totally help others,” he said. “To me, that’s real success, when you can say, ‘I don’t need it anymore. I’m going to do something else.’”

Of course, if we really want to rescue Jobs’s reputation, we can still do. It could be the case that Jobs was very charitable, but he does so completely anonymously or perhaps that he has preferred to reinvest his wealth in gaining more wealth and only donating after his death; a Buffet-like strategy that—ex post—would seem to be a very wise one given the stock performance of AAPL. Jobs’s death October 2011_13ya means that this theory is falsifiable sooner than I had expected while writing this essay. Based on Jobs’s previous charitable giving, and the general impression I have from the hagiographic press coverage is Apple itself is Jobs’s charitable gift to the world (which I can’t help but suspect either influenced or is influenced by the man himself). My own general expectation is that he will definitely not donate ~99% of his wealth to charity like Buffett or Gates (80%), probably not >50% (70%), and more likely somewhere in the 0-10% range (60%). As of 2013-01-01, my predictions have been borne out. If any philanthropy comes of Jobs’s Pixar billions, I expect it to be at the behest of his widow, Laurene Powell Jobs, who has long been involved in non-profits; to quote Isaacson again:

Jobs’s relationship with his wife was sometimes complicated but always loyal. Savvy and compassionate, Laurene Powell was a stabilizing influence and an example of his ability to compensate for some of his selfish impulses by surrounding himself with strong-willed and sensible people. She weighed in quietly on business issues, firmly on family concerns, and fiercely on medical matters. Early in their marriage, she cofounded and launched College Track, a national after-school program that helps disadvantaged kids graduate from high school and get into college. Since then she had become a leading force in the education reform movement. Jobs professed an admiration for his wife’s work: “What she’s done with College Track really impresses me.” But he tended to be generally dismissive of philanthropic endeavors and never visited her after-school centers.

She has also tried her hand at lobbying for immigration reform, and continued her regular donations. Attempting to defend the Jobs’ reputations, is Laura Arrillaga-Andreessen

If you total up in your mind all of the philanthropic investments that Laurene has made that the public knows about, that is probably a fraction of 1% of what she actually does, and that’s the most I can say.

“The trouble with Steve Jobs”, CNNMoney.com, 2008-03-05:

Last year the founder of the Stanford Social Innovation Review called Apple one of “America’s Least Philanthropic Companies.” Jobs had terminated all of Apple’s long-standing corporate philanthropy programs within weeks after returning to Apple in 1997_27ya, citing the need to cut costs until profitability rebounded. But the programs have never been restored.

Unlike Bill Gates—the tech world’s other towering figure—Jobs has not shown much inclination to hand over the reins of his company to create a different kind of personal legacy. While his wife is deeply involved in an array of charitable projects, Jobs’ only serious foray into personal philanthropy was short-lived. In January 1987_37ya, after launching Next, he also, without fanfare or public notice, incorporated the Steven P. Jobs Foundation. “He was very interested in food and health issues and vegetarianism,” recalls Mark Vermilion, the community affairs executive Jobs hired to run it. Vermilion persuaded Jobs to focus on “social entrepreneurship” instead. But the Jobs foundation never did much of anything, besides hiring famed graphic designer Paul Rand to design its logo. (Explains Vermilion: “He wanted a logo worthy of his expectations.”) Jobs shut down the foundation after less than 15 months.