A small fraction of human responses will always be garbage because we are lazy, bored, trolling, or crazy.
Researchers have demonstrated repeatedly in human surveys the stylized fact that, far from being an oracle or gold standard, a certain small percentage of human responses will reliably be bullshit: “jokester” or “mischievous responders”, or more memorably, “lizardman constant” responders—respondents who give the wrong answer to simple questions.
Below a certain percentage of responses, for sufficiently rare responses, much or all of responding humans may be lying, lazy, crazy, or maliciously responding and the responses are false. This systematic error seriously undermines attempts to study rare beliefs such as conspiracy theories, and puts bounds on how accurate any single survey can hope to be.
Non-zero base rate of nonsense human answers. To briefly review Scott Alexander’s lizardman constant: human survey-takers will, with >0% probability, endorse the most absurd items on a survey, for a mix of reasons like laziness, boredom, humor, sabotage, ignorance, and stupidity. For example, 4% of respondents may endorse the claim ‘lizard-people rule the earth’, 5% of atheists believe in God, a surprising number of adults believe you see by shooting beams from your eyes, and so on. This cautions us against taking survey results about extremely unusual people or traits too literally, or expecting perfectly accurate results, as given the lizardman constant and other crud factors, it is entirely possible that some or all of the outliers may just be the lizardman constant at work.
Why? ‘All of the above’. What drives the lizardman constant, such that a small percentage of people will report in response to survey questions believing that reptilians run the world? These people are some unanalyzable mix of lazy, stupid1, ignorant, trolling, ‘joking’, or just mentally deviant, possibly causing extreme anecdotes which are re-amplified by clickbait global media. (Teens delight in falsely reporting being amputees, adoptees, LGBT, binge-drinkers etc, and “joke through the entire thing”; one should also be wary of answers coming from users whose self-reported gender is ‘unicorn’ or ‘Apache helicopter’.)
Human failures of logic & common sense, examples. One interesting example: 75–96% of the almost-million-strong American Independent Party apparently have no idea they joined it and thought they had registered as just “independents”. (Inasmuch as American knowledge of institutions is weak—eg. 2% of Americans have “never heard of” institutions such as “the police” or the “U.S. Supreme Court” or “China”, and 5% disavow knowledge of any such country as an “Israel”—we perhaps should not be surprised at confusion over parties.) Science examples are well-known to demonstrate severe problems with both knowledge and responses, as ~25% say geocentrism is true, and 60% that the Big Bang didn’t happen, the letter ‘o’ has no hole in it, and so on; similarly, in arithmetic, performance can be quite low, with <20% of Americans able to identify the even numbers in a list of 6, and ~10% of Americans able to identify which of 6 are prime (equivalent to random guessing—forget about multiplying 4-digit numbers!); 7% say chocolate milk comes from brown cows & 19% that hamburgers come from pigs & 20% hold concerning beliefs about the origins of fish fingers, which is at least better than claiming to have eaten funistrada (and children have more peculiar beliefs, although we should also note that adult naturalist knowledge is nothing to boast about, as 6–8% of Americans think they could kill a grizzly bear, lion, elephant, or gorilla with their bare hands); Pew’s Religious Landscape Study finds 5% of atheists are “absolutely” or “fairly certain” that they believe in God (I’ll charitably pass over meat-eating rates in vegans/vegetarians as a case of “the spirit is willing but the flesh is sweet”); in the 2016 US presidential election, 14% of undecided voters said Hillary Clinton might be a demon, but they might vote for her; 2% of Clinton supporters said she was & they would; Scott Alexander mentions the eponymous 4% of responders who say lizardmen rule the earth, but, to continue the infernal theme, notes also that 13% say Barack Obama is the Antichrist (5% voted Obama); while 5% say Magic 8-Balls predict the future (and—the monsters—5% approve of using cell phones in movie theaters); and perhaps trafficking with the ruinous powers explains how 4% have been decapitated and yet are still answering surveys, with satanic deals surely helping the 2% who endorse bleach injections as a coronavirus cure2 (not that they actually use bleach, of course). Terrifying as would-be bleach-drinkers may be, are they less frightening than the 1% of Brits who believe you should be imprisoned for 15 years (or more) if you ever fail to wear a seatbelt in your car? (When the safety revolution comes for me, please remember me as a wild & free spirit.)
Such examples could surely be multiplied (literally ad nauseam, even?). Certainly, I (and Scott Alexander) see many bizarre responses any time we are unlucky enough to run a survey which contains a free response field!
Sampling proves presence but not absence in humans too… If challenged on their absurd responses, they will dig their heels in and give ridiculous justifications. Unfortunately, there is no foolproof remedy against lizardman responses (one can use ‘attention checks’ and tests for overclaiming, like letting them endorse lists of items with fakes thrown in to catch the bullshitters), because humans gonna human. The reality is that humans don’t answer questions reliably, accurately, or honestly even close to 100% of the time, and shamelessly fail ‘common sense’ or ‘logic’ or ‘arithmetic’ questions all the time, requiring extensive precautions, careful survey design, and just throwing out a lot of data as garbage.
See Also:
-
And regular people are indeed often fairly incompetent, struggling with basic reading comprehension tasks etc.↩︎
-
The real reason to get the coronavirus vaccine, 9% of vaccinated people know, is for the free microchip courtesy of Bill Gates. (They are, by definition, a different 2% from the 1–2% who regularly tell Opinium Research that “I have not heard of coronavirus” in response to “Q0. How do you feel about Coronavirus?”.)↩︎