Optimization Obtained

As it happens, we now have unfathomable amounts of computing power. What was once a modus tollens is now just a modus ponens.

Corporations, and tech companies in particular as the leading edge, routinely solve planning problems for logistics like fleets of cars or datacenter optimization involving millions of variables; the similar SAT solvers are ubiquitous in computer security research for modeling large computer codebases to verify safety or discover vulnerabilities; most robots couldn’t operate without constantly solving & optimizing enormous systems of equations. The internal planned ‘economies’ of tech companies have grown kudzu-like, sprouting ever larger datasets to predict and automated analyses to plan and market designs to control. The problems solved by retailers like Walmart or Target are world-sized.² (‘“We are not setting the price. The market is setting the price”, he says. “We have algorithms to determine what that market is.”’) The motto of a Google or Amazon or Uber might be (to paraphrase Freeman Dyson’s paraphrase of John von Neumann in Infinite in All Directions, 1988_36ya): “All processes that are stable we shall plan. All processes that are unstable we shall compete in (for now).” Companies may use some limited internal ‘markets’ as useful metaphors for allocation, and dabble in prediction markets, but the internal dynamics of tech companies bear little resemblance to competitive free markets, and show little sign of moving in market-ward directions.

The march of planning also shows little sign of stopping. Uber is not going to stop using historical forecasts of demand to move around drivers to meet expected demand and optimize trip trajectories; datacenters will not stop using linear solvers to allocate running jobs to machines in an optimal manner to minimize electricity consumption while balancing against latency and throughput, in search of a virtuous cycle culminating in the optimal route, “the perpetual trip, the trip that never ends”; ‘markets’ like smartphone walled gardens rely ever more each year on algorithms parsing human reviews & binaries & clicks to decide how to rank or push advertising and conduct multi-armed bandit exploration of options; and so on endlessly.

So, can we run an economy with scaled-up planning approaching 100% centralization, while increasing efficiency and even outcompeting free capitalism-style competitive markets, as Cockshott & Cottrell propose (a proposal occasionally revived in pop socialism like The People’s Republic of Walmart: How the World’s Biggest Corporations are Laying the Foundation for Socialism)?

Artificial Persons

The striking thing about corporations improving is that they don’t; corporations don’t evolve (see the Price equation & multi-level selection, which can be applied to many things). The business world would look completely different if they did! Despite large persistent differences in efficiency between corporations (a long noted paradox), the best management practices or corporations don’t simply ‘clone’ themselves and regularly take over arbitrary industries with their superior skills (and after reaching fixation, eventually succumbing to mutant offspring who have become even more efficient than them, and so on).

We can copy the best software algorithms, like AlphaZero, indefinitely and they will perform as well as the original, and we can tweak them in various ways to make them steadily better (and this is in fact how many algorithms are developed, by constant iteration); species can reproduce themselves, steadily evolving to ever better exploit their niches, not to mention the power of selective breeding programs; individual humans can refine teaching methods and transmit competence (calculus used to be reserved for the most skilled mathematicians, and now is taught to ordinary high school students, and chess grandmasters have become steadily younger with better & more intensive teaching methods like chess engines); we could even clone exceptional individuals to get more similarly talented individuals, if we really wanted to. But we don’t see this happen with corporations. Instead, despite desperate struggles to maintain “corporate culture”, companies typically coast along, getting more and more sluggish, failing to spin off smaller companies as lean & mean as they used to be, until conditions change or random shocks or degradation finally do them in, such as perhaps some completely-unrelated company (sometimes founded by a complete outsider like a college student) eating their lunch.

Why do we not see exceptional corporations clone themselves and take over all market segments? Why don’t corporations evolve such that all corporations or businesses are now the hyper-efficient descendants of a single ur-corporation 50 years ago, all other corporations having gone extinct in bankruptcy or been acquired? Why is it so hard for corporations to keep their “culture” intact and retain their youthful lean efficiency, or if avoiding ‘aging’ is impossible, why copy themselves or otherwise reproduce to create new corporations like themselves? Instead, successful large corporations coast on inertia or market failures like regulatory capture/monopoly, while successful small ones worry endlessly about how to preserve their ‘culture’ or how to ‘stay hungry’ or find a replacement for the founder as they grow, and there is constant turnover. The large corporations function just well enough that maintaining their existence is an achievement³.

Evolution & the Price equation requires 3 things: entities which can replicate themselves; variation of entities; and selection on entities. Corporations have variation, they have selection—but they don’t have replication.

Corporations certainly undergo selection for kinds of fitness, and do vary a lot. The problem seems to be that corporations cannot replicate themselves. They can set up new corporations, yes, but that’s not necessarily replicating themselves—they cannot clone themselves the way a bacteria can. When a bacteria clones itself, it has… a clone, which is difficult to distinguish in any way from the ‘original’. In sexual organisms, children still resemble their parents to a great extent. But when a large corporation spins off a division or starts a new one, the result may be nothing like the parent and completely lack any secret sauce. A new acquisition will retain its original character and efficiencies (if any). A corporation satisfies the Peter Principle by eventually growing to its level of incompetence, which is always much smaller than ‘the entire economy’. Corporations are made of people, not interchangeable easily-copied widgets or strands of DNA. There is no ‘corporate DNA’ which can be copied to create a new one just like the old. The corporation may not even be able to ‘replicate’ itself over time, leading to scleroticism and aging—but this then leads to underperformance and eventually selection against it, one way or another. So, an average corporation appears little more efficient, particularly if we exclude any gains from new technologies, than an average corporation 50 years ago, and the challenges and failures of the rare multinational corporation 500 years ago like the Medici bank look strikingly similar to challenges and failures of banks today.

We can see a similar problem with other large-scale human organizations: ‘cultures’. An idea seen sometimes is that cultures undergo selection & evolution, and as such, are made up of adaptive beliefs/practices/institutions, which no individual understands (such as farming practices optimally tailored to local conditions); even apparently highly irrational & wasteful traditional practices may actually be an adaptive evolved response, which is optimal in some sense we as yet do not appreciate (sometimes linked to “Chesterton’s fence” as an argument for status quo-ism).

This is not a ridiculous position, since occasionally certain traditional practices have been vindicated by scientific investigation, but the lenses of multilevel selection as defined by the Price equation shows there are serious quantitative issues with this: cultures or groups are rarely driven extinct, with most large-scale ones persisting for millennia; such ‘natural selection’ on the group-level is only tenuously linked to the many thousands of distinct practices & beliefs that make up these cultures; and these cultures mutate rapidly as fads and visions and stories and neighboring cultures and new technologies all change over time (compare the consistency of folk magic/medicine over even small geographic regions, or in the same place over several centuries). For most things, ‘traditional culture’ is simply flatout wrong and harmful and all forms are mutually contradictory, not verified by science, and contains no useful information, and—contrary to “Chesterton’s fence”—the older and harder it is to find a rational basis for a practice, the less likely it is to be helpful:

Chesterton’s meta-fence: “in our current system (democratic market economies with large governments) the common practice of taking down Chesterton fences is a process which seems well established and has a decent track record, and should not be unduly interfered with (unless you fully understand it)”.

The existence of many erroneous practices, and the successful diffusion of erroneous ones, is acknowledged by proponents of cultural evolution like Heinrich (eg. Heinrich provides several examples which are comparable to genetic drift spreading harmful mutations, and Primo Levi coined “the onion in the varnish” for such things), so the question here is one of emphasis or quantity: is the glass 1% full or 99% empty? It’s worth recalling the conditions for human expertise (Armstrong 2001_23ya, Principles of Forecasting; Tetlock 2005_19ya, Expert Political Judgment: How Good Is It? How Can We Know?; ed Ericsson 2006_18ya, The Cambridge Handbook of Expertise and Expert Performance; Kahneman & Klein2009): repeated practice with quick feedback on objective outcomes in unchanging environments; these conditions are satisfied for relatively few human activities, which are more often rare, with long-delayed feedback, left to quite subjective appraisals mixed in with enormous amounts of randomness & consequences of many other choices before/after, and subject to potentially rapid change (and the more so the more people are able to learn). In such environments, people are more likely to fail to build expertise, be fooled by randomness, and construct elaborate yet erroneous theoretical edifices of superstition (like Tetlock’s hedgehogs). Evolution is no fairy dust which can overcome these serious inferential problems, which are why reinforcement learning is so hard.⁴

For something like farming, with regular feedback, results which are enormously important to both individual and group survival, and relatively straightforward mechanistic cause-and-effect relationships, it is not surprising that practices tend to be somewhat optimized (although still far from optimal, as enormously increased yields in the Industrial Revolution demonstrate, in part by avoiding the errors of traditional agriculture & using simple breeding techniques)⁵; but none of that applies to ‘traditional medicine’, dealing as it does with complex self-selection, regression to the mean, and placebo effects, where aside from the simplest cases like setting broken bones (again, straightforward, with cause-and-effect relationship), hardly any of it works⁶ and one is lucky if a traditional remedy is merely ineffective rather than outright poisonous, and in the hardest cases like snake bites, it would be better to wait for death at home than waste time going to the local witch doctor.

So—just like corporations—‘selection’ of cultures happens rarely with each ‘generation’ spanning centuries or millennia, typically has little to do with how reality-based their beliefs tend to be (for a selection coefficient approaching zero), and if one culture did in fact consume another one thanks to more useful beliefs about some herb, it is likely to backslide under the bombardment of memetic mutation (so any selection is spent just purging mutations, creating a mutation-selection balance); under such conditions, there will be little long-term ‘evolution’ towards higher optima, and the information content of culture will be minimal and closely constrained to only the most universal, high-fitness-impact, and memetically-robust aspects.

Natural Persons

Contrast that with a human. Despite ultimately being designed by evolution, evolution then plays no role at ‘runtime’ and more powerful learning algorithms take over.

With these more powerful algorithms designed by the meta-algorithm of evolution, a human is able to live successfully for over 100 years, with tremendous cooperation between the trillions of cells in their body, only rarely breaking down towards the end with a small handful of seed cancer cells defecting over a lifetime despite even more trillions of cell divisions and replacements. (And these cancers themselves must solve coordination problems like blood vessel growth—cooperation with someone is always cooperation against someone else.) They are also able to be cloned, yielding identical twins so similar across the board that people who know them may be unable to distinguish them. And they don’t need to use evolution or markets to develop these bodies, instead, relying on a complex hardwired developmental program controlled by genes which ensures that >99% of humans get the two pairs of eyes, lungs, legs, brain hemispheres etc that they need.

Perhaps the most striking efficiency gain from a human is the possession of a brain with the ability to predict the future, learn highly abstract models of the world, and plan and optimize over these plans for objectives which may only relate indirectly to fitness decades from now or fitness-related events which happen less than once in a lifetime & are usually unobserved or fitness events like that of descendants which can never be observed.

RL

Taxonomy of Pain

So let’s consider the possibilities when it comes to pain. There isn’t just “pain”. There is (at the least):

• useless painful pain (chronic pain, exercise)

• useful painful pain (the normal sort)

• useless nonpainful nonpain (dead nerves in diabetes or leprosy⁹ or congenital pain insensitivity^{1011121314 15}; bed sores and RSI are everyday versions demonstrating that even the most harmless activities like ‘lying on a bed’ are in fact constantly causing damage)

• useful nonpainful nonpain (adrenaline rushes during combat)

• useless nonpainful pain (pain asymbolia where they maim & kill themselves, possibly also Lesch-Nyhan syndrome);

• and intermediate cases: like the Marsili family who have a genetic mutation (Habib et al 2018) which partially damages pain perception. The Marsilis do feel useful painful pain but only briefly, and incur substantial bodily damage (broken bones, scars) but avoid the most horrific anecdotes of those with deadened nerves or pain asymbolia.

  Another interesting case is the Scotswoman Jo Cameron, who has a different set of mutations to her endocannabinoid system (FAAH & FAAH-OUT): while not as bad as neuropathy, she still exhibits similar symptoms—aside from odd symptoms like being unusually forgetful, her father may also have been a carrier and died peculiarly, she regularly burns or cuts herself in household chores, she broke her arm roller-skating as a child but didn’t seek treatment until it reset at a “funny angle”, delayed treatment of a damaged hip & then a hand damaged by arthritis until almost too late¹⁶ (and may have damaged her joints so badly because of the lack of pain), took in foster children who stole her savings, etc. Her son carries one of her 2 mutations, and “frequently scalds his mouth with hot drinks and food”. (Biologist Matthew Hill describes the most common FAAH mutation as causing “low levels of anxiety, forgetfulness, a happy-go-lucky demeanor”, and “Since the paper was published, Matthew Hill has heard from half a dozen people with pain insensitivity, and he told me that many of them seemed nuts” compared to Jo Cameron.) A possible case is the absurdly prolific fantasy writer Brandon Sanderson, who routinely puts in 8-hour days of typing (on a couch, to boot), and who turns out to be almost entirely insensitive to pain.

• but—is there ‘useful painless pain’ or ‘useless painful nonpain’?

It turns out there is ‘painless pain’: lobotomized people experience that, and “reactive dissociation” is the phrase used to describe the effects sometimes of analgesics like morphine when administered after pain has begun, and the patient reports, to quote Dennett1978 (emphasis in original), that “After receiving the analgesic subjects commonly report not that the pain has disappeared or diminished (as with aspirin) but that the pain is as intense as ever though they no longer mind it…if it is administered before the onset of pain…the subjects claim to not feel any pain subsequently (though they are not numb or anesthetized—they have sensation in the relevant parts of their bodies); while if the morphine is administered after the pain has commenced, the subjects report that the pain continues (and continues to be pain), though they no longer mind it…Lobotomized subjects similarly report feeling intense pain but not minding it, and in other ways the manifestations of lobotomy and morphine are similar enough to lead some researchers to describe the action of morphine (and some barbiturates) as ‘reversible pharmacological leucotomy [lobotomy]’.²³”¹⁷

And we can find examples of what appears to be ‘painful nonpain’: Grahek2001 highlights a case-study, Ploner et al 1999, where the German patient’s somatosensory cortices suffered a lesion from a stroke, leading to an inability to feel heat normally on one side of his body or feel any spots of heat or pain from heat; despite this, when sufficient heat was applied to a single spot on the arm, the patient became increasingly agitated, describing an “clearly unpleasant” feeling associated with his whole arm, but also denied any description of it involving crawling skin sensations or words like “slight pain” or “burning”.

A table might help lay out the possibilities:

Pain serves a clear purpose (stopping us from doing things which may cause damage to our bodies), but in an oddly unrelenting way which we cannot disable and which increasingly often backfires on our long-term interests in the form of ‘chronic pain’ and other problems. Why doesn’t pain operate more like a warning, or like hunger or thirst? They interrupt our minds, but like a computer popup dialogue, after due consideration of our plans and knowledge, we can generally dismiss them. Pain is the interruption which doesn’t go away, although we could easily imagine pains which did go away or were in fact ‘pleasurable’ (Morsella2005):

Theoretically, nervous mechanisms could have evolved to solve the need for this particular kind of interaction otherwise. Apart from automata, which act like humans but have no phenomenal experience, a conscious nervous system that operates as humans do but does not suffer any internal strife. In such a system, knowledge guiding skeletomotor action would be isomorphic to, and never at odds with, the nature of the phenomenal state—running across the hot desert sand in order to reach water would actually feel good, because performing the action is deemed adaptive.¹⁹ Why our nervous system does not operate with such harmony is perhaps a question that only evolutionary biology can answer. Certainly one can imagine such integration occurring without anything like phenomenal states, but from the present standpoint, this reflects more one’s powers of imagination than what has occurred in the course of evolutionary history.

Hui Neng’s Flag

In the reinforcement learning context, one could ask: does it make a difference whether one has ‘negative’ or ‘positive’ rewards? Any reward function with both negative and positive rewards could be turned into all-positive rewards simply by adding a large constant. Is that a difference which makes a difference? Or instead of maximizing positive ‘rewards’, one could speak of minimizing ‘losses’, and one often does in economics or decision theory or control theory²⁰.

���Do Artificial Reinforcement-Learning Agents Matter Morally?”, Tomasik2014, debates the relationship of rewards to considerations of “suffering” or “pain”, given the duality between costs-losses/rewards:

Perhaps the more urgent form of refinement than algorithm selection is to replace punishment with rewards within a given algorithm. RL systems vary in whether they use positive, negative, or both types of rewards:

• In certain RL problems, such as maze-navigation tasks discussed in Sutton & Barto 1998_26ya, the rewards are only positive (if the agent reaches a goal) or zero (for non-goal states).

• Sometimes a mix between positive and negative rewards⁶ is used. For instance, McCallum 1993_31ya put a simulated mouse in a maze, with a reward of 1 for reaching the goal, −1 for hitting a wall, and −0.1 for any other action.

• In other situations, the rewards are always negative or zero. For instance, in the cart-pole balancing system of Barto et al 1990_34ya, the agent receives reward of 0 until the pole falls over, at which point the reward is −1. In Koppejan & Whiteson2011’s neuroevolutionary RL approach to helicopter control, the RL agent is punished either a little bit, with the negative sum of squared deviations of the helicopter’s positions from its target positions, or a lot if the helicopter crashes.

Just as animal-welfare concerns may motivate incorporation of rewards rather than punishments in training dogs [Hiby et al 2004_20ya] and horses [Warren-Smith & McGreevy 2007_17ya, Innes & McBride 2008_16ya], so too RL-agent welfare can motivate more positive forms of training for artificial learners. Pearce 2007_17ya envisions a future in which agents are driven by ‘gradients of well-being’ (ie. positive experiences that are more or less intense) rather than by the distinction between pleasure versus pain. However, it’s not entirely clear where the moral boundary lies between positive versus negative welfare for simple RL systems. We might think that just the sign of the agent’s reward value r would distinguish the cases, but the sign alone may not be enough, as the following section explains.

What’s the boundary between positive and negative welfare?

Consider an RL agent with a fixed life of T time steps. At each time t, the agent receives a non-positive reward r_t ≤ 0 as a function of the action a_t that it takes, such as in the pole-balancing example. The agent chooses its action sequence (at) t = 1…T with the goal of maximising the sum of future rewards:

$$\sum _{t=1}^T{r}_t\left(a_t\right)$$

Now suppose we rewrite the rewards by adding a huge positive constant c to each of them, r′t = rt + c, big enough that all of the r′_t are positive. The agent now acts so as to optimize

$$\sum _{t=1}^T{r}_t^{\prime }\left(a_t\right)=\sum _{t=1}^T\left(\right(r_t)a_t+c)=Tc+\sum _{t=1}^T{r}_t\left(a_t\right)$$

So the optimal action sequence is the same in either case, since additive constants don’t matter to the agent’s behavior.⁷ But if behavior is identical, the only thing that changed was the sign and numerical magnitude of the reward numbers. Yet it seems absurd that the difference between happiness and suffering would depend on whether the numbers used by the algorithm happened to have negative signs in front. After all, in computer binary, negative numbers have no minus sign but are just another sequence of 0s and 1s, and at the level of computer hardware, they look different still. Moreover, if the agent was previously reacting aversively to harmful stimuli, it would continue to do so. As Lenhart K. Schubert explains:⁸ [This quotation comes from spring 2014 lecture notes (accessed March 2014) for a course called “Machines and Consciousness”.]

If the shift in origin [to make negative rewards positive] causes no behavioral change, then the robot (analogously, a person) would still behave as if suffering, yelling for help, etc., when injured or otherwise in trouble, so it seems that the pain would not have been banished after all!

So then what distinguishes pleasure from pain?

…A more plausible account is that the difference relates to ‘avoiding’ versus ‘seeking.’ A negative experience is one that the agent tries to get out of and do less of in the future. For instance, injury should be an inherently negative experience, because if repairing injury was rewarding for an agent, the agent would seek to injure itself so as to do repairs more often. If we tried to reward avoidance of injury, the agent would seek dangerous situations so that it could enjoy returning to safety.¹⁰ [This example comes from Lenhart K. Schubert’s spring 2014 lecture notes (accessed March 2014), for a course called ‘Machines and Consciousness.’ These thought experiments are not purely academic. We can see an example of maladaptive behavior resulting from an association of pleasure with injury when people become addicted to the endorphin release of self-harm.] ²¹ Injury needs to be something the agent wants to get as far away from as possible. So, for example, even if vomiting due to food poisoning is the best response you can take given your current situation, the experience should be negative in order to dissuade you from eating spoiled foods again. Still, the distinction between avoiding and seeking isn’t always clear. We experience pleasure due to seeking and consuming food but also pain that motivates us to avoid hunger. Seeking one thing is often equivalent to avoiding another. Likewise with the pole-balancing agent: Is it seeking a balanced pole, or avoiding a pole that falls over?

…Where does all of this leave our pole-balancing agent? Does it suffer constantly, or is it enjoying its efforts? Likewise, is an RL agent that aims to accumulate positive rewards having fun, or is it suffering when its reward is suboptimal?

Pain As Grounding

So with all that for background, what is the purpose of pain?

The purpose of pain, I would say, is as a ground truth or outer loss. (This is a motivational theory of pain with a more sophisticated RL/psychiatric grounding.)

The pain reward/loss cannot be removed entirely for the reasons demonstrated by the diabetics/lepers/congenital insensitives: the unnoticed injuries and the poor planning are ultimately fatal. Without any pain qualia to make pain feel painful, we will do harmful things like run on a broken leg or jump off a roof to impress our friends²², or just move in a not-quite-right fashion and a few years later wind up paraplegics. (An intrinsic curiosity drive alone would interact badly with a total absence of painful pain: after all, what is more novel or harder to predict than the strange and unique states which can be reached by self-injury or recklessness?)

If pain couldn’t be removed, could pain be turned into a reward, then? Could we be the equivalent of Morsella’s mind that doesn’t experience pain, as it infers plans and then executes them, experiencing only more or less rewards? It only experience positive rewards (pleasure) as it runs across burning-hot sands, as this is the optimal action for it to be taking according to whatever grand plan it has thought of.

Perhaps we could… but what stops Morsella’s mind from enjoying rewards by literally running in circles on those sands until it dies or is crippled? Morsella’s mind may make a plan and define a reward function which avoids the need for any pain or negative rewards, but what happens if there is any flaw in the computed plan or the reward estimates? Or if the plan is based on mistaken premises? What if the sands are hotter than expected, or if the distance is much further than expected, or if the final goal (perhaps an oasis of water) is not there? Such a mind raises serious questions about learning and dealing with errors: what does such a mind experience when a plan fails? Does it experience nothing? Does it experience a kind of “meta-pain”?

Consider what Brand (The Gift of Pain again, pg191–197) describes as the ultimate cause of the failure of years of research into creating ‘pain prosthetics’, computerized gloves & socks that would measure heat & pressure in real-time in order to warn those without pain like lepers or diabetics: the patients would just ignore the warnings, because stopping to prevent future problems was inconvenient while continuing paid off now. And when electrical shockers were added to the system to stop them from doing a dangerous thing, Brand observed patients simply disabling it to do the dangerous thing & re-enabling it afterwards! Less exotically, consider professional athletes: promising careers end every day due to injuries, and one of the most important playing skills of a top athlete in the NBA or NFL is being able to not play.

What pain provides is a constant, ongoing feedback which anchors all the estimates of future rewards based on planning or bootstrapping. It anchors our intelligence in a concrete estimation of bodily integrity: the intactness of skin, the health of skin cells, the lack of damage to muscles, joints sliding and moving as they ought to, and so on. If we are planning well and acting efficiently in the world, we will, in the long run, on average, experience higher levels of bodily integrity and physical health; if we are learning and choosing and planning poorly, then… we won’t. The badness will gradually catch up with us and we may find ourselves blind scarred paraplegics missing fingers and soon to die. A pain that was not painful would not serve this purpose, as it would merely be another kind of “tickling” sensation. (Some might find it interesting or enjoyable or it could accidentally become sexually-linked.) The perceptions in question are simply more ordinary tactile, kinesthetic, thermoreceptor, or other standard categories of perception; without painful pain, a fire burning your hand simply feels warm (before the thermal-perceptive nerves are destroyed and nothing further is felt), and a knife cutting flesh might feel like a rippling stretching rubbing movement.

We might say that a painful pain is a pain which forcibly inserts itself into the planning/optimization process, as a cost or lack of reward to be optimized. A pain which was not motivating is not what we mean by ‘pain’ at all.²³ The motivation itself is the qualia of pain, much like an itch is an ordinary sensation coupled with a motivational urge to scratch. Any mental quality or emotion or sensation which is not accompanied by a demandingness, an involuntary taking-into-consideration, is not pain. The rest of our mind can force its way through pain, if it is sufficiently convinced that there is enough reason to incur the costs of pain because the long-term reward is so great, and we do this all the time: we can convince ourselves to go to the gym, or withstand the vaccination needle, or, in the utmost extremity, saw off a trapped hand to save our life. And if we are mistaken, and the predicted rewards do not arrive, eventually the noisy constant feedback of pain will override the decisions leading to pain, and whatever incorrect beliefs or models led to the incorrect decisions will be adjusted to do better in the future.

But the pain cannot and must not be overridden: human organisms can’t be trusted to simply ‘turn off’ pain and indulge an idle curiosity about cutting off hands. Note that we can kill ourselves by starvation or thirst, but we cannot kill ourselves by refusing to sleep, or have a heart beat, or breathe—unless one suffers from the (extremely lethal) central hypoventilation syndrome, that is. We are insufficiently intelligent, our priors insufficiently strong, our reasoning and planning too poor, and we must do too much learning within each life to do without pain.

A similar argument might apply to the puzzle of ‘willpower’, ‘procrastination’. Why do we have such problems, particularly in a modern context, doing aught we know we should and doing naught we oughtn’t?

On the grave of the ‘blood glucose’ level theory, Kurzban et al 2013 (see later Shenhav et al 2017) erects an opportunity cost theory of willpower. Since objective physical measurements like blood glucose levels fail to mechanically explain poorer brain functionality or why strenuous activities like sports are ‘restful’ & reduce ‘burnout’, similar to the failure of objective physical measurements like lactate levels to explain why people are able to physically exercise only a certain amount (despite being able to exercise far more if properly motivated or if tricked), the reason for willpower running out must be subjective.

To explain the sugar-related observations, Kurzban et al 2013_11ya suggest that the aversiveness of long focus and cognitive effort is a simple heuristic which creates a baseline cost to focusing for ‘too long’ on any one task, to the potential neglect of other opportunities, with the sugar interventions (such as merely tasting sugar water) which appear to boost willpower actually serving as proximate reward signals (signals, because the actual energetic content is nil, and cognitive effort doesn’t meaningfully burns calories in the first place), which justify to the underlying heuristic that further effort on the same task is worthwhile and the opportunity cost is minimal.

The lack of willpower is a heuristic which doesn’t require the brain to explicitly track & prioritize & schedule all possible tasks, by forcing it to regularly halt tasks—“like a timer that says, ‘Okay you’re done now.’”²⁴ If one could override fatigue at will, the consequences can be bad. Users of dopaminergic drugs like amphetamines often note issues with channeling the reduced fatigue into useful tasks rather than alphabetizing one’s bookcase. In more extreme cases, if one could ignore fatigue entirely, then analogous to lack of pain, the consequences could be severe or fatal: ultra-endurance cyclist Jure Robič would cycle for thousands of kilometers, ignoring such problems as elaborate hallucinations, and was eventually killed while cycling. The ‘timer’ is implemented, among other things, as a gradual buildup of adenosine, which creates sleep homeostatic drive pressure and possibly physical fatigue during exercise (Noakes2012, Martin et al 2018), leading to a gradually increasing subjectively perceived ‘cost’ of continuing with a task/staying awake/continuing athletic activities, which resets when one stops/sleeps/rests. (Glucose might work by gradually dropping over perceived time without rewards.) Since the human mind is too limited in its planning and monitoring ability, it cannot be allowed to ‘turn off’ opportunity cost warnings and engage in hyperfocus on potentially useless things at the neglect of all other things; procrastination here represents a psychic version of pain.

From this perspective, it is not surprising that so many stimulants are adenosinergic or dopaminergic²⁵, or that small children might especially struggle with mental fatigue (there is a world full of novel opportunities tempting them away), or that many anti-procrastination strategies (like Getting Things Done or the Procrastination Equation) boil down to optimizing for more rewards or more frequent rewards (eg. breaking tasks down into many smaller tasks, which can be completed individually & receive smaller but more frequent rewards, or thinking more clearly about whether something is worth doing): all of these would affect the reward perception itself, and reduce the baseline opportunity cost ‘pain’. This perspective may also shed light on depression²⁶, or on occupational burnout and why restorative hobbies are ideally maximally different from jobs and more miscellaneous observations like the lower rate of ‘hobbies’ outside the West: burnout may be a long-term homeostatic reaction to spending ‘too much’ time too frequently on a difficult not-immediately rewarding task despite earlier attempts to pursue other opportunities (perhaps tasks which would never be rewarding), which were always overridden, ultimately resulting in a total collapse²⁷; and hobbies ought to be as different in location and physical activity and social structure (eg. a solitary programmer indoors should pursue a social physical activity outdoors as soulcraft) to ensure that it feels completely different for the mind than the regular occupation; and in places with less job specialization or fewer work-hours, the regular flow of a variety of tasks and opportunities means that no such special activity as a ‘hobby’ is necessary. A further analogy might be to communication: email and chat message overload, compared to ‘worse’ communication technologies, may reflect a lack of ‘pain’ to the sender—resulting in ‘damage’ to the overloaded recipient.

Perhaps if we were superintelligent AIs who could trivially plan flawless humanoid locomotion at 1000_1,024yaHz taking into account all possible damages, or if we were emulated brains sculpted by endless evolutionary procedures to execute perfectly adaptive plans by pure instinct, or if we were simple amoeba in a Petri dish who had no real choices to make, there would be no need for a pain which was painful. And likewise, were we endlessly planning and replanning to the end of days, we should never experience akrasia, we should merely do what is necessary (perhaps not even experiencing any qualia of effort or deliberation, merely seeing events endlessly unfold as they always had to). But we are not. The pain keeps us honest. In the end, pain is our only teacher.

Meta-Learning Paradigms

Metz et al 2018

Knuth

yosefk observes that Chuck Moore’s systems designed using Forth & custom hardware/OS/userland are breathtakingly more efficient than ‘standard’ approaches would yield, and this is because Moore takes a global perspective and optimizes “end-to-end”: changing the language if that makes the OS simpler, changing the hardware if that’d make the text editor smaller, redefining the problem if necessary, and making large globally-coherent sets of changes that more myopic or constrained programmers could not or would not do. They are less ‘designed’ than evolved (but evolved by intelligence), and share similar traits: a general contempt for rigid abstraction or principles like “modularity”, a reliance on ‘coincidence’ for correctness, and incredible performance (in both terms of efficiency and in solving whatever the problem is). This approach makes for amazing custom ‘one-off’ systems; but these are inherently difficult to understand by lesser mortals, the general ‘just git gud’ approach unreplicable³¹, and the system may not be modifiable by lesser mortals (even if the original designer could easily modify it or just rewrite it overnight to be brilliantly optimized for the new set of requirements). Similar issues bedevil Lisp systems & programmers: they can, and so they do.

This reminds me of Donald Knuth. Knuth is one of the most brilliant computer scientists ever, who does things like program an ALGOL compiler by himself, mostly in his head, for a summer job. He writes projects like the T_eX system by sitting down and spending days thinking hard about it, creating a single program littered with poor software-engineering like global variables & GOTOs but which is nevertheless lightning-fast & almost bug-free. T_eX is not his only programming language either, he has created others like METAFONT (for fonts) or MIX/MMIX (an assembly language & computer architecture to provide simple & timeless implementations of his TAOCP programs).

Perhaps the most striking thing about these various languages is that everyone who uses them loves the quality of the output & what you can do with them, but hate the confusing, complicated, inconsistent, buggy experience of using them (to the extent that there is a whole cottage industry of people attempting to rewrite or replace T_eX/LaTeX, typically copying the core ideas of how T_eX does typesetting—the ‘box and glue’ paradigm of how to lay out stuff onto a page, the Knuth-Plass line-breaking, the custom font families etc—doing all that reimplementation work just so they don’t have to ever deal with the misery of T_eX-the-language). Knuth himself, however, appears to have no more difficulty programming in his languages than he did writing 1960s assembler or designing fonts with 60 parameters. As he puts it, he ignores most programming techniques and just writes the right code, “Otherwise, lots of time is wasted on activities that I simply never need to perform or even think about.” If you need to make a complex use of T_eX like typesetting an entire Bible, then Knuth says you should simply read it, and fork it however necessary. That should be easy—after all, that’s what he would do!

How do you write code as well as Don Knuth? The answer seems to be, well, ‘be as naturally gifted as Don Knuth’. A major omission in his œuvre is that he has never done major work on operating systems (which cannot be kept in one’s head), and his major effort in improving software engineering, “literate programming”, which treats programs as 1 long story to be told, fell stillborn from the presses. (Knuth, who has the privilege of an academic in being able to ship a program without any worries about maintenance or making a living, and just declaring something done, like T_eX, perhaps does not appreciate how little real-world source code is like telling stories and instead readers decode them.)

So this is a bit of a problem for any attempts at making programming languages more powerful or more ergonomic, as well as for various kinds of ‘tools for thought’ like Douglas Engelbart’s intelligence amplification program. It is terribly easy for such powerful systems to be terribly hard to learn, and the ability to handle extreme levels of complexity can cripple one’s ability to remove complexity. (The ability to define all sorts of keyboard shortcuts & abbreviations invoking arbitrary code dynamically in your Lisp machine text editor is great until you ask ‘how am I going to remember all these well enough to save any time on net?’) Just as it tends to be easier to write your own incorrect code than to understand old correct code, it is easier for large groups of people to write repetitive concrete code case by case than to understand the abstract code that solves it all. Tasking people like Knuth or Engelbart to develop such things is a bit like asking a top sports player to be a coach: it’s not the same thing at all, and the very factors which made them so freakishly good at the sport may damage their ability to critique or improve—they may not be able to do much more than demonstrate how to do it well, and say, ‘now you do that too and git gud’.

From an AI perspective, this is interesting because it suggests that AIs might be powerful even while coding with human-legible code. If the problem with the ‘Moore/Knuth approach’ is that you can’t clone him indefinitely to rewrite the system every time it’s necessary, then what happens with AIs which you can ‘just clone’ and apply exclusively to the task? Quantity has a quality all its own. (For a fictional example, see Vernor Vinge’s SF novel A Deepness in the Sky, where talented individuals can be put into a ‘Focused’ state, where they become permanently monomaniacally obsessed with a single technical task like rewriting computer systems to be optimal, and always achieve Knuth-level results; giving their enslavers de facto superpowers compared to rivals who must employ ordinary people. For a more real-world example, consider how Google fights bitrot & infrastructure decay: not by incredibly sophisticated programming languages—quite the opposite, considering regressions like Go—but employing tens of thousands of top programmers to literally rewrite all its code every few years on net, developing an entire parallel universe of software tools, and storing it all in a single giant repository to assist the endless global rewrites of the big ball of mud.)

Internet Community Design

Internet community architecture can be seen as a bi-level optimization design too. There are fast and slow methods of interaction, and ideas or knowledge (or just ‘memes’) are created, varied, and selected on. These interactions happen inside discrete communities which are themselves created, varied, and grow. So, they are instances of multilevel selection.

The “warrens and plazas”³² interpretation of communities is a 2-level design. The classic example used to be Usenet and FAQs: the fast daily (or even minute by minute) discussion would happen, and knowledge would be distilled down into FAQ entries to save time, foster a higher level of discussion, and spread the knowledge outside of participants. A more contemporary example is Reddit: the fast flux of link submissions and comments can be distilled into “stickied” (permanent) links, and a simple ‘wiki’ system of editable pages (enabling FAQs and more). Some subreddits for niche interests (often gaming or medical-related) have built up considerable knowledge bases and greatly advanced their particular niche. Discord has done well in marrying the relatively slow IRC-style chat channels with the even faster-paced voice communication of gaming, while simultaneously supporting long-term use-cases through stickied (‘pinned’) comments which can contain complicated formatting like blockquotes & be edited, search of full channel histories, bots, allowing many channels/servers with complicated security permissions, etc. (It has done less well in enabling any of this to be archived or exported.)

Counter-examples also exist. Many social networks are actively hostile to any kind of 2-level structure, emphasizing one level at the expense of another. The value of each time-scale can be seen in how social networks can thrive while targeting only a single time-scale. Facebook is moderately hostile to long or in-depth posts; they can exist, but no real support is given to them, capabilities like formatting are minimal to nonexistent, and the UI & all affordances are 100% oriented to ‘most recent first’. Slack chat is the evil twin of Discord: its free plan destroys history almost immediately, and should one pay through the nose for full Slack, one quickly discovers that it’s email but worse. Twitter goes further, providing essentially no support for any kind of longform at all, never mind editable wiki/FAQ pages; but just because a technology does not enable a use case doesn’t mean users don’t need it, it just means they’ll have to do it painfully and worse than if it did, and so Twitter users struggle with threads to provide some sort of slow structure to the usual nauseatingly evanescent stream of fluff tweets. Instagram & TikTok are even worse, and Snapchat makes no bones of trying to destroy even the possibility of a slow stream. YouTube enables slow content quite well, and is excellent at education; it seems to struggle with fast, though—comments were a notorious cesspit for many years, and chatting or interactive streaming is something it’s been trying to catch up with compared to pioneers like Twitch.

Weird hybrid examples exist. Consider 4chan, famously the meme-maker to the Internet. A chan consists of flat ‘threads’ of comments one after another (any tree being implicit), ordered by most-recently-updated thread, with the last thread being deleted after a certain timeout; threads are nested within a general topic or ‘board’. 4chan threads typically move fast and disappear within days. At first glance, this might seem to preclude any kind of progress. But 4chan is nevertheless famous for incubating memes and projects over the course of many threads (all long since deleted by the final success). How? Part of it is that successful threads may export to other boards; then other boards export to slow sites like specialist wikis or other social media networks, like Twitter, Facebook, and Reddit. So there is a 3-level selection: a comment within a thread, interesting threads within the board, and interesting board contents within the Internet. Threads select brutally for memetic efficiency (the ticking clock makes chan threads almost literally a viral evolution lagoon), and while this selection is extremely error-prone and inaccurate, there are so many comments & memes, and fans of a meme will maintain personal archives and keep posting variants (being anonymous, they are free to keep posting without care or consequence), that the memetic virus can keep mutating and evolving until perhaps it starts percolating outwards. (And if it doesn’t, oh well, plenty more where that came from!)

A broad perspective on this is to think of graphs of communities, where each node is a community of a certain size operating at a certain speed with differing norms about quality/topic/esthetics/anonymity etc. If we think of each community as generating & filtering memes, then there are tradeoffs between size, accuracy of filtering, and throughput. If you have 1 very large community, it will have extremely accurate selection on popularity (fitness) of a given meme, because it is averaging the assessments of almost everyone (despite steeply diminishing returns to spending more people to do selection); however, it will struggle to keep up with potential throughput and its queues will overflow creating a bottleneck impeding bursty collaboration of exciting new ideas, and where will new memes come from if slightly-inferior variants are harshly punished immediately compared to the current fit baseline? Over-exploitation will become boring, driving away users—at best, stasis; at worst, decadence, exhaustion, & collapse. If you have lots of tiny communities, they will undergo extreme levels of “genetic drift” due to randomness in popularity, and the fitness of their best meme will typically be quite poor; but on the other hand, it is likely that at least one of those communities has random-walked its way to something neat (if only you could figure out which one…) Depending on how these communities are connected, these neat new variants may be confined to a ghetto and eventually die out due to randomness (either themselves or perhaps the communities) if they can’t grow fast enough to reach fixation; but if you make them all hyper-connected, you may just wind up constructing the 1 large bottlenecked community again!

The architecture of speed and size is probably responsible for a lot of the ‘feel’ of social networking. Twitter, for example, is lauded for giving access to the latest by the greatest anywhere, but produces a certain exhaustion and apathy and chronic low-grade anxiety and lowest-common denominator humor, and this is the flipside: because it emphasizes only the latest, there is no progression or collaborative creation (people can advertise on Twitter to collaborate elsewhere, or ask specific questions, or learn about something on Twitter, but there is nothing like a “Twitter FAQ thread” or long-term collaboration on Twitter), and because it can be from anywhere, the norms are unpredictable and “context collapse” means an entire outside community could decide to coordinate to make you the target of the latest ‘5-minute hate’.

Pavlogiannis et al 2018 (“Construction of arbitrarily strong amplifiers of natural selection using evolutionary graph theory”) considers this sort of scenario and finds that good graph structures & distributions tend to look like a hierarchical “star” or “hub-and-spoke” (or perhaps the ubiquitous “bow-tie”). There are many small ‘local’ nodes at the periphery, which focus on ‘generating’ innovations, and these feed in a generally one-way direction into progressively larger nodes focused on ‘selecting’, which eventually reach a few ‘global’ nodes which are connected to all the peripheries again. (Like in biological evolution, the number of nodes or ‘populations’ can matter a lot, as does the history of the populations, which may have an ‘effective’ population count much smaller the visible one.)

Large masses of raw material, be they writing, or images, or videos, or sick skating techniques, are collaboratively written, proceeding from rough draft through editing to the final perfected version. As a kid I wondered vaguely how famous intellectuals could have “30 boxes of notebooks” or “20 volumes of collected letters” or “10 volumes of unpublished papers”—wasn’t writing and thinking hard, how did they have time for all that incessant note-taking and letter-writing, while still living and researching and actually writing the published work they were famous for? The answer, it turned out, is simply that writing is a sequential collaborative process: those letters and unpublished papers were part of a pipeline; it was not “letters vs books” but “letters into books”. A heuristic like the rule of three (“if you find yourself explaining something for the third time, write it up”) is about deciding what to promote up a level: the repetition implies that it’s important, and conveniently, three rough drafts are already available.

This will suddenly all sound eerily familiar: it is our old friend reinforcement learning and its explore-exploit tradeoff, with outer evolutionary losses and inner learned losses, all over again! Too small a batch size and you don’t learn anything; too large, and it takes an eternity to improve; too little exploration & too much greedy exploitation, one learns unnecessarily slowly & may get trapped in a local optimum, but too much exploration ensures one never learns anything before bouncing off to the next random point; breaking up into multiple agents and populations can cover more ground than a single uniform population but only if they are balanced properly and transfer improvements; hierarchical structure can enable deep exploration and modularity, where a monolithic structure flails around locally but can be done poorly; an evolutionary loss is extremely inefficient compared to a learned inner loss explicitly optimizing for proxy goals, yet, without the evolutionary loss, the proxy goals may be wrong; and so on. But with our graph to explore and amplify, and individual humans as myopically Bayesian agents, we discover our overall community looks like a giant Thompson sampling engine!

So this offers a general theory of Internet community design: one wants an architecture which is hierarchical, supporting a smooth flow of content from a wide variety of small peripheral nodes operating on fast time-scales with their own unique norms fostering social contagion of ambition with incentives for directed exploration of new niches or uncovered territory³³, upwards through a hierarchy of larger slower nodes with gradually more intense filtering & more conventional norms (including escalating reliance on reputation), to a final global central ‘arena’ where the best can duke it out for transmission back to all peripheral nodes. The design should not privilege a particular time-scale, and should enable linking and copying through the various time-scales; nodes should be constrained in size, and broken up if necessary to keep them at an efficient size.

One could imagine a Reddit which integrated chat, links, & wiki pages to create a smoother transition between nodes and directly support promotion:

1. a subreddit has a chat channel which defaults to anonymous and is not logged or archived, with the minimum possible moderation;

2. blocks of fast-time chat (seconds to minutes), however, can be highlighted and right-clicked to automatically turn into a comment on a link or their own text post, ‘promoting’ them up one level to slower-time, where they can be discussed over hours to days;

3. such comments, perhaps on some topic of sudden new interest, may be further selected, and transcluded into a new wiki page devoted to that topic (crude, but a starting point as a comment index), which can then be hand-edited later to add in additional commentary, links, citations, etc;

4. these pages become a long-term resource for that subreddit, and perhaps turn out to be of broader interest, being crossposted to other, bigger, subreddits,

5. and, amazingly enough, eventually reach /r/all where it is pushed to all users as part of their default feed—adding a new thing to the global ‘common knowledge’, which (return to #1) some other subreddit chat might idly discuss for a while and then make an unexpected breakthrough, kicking off a new cycle.