AI

Matters Of Scale:

• Hardware:

  • “Podracer architectures for scalable Reinforcement Learning”, Hessel et al 2021 (highly-efficient TPU pod use: eg. solving Pong in <1min at 43 million FPS on a TPUv3-2048); “Google details new TPUv4 AI accelerator chips” (2.7× TPUv3 chips; up to TPUv4-4096 pods, yielding >1 ExaFLOPS; public access later in 2021)

  • “ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning”, Rajbhandari et al 2021 (~1 trillion parameters per 16 GPUs/DGX-2-node, scaling to >512 GPUs ~40% efficiency)

  • “GSPMD: General and Scalable Parallelization for ML Computation Graphs”, Xu et al 2021 (Google upgrade of GPipe/GShard arch to match MS DeepSpeed: “…50%–62% compute utilization on 128–2048 Cloud TPUv3 cores for models with up to one trillion parameters”)

  • “DLRM: High-performance, Distributed Training of Large-scale Deep Learning Recommendation Models”, Mudigere et al 2021 (ZionEX software/hardware platform for training extremely large embeddings—while embeddings aren’t ‘real’ parameters & things like DynamicEmbedding will never learn tricks like GPT-3 no matter how big, they present similar challenges); “RecPipe: Co-designing Models and Hardware to Jointly Optimize Recommendation Quality and Performance”, Gupta et al 2021

• “From Motor Control to Team Play in Simulated Humanoid Football”, Liu et al 2021 (curriculum training of a single NN from raw humanoid control to coordinated team-wide soccer strategy; neat to compare with Hill et al 2020 in terms of agent abilities)

• “Wav2vec-U: Unsupervised Speech Recognition”, Baevski et al 2021

• “Anthropic” public-benefit-corp/startup launched (founded by the Amodeis; $124M investment for scaling “reliable and steerable AI systems”); “Cooperative AI Foundation” (CAIF) launched

• “MLP-Mixer: An all-MLP Architecture for Vision”, Tolstikhin et al 2021 (another FC paper removing even more inductive biases—ponies are all you need: “Mixer improves more rapidly with data than ResNets, or even ViT, and the gap between large scale Mixer and ViT models shrinks until the performance is matched on the entire dataset…” The Bitter Lesson truly is the single bitterest lesson in ML, isn’t it? The more people tweet about how MLP-Mixer is overhyped because is −X% worse than the ultra-hand-optimized baseline or requires Y× more FLOPS, the more they demonstrate precisely why this sort of research is so important! And showing, incidentally, that Transformers are still under-researched if such a fundamental fact could have been missed for so long.)

• “Data-Efficient Language-Supervised Zero-Shot Learning with Self-Distillation”, Cheng et al 2021 (CLIP-like performance scaled down to n = 3m using soft labels generated by a Conceptual Captions-pretrained model)

• “SR3: Image Super-Resolution via Iterative Refinement”, Saharia et al 2021; “Diffusion Models Beat GANs on Image Synthesis”, Dhariwal & Nichol2021 (DDPM¹ finally surpass BigGAN-deep on ImageNet 512px images at similar compute-cost, as expected from their good scaling); “Cascaded Diffusion Models for High Fidelity Image Generation”, Ho et al 2021

• “Learning to summarize from human feedback”, Stiennon et al 2020

• “Grokking: Generalization Beyond Overfitting On Small Algorithmic Data Sets”, Power et al 2021 (discussion; new scaling effect, ‘grokking’: sudden perfect generalization emerging many epochs after training-set overfitting on algorithmic tasks when training in flat shallow loss landscapes); “Knowledge distillation: A good teacher is patient and consistent”, Beyer et al 2021 (training much smaller models merely requires hundreds of thousands or millions of epochs)

• “Scaling End-to-End Models for Large-Scale Multilingual ASR”, Li et al 2021

• “The Shape of Learning Curves: a Review”, Viering & Loog2021

• “Reward is enough”, Silver et al 2021 (a DRL manifesto: reward losses enough at scale of compute/parameters/tasks to induce all important capabilities like memory/exploration/generalization/imitation/reasoning)

• Inverse Scaling Prize: $100k prize for finding tasks that cause worse perf in large language models (deadline: 2022-08-27)

• Scaling Down: lazy: a tool for running processes in idle time (how to train on a GPU without destroying your GUI’s usability! lazy pauses runs briefly while you interact with your desktop, letting you do months-long runs without going crazy or resorting to Colab etc. This enables hobbyists to go after previously-infeasible model sizes); EleutherAI releases a 6b-parameter GPT-3 model, GPT-J (are you still using GPT-2/GPT-Neo? upgrade!); “Aggregating Nested Transformers”, Zhang et al 2021/“Less is More: Pay Less Attention in Vision Transformers”, Pan et al 2021

• “ByT5: Towards a token-free future with pre-trained byte-to-byte models”, Xue et al 2021 (character models—not just feasible but desirable; we’ll get our rhyming & pun-making language models yet!)

• “Machine Learning Attacks Against the Asirra CAPTCHA”, Golle 2008_16ya (a look back on a decade of CV progress: months of work for 80% cat vs dog with SVM ensembles in 2008_16ya; 5min in Fast.ai for 99% accuracy in 2018; for even more perspective, Cireşan 2012)

Genetics

Everything Is Heritable:

• “Bi-ancestral depression GWAS in the Million Veteran Program and meta-analysis in >1.2 million individuals highlight new therapeutic directions”, Levey et al 2021

• “The complete sequence of a human genome”, Nurk et al 2021 (media)

  • “A complete reference genome improves analysis of human genetic variation”, Aganezov et al 2021

• “Using DNA to predict intelligence”, von Stumm & Plomin2021 (review)

• “Long read sequencing of 3,622 Icelanders provides insight into the role of structural variants in human diseases and other traits”, Beyter et al 2021

• “Rapid Sequencing–Based Diagnosis of Thiamine Metabolism Dysfunction Syndrome” (sequence everyone!)

Engineering:

• “Sense codon reassignment enables viral resistance and encoded polymer synthesis”, Robertson et al 2021 (“ultra-safe cells”: synthesizing an entire E. coli genome with swapped codons for complete viral immunity)

• “In vivo CRISPR base editing of PCSK9 durably lowers cholesterol in primates”, Musunuru et al 2021

• Optogenetics: “Partial recovery of visual function in a blind patient after optogenetic therapy”, Sahel et al 2021 (media); “Wireless multilateral devices for optogenetic studies of individual and social behaviors”, Yang et al 2021 (media)

• “Retron Library Recombineering (RLR): High-throughput functional variant screens via in vivo production of single-stranded DNA”, Schubert et al 2021

• “First genetically modified Oxitec mosquitoes released in the United States”

• “Genomic characterization of world’s longest selection experiment in mouse reveals the complexity of polygenic traits”, Palma-Vera et al 2021

• “Surrogate broodstock to enhance biotechnology research and applications in aquaculture”, Jin et al 2021

• “Utility of polygenic embryo screening for disease depends on the selection strategy”, Lencz et al 2021

• “Limit on lab-grown human embryos dropped by stem-cell body: The International Society for Stem Cell Research relaxed the famous 14-day rule on culturing human embryos in its latest research guidelines”

• “Useful Mutants, Bred With Radiation” (on atomic gardening)

Statistics/Meta-Science

• “Correlated Failures” in HDDs/SSDs

• “How a Publicity Blitz Created The Myth of Subliminal Advertising”, Rogers 1992_32ya (the famous movie-theater/popcorn-sales experiment never happened—subliminal advertising was the Cambridge Analytica of the 1950s)

Politics/Religion

• “Clarifying the Structure and Nature of Left-Wing Authoritarianism (LWA)”, Costello et al 2021

• “Book Review: The Decline and Fall of the Roman Empire” (excerpts)

Psychology/Biology

• “A connectomic study of a petascale fragment of human cerebral cortex”, Shapson-Coe et al 2021 (“…This “digital tissue” is a ~660,000× scale up of an earlier saturated reconstruction from a small region of mouse cortex, published in 2015 (Kasthuri et al 2015). Although this scaleup was difficult, it was not hundreds of thousands of times more difficult and took about the same amount of time as the previous data set (~4 years)…The rapid improvements over the past few years…argues that analyzing volumes that are even 3 orders of magnitude larger, such as an exascale whole mouse brain connectome, will likely be in reach within a decade.” See also “Accelerating progress in brain recording tech”.)

• “Neuroimaging evidence for a network sampling theory of individual differences in human intelligence test performance”, Soreq et al 2021; “The neural basis of intelligence in fine-grained cortical topographies”, Feilong et al 2021; “Predicting intelligence from brain gray matter volume”, Hilger et al 2020 (towards the mechanistic reification of g: per P-FIT, it is global efficiency/total cognitive resources which can be spent on learning & orchestrating specialized capabilities); if we consider recent human brain imaging studies, cross-species comparisons, and deep learning as converging, I would offer as a speculation the following:

  The Master Synthesis: intelligence is execution of small simplicity-weighted programs, best discovered by search over smooth loss landscapes like that of highly-overparameterized differentiable networks containing lottery-ticket subnetworks which are ensembled/averaged over, approaching Bayes-optimal reasoning in the limit (as nearest-neighbors-like high dimensional interpolation / memorization gives way to algorithmic generalization / interpolation on a more abstract level); this can be implemented by large numbers of similar neurons trained using any of the many approximations to backprop; general flexible behavior which cannot be feasibly specialized is guided by what is ‘left over’ from basic tasks (driving encephalization quotient & allometric scaling laws); human intelligence’s g is real but is the overall ‘pool’ of neural resources which derives from overall body integrity because the number of neurons, their density, their myelination, resistance to damage and infection etc, is causally downstream of all body and developmental systems, creating a huge mutational target; the brain regions specialize and differentiate, and their orchestration (or lack thereof) contributes to observed performance on tasks tapping into multiple specialized regions; as tasks rely on fewer regions or approach intrinsic ceiling, g ceases to be observable and task-specific influences matter most.

  • Why do larger animals need so much more neurons to control their body, when one would expect the hierarchical structure to be efficient?

    One possibility from an ANN perspective is the tradeoff between width & depth (wide vs deep models learn different things): wide shallow nets have low latency, but tend to be parameter-inefficient compared to deeper nets (perhaps because they lern more redundant but parallel representations?). Because larger animals live in the same world as smaller ones and still need to act with reasonable latency on the millisecond to second time scale, they presumably are forced towards wider nets, and away from a latency-unconstrained parameter or FLOPS-optimal architecture & scaling.

• “MDMA-assisted therapy for severe PTSD: a randomized, double-blind, placebo-controlled phase 3 study”, Mitchell et al 2021 (d = 0.9 over therapy); “Effects of Psilocybin-Assisted Therapy on Major Depressive Disorder”, Davis et al 2021

• “Why Animals Don’t Get Lost: Birds do it. Bees do it. Learning about the astounding navigational feats of wild creatures can teach us a lot about where we’re going” (on spectacular but still mysterious feats of animal navigation)

• “In The Future Of Collecting, Is Anyone Having Fun?” (on Bobblehead collectors)

• “Linking Brain Biology to Intellectual Endowment: A Review on the Associations of Human Intelligence With Neuroimaging Data”, Dizaji et al 2021

• “The Best And The Rest: Revisiting The Norm Of Normality Of Individual Performance”, O’Boyle & Aguinis 2012_12ya (performance is log-normal)

• “A conserved strategy for inducing appendage regeneration”, Abrams et al 2021 (slight regrowth of damaged mouse limbs by drinking sugar+amino-acid-supplemented water)

• “Know Your Amphetamines”, Scott Alexander

• “Feeling Small: Exploring the Tactile Perception Limits [of Humans]”, Skedung et al 2013

• “The Board Game of the Alpha Nerds: Before Risk, before Dungeons & Dragons, before Magic: The Gathering, there was Diplomacy” (WP; “I still don’t know whom I should have trusted, if anyone. All I know is that I felt stupid, stressed out, humiliated, and sad.”)

Technology

• “I walk the (beta-stability) line: How counting neutrons explains nuclear waste”

• “Making is Show Business now”, Alex Danco

• “Shop Class as Soulcraft: The case for the manual trades”, Crawford2006

• “Spintronics: Build mechanical circuits”, Kickstarter (followup to Turing Tumble)

Economics

• “RCTs to Scale: Comprehensive Evidence from 2 Nudge Units”, DellaVigna & Linos2020 (nudge effects overestimated by 6.2

• “No causal associations between childhood family income and subsequent psychiatric disorders, substance misuse and violent crime arrests: a nationwide Finnish study of >650,000 individuals and their siblings”, Sariaslan et al 2021; “Parental income and mental disorders in children and adolescents: prospective register-based study”, Kinge et al 2021

• “Everything You Might Want to Know about Whaling”, Matt Lakeman

• Exploding Nash Equilibrium For Trustless Trade

Fiction

• “Love Is the Plan the Plan Is Death”, James Tiptree Junior (WP)

Miscellaneous

• “The Strange Story of Dagobert, the Duck Tales Bandit: In the ’90s, a frustrated artist in Berlin went on a crime spree—building bombs, extorting high-end stores, and styling his persona after Scrooge McDuck. He soon became a German folk hero.” (WP; another reminder for Americans—odd as it may seem, Donald Duck is extremely popular overseas; see also the unknown-in-the-USA character John D. Rockerduck or beloved Scandinavian tradition From All of Us to All of You who 2020 airing set an all-time record of >4.5m viewers)

• List of atmospheric optical phenomena (How many would you recognize from a distance or plane? How many have you even heard of?)

• Baron Franz Nopcsa von Felső-Szilvás (noted geologist, paleontologist, anthropologist, homosexual, & skyjacker)

• Krishnacore