If you’re just tuning into the Not-So-Easy series, it’s about tasks that I think generative AI is likely to struggle with. In the previous chapter Embodied Cognition I covered the challenge of sensing and controlling a physical body in space. In this post, I’m covering Social Fictions: the made-up community rules humans use to navigate their social worlds. As highly social animals, we find social fictions are so intuitive that we often confuse them for facts.

Social fictions are not facts. They are multi-agent agreements that can be revised, interpreted, or ignored. They hold force only insofar as someone is willing to take the responsibility for enforcing them. Social fictions exist only as continuous, ever-evolving practice. And they’re held together by combination of personal relationships, and direct enforcement. To be better social actors, AI’s don’t need better skills, they need better friends.

Open Up in the Name of the Law!

What do you do if someone knocks on your door and demands to be let in?

Well, you don’t have to let them in. Unless they are a uniformed officer of the law and they have a search warrant, then you do. Unless the warrant is from a different jurisdiction, then you don’t have open the door. Unless they get a special exception, then you do have to open the door. Unless you win your case, then you don’t. Unless they win their appeal, then you do again. Unless the whole scene was taking place inside a play, then it’s fiction. Unless the play is given legal force through an act of Congress, then it’s all real again!

Can you imagine explaining that to a robot? It’s not just that social rules are a complex snarl of historical laws and norms. It’s that these social fictions are constructed. They can be anything we want them to be. We have an entire profession dedicated to the fact that it’s not always clear how our social fictions ground out in reality. Lawyers are people whose job is to negotiate what the law means in the real world. Sometimes they can do this by checking precedent, but often they have to generate their own interpretations.

A few years ago—before ChatGPT was publicly available—one of my students developed a “patent generator” for his capstone project. It was able to create plausible-looking patents for new technologies. I called up a patent attorney I had worked with at IBM research who was now working at the US patent office and I showed him some of the patents. He said that the AI-generated patents looked indistinguishable from the technical gibberish he had to read through on a daily basis, but also that he would never “accidentally” approve a fake patent.

When I asked him why, he said patent attorneys’ desks are already swamped. What’s in the documents is not really why a patent gets approved—it’s much more driven by the system of professional relationships. My attorney colleague said that his usual strategy is to ignore submissions unless another attorney he knows calls him up and badgers him to approve their client’s work.

So the law rests not just on a set of concrete rules—the sort of thing a machine could learn—but also on a web of specific relationships. Furthermore, the law is something like a best-case scenario for social fiction. It’s an area where if there isn’t exactly an objective truth at least there’s agreement that there should be consensus. Most social domains are fuzzier.

Fashion

Which of these is more appropriate to wear to a wedding?

The answer of course is both and neither. If you’re the bride—and it’s culturally appropriate—these are both lovely wedding day outfits. But if you’re not the bride, you probably will not be appreciated if you show up in either one!

There’s nothing intrinsically correct about wearing a wedding dress at all—you can also get married at the courthouse in jeans if you like.

When we train a neural network, we often think of it in terms of inferring a function from examples. But there isn’t a “wedding dress” function. There’s no objective answer to what a wedding dress looks like, not because there’s a lack of labeled data or a failure to correctly parse the concept of a wedding, but rather because each wedding dress expresses a different set of uniquely constructed cultural traditions. Each instance is the expression of a unique combination of factors: cultural, historical, personal, and aesthetic. The bride is trying to communicate something about herself by locating her individuality relative to the universe of possible brides.

Now of course, we could get AI to fake it. If we memorize a huge set of examples, it might seem to be drawing a novel example to an observer who has not seen as many examples as the model has. Imagine if you’d never seen an Indian wedding and the AI suggested a beautiful scarlet silk and gold ensemble. You’d be pretty excited at how “original” it seemed!

But even at best, “innovation by memorization” is brittle and time-bound. Culture does not stand still; it is constantly edited and reinvented. The most obvious example of which is language itself.

Language

This is the opening stanza of the famous epic Beowulf written in Old English:

Hwæt. We Gardena in geardagum,

þeodcyninga, þrym gefrunon,

hu ða æþelingas ellen fremedon.

The modern translation (from English to English) is: “So. The Spear-Danes in days gone by and the kings who ruled them had courage and greatness.”

The first word “Hwæt” is the subject of much controversy. “Hwæt” is the word an Old English speaker would use to get your attention. Some translations render “Hwæt” as “Hail!” or “Hear me!” Most modern translators prefer “Hey!” or “Listen!” Maria Dahvana Headley chose “Listen, Bro!”

These translations reflect different goals. “Hear me!” tends to emphasize the poem’s historical authority—it makes the poem sound like it comes from a classic, almost mythical past. “Bro!” underlines the fact that this is a poem about drunk guys boasting. Both the formal and the fratty versions are accurate. But neither is neutral. Different renderings of “Hwæt” allow translators to influence the audience’s perception of the original text. Even refusing to translate “Hwæt” only serves to reinforce the fact that English has changed so much in the last thousand years that our own language is no longer intelligible to us.

The Game’s Afoot!

The goals pursued by fashion and language are relatively benign. Brides and translators want to express certain identities and interpretations, but they aren’t actively adversarial. This is actually fairly rare. Language is not primarily for passing information around. It’s for passing information strategically. Most communication—as Wittgenstein described it—is like moves in a game. We use speech to make something happen. We can open a window either by using our hands, or by saying “Do you think it’s stuffy in here?”

There are some AI models of subtle social games. A good example is the deep reinforcement learning work in no-limit Texas Hold-em. Poker is a hard AI problem because there are so many different possible bets. If you’re playing for million-dollar pots, you can make a million different bets. What’s the right one?

But human pros aren’t just examining betting behavior. Here’s a breakdown of the information used by Tom Dwan to win the largest hand of poker ever televised. Dwan used a combination of game knowledge and the specific language used by the other players. He reasoned based on what his opponent said, what his opponent didn’t say, and drew conclusions about what his opponent was trying to get Dwan to do.

And in poker is sort of a best-case scenario. At least there are clear rules and hands that get revealed so you can find out for sure if someone is bluffing. Most social games are Calvinball. That is, they are metagames in which the players are aggressively negotiating the game’s rules on the fly. Creating contradictory or competitive sub-games is an ordinary move in a social meta-game.

When I’ve discussed the difficulty of modeling this social meta-game with AI folks, they often point to optimal strategies Ultimately they think that a sophisticated AI will be able to figure out some sort of “best” strategy that responds only to the underlying statistics. Sure, human players may be able to manipulate each other into making bad plays, but an “optimal” AI will just ignore all such manipulations. Then, because such an AI would always win, it would eventually outcompete the flighty, superstitious humans.

I think this is probably correct for most games with formal rules like poker. But I think it profoundly underestimates the importance of non-optimal, self-referential behavior! Many of our most trusted social roles depend on making bad bets.

Consider the Court Reporter

The court system is an adversarial system—each side’s attorney is an advocate. They push for a maximalist version of what their client wants. Even if the attorney believes that their client is in the wrong, it’s their job to try to get them as much as possible. That’s the job of an advocate.

But having parties invested in the outcome of an explicit competition doesn’t mean the truth goes out the window. To the contrary, an explicit competition actually helps make it more likely to get to the truth. The side that presents the argument best able to weather the other advocate’s attacks is less likely to have relied on secrets, biases, or shaky reasoning.

But while competition is useful for revealing the truth, we can’t simply let the attorney’s themselves decide who won: they would each claim victory and end up deadlocked. So the job of deciding who’s presented the most compelling argument has to rest with an external judge or jury.

This is a good enough system for resolving disputes, but it’s limited when communicating the result of the trial to the external world—especially if something goes wrong. If one of the parties is unhappy there’s no way to be sure if they’re telling the truth. We can’t interview the other side—they’ll just say they should win. We can’t interview the judge or jury, they’ll just give us the same decision as before.

Thus, there needs to be one more person in the courtroom. A fourth agent whose job is to act as an advocate for everyone who’s not in the room during the trial.

Enter the court reporter.

The court reporter is the answer to a balance problem with the legal system. The presence of appeals courts, case review, and the public at large creates the need for a reliable record. The presence of advocates at the trial mean that this record must be kept by an independent party.

This party must be trustworthy. So court reporter because they’ve pledges—on pain of prison and censure—to bear responsibility for the external record. They are disinterested in who wins the trial, but are deeply committed to accurately capturing what happens. The court reporter’s legitimacy comes from strength of their oath not their excellence in shorthand. What possible oath could a robot take? What possible punishment would it be meaningful for an algorithm to suffer? How could an AI make restitution?

Thus, court reporter is my go-to example of a job that is both trivial to automate and also will never be automated. An AI could certainly record what goes on in the court room—in fact most court reporters already supplement their notes with audio recordings and speech-to-text. But that’s not the job. The court reporter’s job isn’t really about writing down what happens. It’s about going to jail if you screw up.

Share

Social Responsibility

Ultimately, social interaction requires persuasion. Aristotle liked to break this persuasion into three elements: the rational appeal of logos, the emotional appeal of pathos, and the moral appeal of ethos.

In my previous Hard Problems for Robots post, I pointed out that teaching robots embodied skills, while difficult, is ultimately a tractable engineering problem. I want to draw a parallel here. I think that that teaching robots logos and pathos is tricky, but ultimately doable. Hallucinations degrade ChatGPT’s logos, but it’s reasoning can be improved. And the persistent Hollywood trope that robots can’t understand human pathos, understanding what people are feeling is a learnable, if subtle, skill.

But as I hope the examples I’ve given illustrate the thornier problem of ethos. For laws, patents, fashion, language, games, and oaths the power of your social performance depends on the strength of your connection to the social group you’re performing for. It’s shared values, identity, and commitments that matter. Your audience listens because they have robust ties to you, not because of anything you’re saying. There’s a reason towns tend to choose mayors who have multi-generational ties to their city over competent outsiders.

These long-term relationships are not a technical problem to solve. Social ethos comes from shared history. The major reason AI feels so socially icky isn’t that it can’t understand humans or say the right thing. It’s the same reason that a stranger showing up at your family reunion is weird. They aren’t part of your social circle.

If a robot wanted to be trusted, it would would need to spend years with you Bicentennial Man style. It would have to actually come to family reunions, hang out, engage in repeated cooperation, and so on. You’d learn—over literal decades—that you could trust this particular individual example of a robot. Your confidence wouldn’t be rooted in the robot’s capabilities, but rather in your own repeated personal experiences. Ultimately, that’s why you can’t automate ethos. Because the quickest way to build it is to build it as slowly as possible.

In my last AI critique, I discussed the sociological objections to “If Anyone Builds it, Everyone Dies” (IABiED) by Eliezer Yudkowsky and Nate Soares. I argued that the book’s core concerns—dodgy alignment, black-boxing, and malinvestment—were reasonable. However, these concerns were wrapped up in a larger, almost theological framework that seemed divorced from reality. Rather than treating AI as a totally novel, science-fiction concept, I suggested that it had a lot more in common with the super-human intellects we already navigate on a daily basis: corporations, governments, institutions, etc.

In this second piece I want to explore the cognitive science side of the book. Unlike the book’s sociological argument—where the pieces were good but the whole was rotten—the cognitive science side has a plausible overarching idea that is composed almost entirely of bad facts.

Compound interest intelligence

The central, organizing cognitive science idea presented in IABiED is an “intelligence runaway.” The authors contend that human intelligence seemingly outstrips that of animals by order of magnitude. They draw the reasonable inference that this suggests that human intelligence is the result of exponential growth.

Compound interest is a familiar example of this kind of runaway. When someone invests money, at first, their returns are heavily dependent on the size of the initial investment. Yet over time, as the returns are reinvested, each year’s gain compounds upon the previous. If the investment makes 10% a year, the first year returns $10 on a $100 investment, while the second year returns $11—because the money made in the first year is still in the pool. This is a runaway process—eventually the accelerating compound interest dominates the initial investment. Even a modest initial investment can become a huge amount of money. There’s even a famous case, Thelluson v. Woodford, that limits perpetual trusts to prevent them from eating the market.

Yudkowsky and Soares argue that intelligence is a property that can compound upon itself, and that the large variations we see among living species in intelligence are evidence of such a compounding process. Their worry is that if AI becomes a beneficiary of compounding it could result in a runaway “super intelligence” with cognitive abilities so greatly beyond our own that it makes us look like bugs. The authors aren’t sure if this compounding will be a result of self-evolving code, bigger training data sets, or greater processing power, but whatever the path, they worry about the outcome.

I think this is decent reasoning. If you think humans are exponentially smarter than animals, it’s reasonable to go looking for an exponential growth process. Now, as I covered in the previous piece, I think this is a category error. I think “human intelligence” has little to do with individual brain power. I think the large technological differences between humans and animals are a consequence of the cultural aggregation of collective intelligence. But if I’m wrong, then an intelligence runaway is a plausible global organizing concern.

To know if this intelligence runaway holds water though, we’ll need to unpack it to see if it’s made out of plausible pieces. Here are the core bits of the claim as near as I can tell:

1. Intelligence is substance-like and homogenous. This is necessary for quantitative comparisons between brains with “more” or “less” intelligence to be meaningful.

2. Intelligence compounds upon itself. Through evolution, learning, or simply the provision of greater compute resources.

3. There should be a clear hierarchy of smartness. If variation in intelligence among species is due to compounding, it should be possible to meaningfully order them by degree of intelligence attained.

4. Intelligence is world-changing. Mental stuff can be reliably translated into physical changes to the world.

These are testable claims. If the authors are correct in these assertions, we should not observe heterogenous intelligence: sophisticated in one area, weak in another. More intelligence should always be better—the compounding process should never make a brain dumber. Species should obey a clear intelligence hierarchy—with more sophisticated brains invariably resulting in more sophisticated behavior across categories. Finally, intelligent creatures should be able to more reliably instantiate their ideas as physical changes in their environment than dumb creatures.

I think I can give counter examples for all of these.

Undifferentiated Brain Goop

The first of these claims—that intelligence acts like a substance where one can have “more” or “less”—doesn’t pass even cursory scrutiny. Even sophisticated, state-of-the-art generative models sometimes struggle with easy questions such as “how many times does the letter R appear in the word strawberry?” or “If the car wash is 100m away, should I walk or drive?” These are questions most kindergarteners can answer. If these models are so smart, why do they have these weird gaps?

And this is already being quite charitable: these are text-based questions. Try asking a large language model to make you a cup of tea! All the training on text in the world won’t help you to open the cabinet and get out a cup. Oddly, you’ll have better luck with a Nespresso—a robot that is not entirely dumb but isn’t perceived as a nascent super intelligence. Real applications of intelligence require specific skills that respond to the affordances and informational structure of the specific problem they are trying to solve.

These kinds of tests are good ways to reveal the narrow optimization that models are actually carrying out in their learning. It’s tempting to infer that AI systems are broadly intelligent because humans are very good at generalizing from narrow demonstrations of competence. If a model can write a coherent paragraph, and we infer that it has all of the skills a human child would learn on their way to writing a coherent paragraph. If it can generates realistic images, we infer some visual comprehension. But AI’s intelligence is far more heterogenous and specific: some skills are highly tuned while others are simply missing.

IABiED leverages this misunderstanding when it describes both ChatGPT and self-driving cars as examples of AI. These are totally different technologies ChatGPT is a digital media generator trained on reddit and Instagram. Waymo uses Deepmind’s Genie 3 world model along with a number of physics simulators, cameras, LIDAR, online sensor feedback, and a constantly updated, proprietary maps of San Francisco. This specialization is necessary: the Waymos currently in New York still have human drivers because the city map just isn’t as complete. You can certainly improve each of these technologies with more engineering, training, and resources. But they need different resources. You wouldn’t get better performance from Genie 3 by training it on a bunch of reddit posts. If you plugged ChatGPT into a robotaxi—How would you even do that? What would that mean? Where is the plug?—it wouldn’t become more intelligent. It would just stop working.

And human cognition is not especially unified either. Neuroscientists generally think of brains as coalitions of semi-specialized processors with different strengths, constraints, and priorities. Depending on the task, adding more processing can have no effect or can even impair performance. There are lesion and hormone studies showing precisely these kinds of dissociations.

It’s almost as if intelligence is about bringing the specific, appropriate information to mind, and using it to guide specific, appropriate behaviors. But if that’s true, then intelligence isn’t some sort of general purpose brain goop. It can’t be abstracted away from specific contexts and it’s meaningless to make simplistic quantitative comparisons.

Sometimes learning makes brains dumber

The context-dependence of intelligent behavior also has consequences for training. One of the core assumptions of IABiED is that processes that enhance AI performance—evolution, training, more compute, etc.—will result in compounding gains. As someone who’s spent the last 20 years or so training neural networks, this has not been my experience. More training usually makes networks dumber.

One pretty common outcome of training a neural network looks like this:

This model learns the training set really well, but the validation set—the one we are testing its performance on—bounces around and doesn’t learn much. It’s training! It’s computing gradients and spending compute cycles, but the model isn’t really improving it’s performance on the test set. We’re not getting smarter, we’re just creating waste heat.

Another common outcome is overfitting:

When we train a model, we show it part of the data, and hide part of it. We test it’s performance on the part we hide. The model here is getting better at predicting the data we show it, but after some initial improvement, the training is actually starting to reduce performance on the hidden test set.

What’s happening here is that the model is just memorizing the individual data examples without finding any general structure or patterns that it can use to predict out-of-sample examples. Rather than learning statistical patterns—like what visual features differentiate mammals and fish—the model just memorizes that when you see image #3 you press the button labeled “fish.” It knows nothing about fish despite a perfect score on the test set. If the samples are unbalanced—say there are a million fish and ten cats—then the model can do extremely well by just guessing “fish” every time. An overfit model is exactly like a student who has memorized the test answer key but never studied—they’ll fail all tests except exactly the one they memorized.

But there are even worse scenarios! Here’s a case of catastrophic interference:

This model was first trained to add 1 to any number you gave it as an input. Then it was trained to add 2 to any input. It promptly forgot how to add 1.

Some of my dissertation actually digs deep into this phenomena. What I point out about it is that there’s actually no mathematical reason to expect the model to remember n+1 when you train it on n+2. When you trained it on n+1, it started with randomly wired network. When you train it on n+2 the training algorithm doesn’t “know” that you’d previously trained on n+1, it assumes the network is randomly wired. It’s not like the model has a memory, or eyes, or reason to be suspicious of the machine learning scientist.

Generally the way you overcome catastrophic interference is with “interleaved learning” — in other words, you need to train on n+1 and n+2 at the same time. But there’s a cost: it takes about twice as long. The model has to learn a set of weights that gets all of the answers right at the same time—and there are twice as many examples.

The experiments I did in my thesis were aimed at getting at why. One experiment that I think did a good job of explaining the phenomena showed that if you trained the model on a random problem—like predicting random pairs of binary digits—you’d get catastrophic interference. The old patterns would be wiped out by new ones. But, if you started with a structured problem —like converting binary digits to Gray code—the model would actually start to learn patterns it hadn’t even seen yet!

This is because what the model was actually learning was a rule rather than just memorizing matching pairs. If the problem is structured, it’s possible (with a good set of examples) to learn the rule that generates all the patterns. This is a much more compact representation, so it can be learned more quickly, and it generalizes to new circumstances.

The problem with the real world is that it’s a mixture of structured and random. If you know my name is Patrick you can infer that my family is more likely to be ethnically Scottish than they are to be Bangladeshi. But knowing my name is Patrick probably won’t help you to predict the last four digits of my phone number—because those were assigned randomly.

And this is a big part of why we don’t see “intelligence runaways.” Because intelligence is about applying the right information to the right context in the right way, you always need a mixture of general rules and specific facts. Learning this mixture involves inferring the rules from examples, and also keeping track of all the random bits that don’t fit with any of your available rules. This is an unavoidable trade-off. A rule is a choice to neglect exceptions. Either you hold onto all of the individual bits of data, or you assume they model some common structure.

And more compute resources won’t dig you out of this. If your plan is to hold onto all the bits and relearning everything you’ve ever experienced each time you see a new example then you’ve got yourself a combinatorial explosion. These are mathematically intractable. Even for pretty small interleaved relearning you’re looking at an impossible amount of compute power. It’s not a “build a more data centers” problem, it’s a “convert every atom in the universe into a super computer and have it run for longer than the current universe has existed” kind of problem. Additional accuracy is crazy costly. Try to be perfect and you’ll end up getting slower. Try to be fast and you’ll need to accept being vague. You pays your money and you get your choice.

So more training or resources aren’t always going to enhance your intelligence. But what about evolution? Do we see a clear stepladder of smartness in animal species? Well, in a word: no.

Animals are pretty smart actually

One of the most useful things you can learn as a neuroscientist is how smart bugs are. Arthropods are all around us all the time. It’s tempting to look at a fly banging its head against a screen and think that these are pretty dumb organisms. But this is not the case. Here are some fun examples:

Ants will explore their environments in what appears to be a random walk until they find some food. Then they will turn and walk directly back to the nest in a straight line, so they are obviously keeping track of the direction and distance to home even as they explore complex routes. They’re pretty robust to being moved as well. The only thing that’s sure to screw them up is strapping tiny stilts to their legs—yes someone did this. Yes, neuroentomologists are odd people.

Honeybees famously dance to signal the direct and distance to nectar-containing flowers. More critically, this is the only symbolic language in the animal kingdom: the direction, distance, and flower information is not encoded in the position of the bee-it’s encoded in the frequency of the dance and the number of passes. Even primates haven’t mastered this level of abstraction.

And as we mentioned on the podcast: jumping spiders are really smart! Like, about as smart as a cat smart. They eat other spiders and engage in some absolutely fiendishly clever hunting behaviors. Some species of jumping spider will put a leg on a prey-species web and wiggle it in a pattern that mimics an approaching mate. But they only do this if their prey species is receptive to sexy spider times! Otherwise they’ll simulate a snared insect, or damage to the web. Or if none of these traps are likely to work, they’ll just lower themselves from above Tom Cruise in mission impossible style. Is this theory of mind? Are jumping spiders looking at their prey and thinking about what is going on in it’s little spidery head? Put your best guess in the comments.

All three of these These critters’ brains are all far older evolutionarily than any vertebrates. Arthropod brains are pretty simple: a few tens of thousands of neurons. They are not anywhere near the “smartest” animals. It’s probably pretty reasonable to argue that a dolphin is “exponentially” smarter than a bee. But these little bugs are able to do many things that our most advanced AIs cannot do. Even if you think they’re robotic, they are immensely sophisticated robots with hundreds of thousands of embedded sensors, thousands of sophisticated motors, conformable grasping hairs etc.

And this makes fitting them into a simple “intelligence hierarchy” a sucker’s game. Is ChatGPT dumber than a jumping spider? It certainly doesn’t know how to wiggle a web just-so. It doesn’t know much about how to prey on spiders. Neither do most humans. One of the measures of “general intelligence” pushed by IABiED is “when AI can outperform every human expert.” But what about animal experts? There is a much larger scope of intelligent behavior than the book seems prepared to discuss.

This is a necessary complexification because within the narrow domain of their ecological niche, many animals outperform both humans and our sophisticated technology. Much is made through the book of human processing being “slow” relative to AI, but so long as you pick the right task, humans are pretty slow relative to animals too. While human movies run at 23 frames per second, pigeons would see that as a slide show, they’d need about 150 frames per second before they started to perceive smooth movement between frames because they have much faster visual processing. Bats and toothed whales navigate in complete darkness using ultrasound because they can detect sub-microsecond differences in the arrival of sound waves. The modern Tokyo metro took humans about 30 years to layout. A literally brainless slime mold solved for the layout in about 26 hours. Are these things ‘smarter’ than us?

What’s more, much of cognitive science is devoted to understanding cognitive faculties that don’t neatly fit into a human individual. Language is an ever-evolving cultural construct that no single human has command of. Economies are distributed processors that involve multiple agents solving problems no single agent could solve alone. Are these polyencephelic, trans-cultural constructs super intelligent? That’s a weird question. Sometimes they do things no individual human is dumb enough to do. The notion of a clear hierarchy of intelligence just does not appear in cognitive science, humans are not clearly more intelligent than animals, multi-agent systems, are not clearly more intelligent than humans.

This holds at the level of individual neurons too. IABiED argues that silicon transistors are orders of magnitude faster, more reliable, longer-lasting, and cheap than neurons. They also make some unfavorable comparisons at the brain level between the number of neurons in the brain, the number weights-per-neuron, and the number of processes-per-second to argue that data centers are close to producing super-brain-level processing power.

These kinds of comparisons are a red flag for neuroscientists. It’s deeply misleading to compare neurons and transistors. It’s a bit like comparing a single bit of processing on a computer chip to an entire core and concluding that the core is “slower.” It’s slower because the neuron is doing far more complicated processing. Transistors look good if you’re measuring operations-per-second: transistors perform individual bit-wise operators billions of times per second while Neurons fire action potentials only about 1000 times per second. But if you zoom in on neurons, you’ll find that each one has thousands of synapses (where the actual processing takes place), each synapse has hundreds or thousands of membrane receptors (the things that actually measure the incoming signal), and that the kinematics of each receptor allows them to “count” tens to hundreds of neurotransmitter binding events per second. If compare a single transistor bit flip to an entire neuron firing, you’ll find that the bit flip is a much faster process. But if you compare passing floating point numbers (aka “FLOPS” the standard unit for measuring processing power) then each neuron is producing 10-100 billion FLOPs while each transistor is only producing about 15 thousand FLOPs.

Does this mean it’s correct to say a neuron is thousands of times more powerful than transistors? Who knows! There’s so much complexity: how are error correcting codes implemented? What kind of processing are we doing? How is threading and coordinated clocking of the system handled? Neurons and transistors are apples and oranges. Making a direct comparison between them is always going to be misleading. It’s appropriate to be super suspicious of any kind of numerical comparison between biology (where numbers vary across orders of magnitude), and technology (where the designer chooses precise values).

Rather than finding a clear hierarchy with clear “stages” of intelligence that seem to be the result of a single clear compounding curve at every scale there’s a higgledy-piggledy mess of parts of unclear provenance or superiority. This makes a lot of sense if “intelligence” is building little bespoke bits to respond to specific problems and balance unique trade-offs that emerge in a complex world.

And speaking of a complex physical world…

Very Smart and Useless

The book’s final major logical leap is to assert that intelligence can always be translated into action. It assumes that coming up with the idea is the hard part, and that carrying out ideas is trivial. This leads it to worry that as AI gets smarter it will become more competent in its ability to shape reality.

As a smart person who accomplishes very little, I feel that I have some salient personal experience here.

In my last article’s discussion of AlphaFold’s protein folding breakthrough, I described some of the factors that made it a tractable problem for machine learning. I’m familiar with protein folding because I worked for a while in a neuropharmacology lab that did drug discovery. Knowing the particular configuration of proteins that made up the receptors we were designing drugs to target was super useful for us. But even though we were working before AlphaFold, we already knew what those receptors looked like—that’s why we were able to do drug discovery with them. Protein folding wasn’t a big blocker in the lab. Even knowing the shape of the target the professor still had to design molecules that would get jammed in there. The postdoc had to synthesize them. One grad student had to breed frogs and surgically extract frog eggs. Another had to clone the human RNA coding for the receptor. And I, the lowly lab tech, had to inject that RNA into the frog eggs, dump the candidate molecules onto them, and then dump radio-labeled glutamate on the eggs to see whether the drug was doing anything. Then I’d pass those results back up the chain to be analyzed and written up.

The lab did this every day for years! If AlphaFold had given us more proteins to target we wouldn’t have had the time to do it. We were at 100% capacity just working on the one we had. And none of what we did was amenable to AI. Have you heard of any break throughs in AI frog surgery?

That’s not to say that this process couldn’t be automated, some of the steps I’ve described have been. But that’s not due to intelligence per se, it’s engineering. The trick is to actually get the damn thing working. It involves physical construction, iterative design, and trial and error.

IABiED often waves away these practical constraints. At one point the book asks:

“What is the most efficient solar panel currently on Earth?”

The big gotcha is that the answer is “grass”: a self-assembling, self-replicating solar collector. Setting aside for a moment that “algae” is clearly a better answer, equating grass and solar panels is bonkers. Almost all of the energy captured by a blade of grass goes into making the grass. A grass cell’s chloroplasts generate tiny voltage gradients which they use to make sugars. You don’t see very much grass hooked up to server farms. IABiED suggests that it’s because grass wasn’t dreamed up by super intelligent AI. I suspect it’s probably because there’s little surplus transmissible power. Grass does support higher energy density organisms: they scoop up tons of it and break it down to capture the stored energy. We call these organisms “cows.”

The authors of the book seem to suspend thermodynamics when it’s inconvenient. The AI can invent magic self-replicating solar grass that somehow self-replicates for free and also provide surplus gigawatts of electricity. But why would the AI need to? In this fairy tale the AI already had all the energy it needed to invent magic grass!

Another example from the book involves the AI creating a deadly super-virus. This plague is supposed to A) have a long incubation period, B) spread quickly, and C) be reliably lethal. A virologist can tell you that this is a “pick two” scenario. If a virus’s genetic material is just kinda hanging in the host cells chilling it’s not actively hacking those cells to make a ton of itself. It won’t be especially infectious or lethal. Conversely, if a virus was so deadly that it killed you on the spot, it also wouldn’t be highly infections—the body it relies on to make the virus is instantly dead. There’s a trade-off between infectiousness and lethality due to how viruses self-replicate. Covid-19 was notable because the disease was actually pretty close to a worst-case scenario—at an almost optimal point on the infectiousness/lethality curve.

But SARS-CoV-2 didn’t stay there! It rapidly evolved to be less lethal. Viruses don’t “want” to kill their hosts, they need their hosts! The most successful viruses are things like papillomaviruses or herpes simplex-1 that just kind of ride around in everybody all the time causing as little trouble as possible. One of my virology professors said “the goal of every virus is to become part of its host’s genome.”

As in the grass example, the pragmatics of evolutionary virology are suspended for the super-intelligent AI. It invents things that work on the first time without tedious experimentation or physical constraints. It does this for novel things where no training data exists. It makes dramatic extrapolations from the things we observe in real life. It does not face unintended consequences for it’s actions, the things it makes behave in completely predictable ways. It’s not so much intelligent as “magic.”

Vibes-based Scholarship

The use of magical thinking is a habit in IABiED. It repeatedly treats implementation problems as if they were problems of imagination. If the AI can dream it, it can happen.

Arthur C. Clarke had three famous “laws” of prognostication:

1. When a distinguished but elderly scientist states that something is possible, he is almost certainly right. When he states that something is impossible, he is very probably wrong.

2. The only way of discovering the limits of the possible is to venture a little way past them into the impossible.

3. Any sufficiently advanced technology is indistinguishable from magic.
   

IABiED often feels like a book that attempts to apply these three laws to AI. It assumes that the best-informed scientists on the topic are wrong. It assumes that anything past the limits of what is known is therefore possible. It believes that sufficiently advanced AI can do magic. In so doing, it forgets Isaac Asimov’s corollary to Clarke:

When, however, the lay public rallies round an idea that is denounced by distinguished but elderly scientists and supports that idea with great fervor and emotion – the distinguished but elderly scientists are then, after all, probably right.^[15]

This book contains a lot of fervor and emotion. It is remarkably uninterested in educating it’s reader about AI, cognition, biology, physics—indeed any science whatsoever. Its reasoning is poor and sometimes contradictory. It is dismissive of complexity and fact. It is under-researched, and anti-intellectual. It thumbs its nose at scholarship.

As I read through the examples the authors provided I started to form a hypothesis as to why. There are very few published journal articles in the book, but mysteriously this is one of them;

Pavličev, M., Romero, R., & Mitteroecker, P. (2020). Evolution of the human pelvis and obstructed labor: new explanations of an old obstetrical dilemma. American journal of obstetrics and gynecology, 222(1), 3-16.

What is an article about the evolution of the human pelvis doing in a book on super intelligence? The citation appears in an argument about the evolutionary constraints on human brains. Because human brains have to pass through a human pelvis during birth, there is a trade off between fetal brain size and childbirth survival rates. This limits the volume of the brain tissue in neonates and slows down the evolution of intelligence. The book is making a point about why increases in artificial intelligence might be more sudden than increases in biological intelligence. It is not an argument entirely without merit.

Yet this citation is entirely without merit, because the paper in question is presenting evidence against the co-evolutionary bottleneck hypothesis. This paper is about how the human pelvis shape cannot be solely explained by reference to infant brain size. This is why the paper’s title promises “new explanations.” The co-evolutionary bottleneck is an old hypothesis the paper is arguing against. In other words, Yudkowsky & Soares present an article as supporting evidence for a hypothesis the article was specifically written to debunk. If the authors read this paper, they did not understand it. Perhaps they simply asked an LLM to give them a related citation and uncritically included what it returned.

Intelligence for Mortals

Ultimately, the real risk that I think a book like IABiED poses is that it encourages panic an ignorance as a response to a complicated future.

The book begins by saying that intelligence has no single clear definition. But rather than working toward one, it spends its pages on vibes-based speculation about what a nefarious superintelligence might do. It waves aside facts, details, specifics, constraints, and complexities in favor of a ghost story.

Ironically, this gives us a useful definition of intelligence by contradiction. Intelligence is the capacity to brings precisely the right facts and details to bear. It is specific to the circumstances at hand. It studiously accounts for the particular forces that bear on a problem and the intrinsic uncertainty that follows from that complexity. It knows what can’t happen as well as what can. Intelligence is opposite of regurgitating loose associations.

Whether you think the current wave of generative AI is an existential threat or a bunch of overheated puffery, surely you would prefer to approach the future with actually sharp wits rather than IABiED’s sophistry.

I try to avoid writing screeds. Life is too short to worry about someone being wrong on the internet. And it’s double-silly to give sensationalism bad press.

But a journal club I’m in recently took a break from dense comp-neuro papers to read If Anyone Builds It, Everyone Dies (IABiED) by Eliezer Yudkowsky and Nate Soares. Because of its popularity, the library took forever to get me the book, and I ended up missing the chance to chat about it with them. But I couldn’t let my notes on it go to waste, so I’m going to share them in this AI Critique.

There are two separate pieces to this Critique. This first one focuses on the philosophical framing of the Robopocalypse described in the book and argues that it’s a garden-variety, religious rapture story masquerading as serious thinking about AI. The second piece focuses more on the science claims of the book. Mild spoilers: the book’s “scholarship” is so poor that it ends up presenting a framework for intelligence which is both internally contradictory and completely unrecognizable to cognitive scientists. It is not an academic book. It’s collated blog articles blurbed by Grimes and Mark Ruffalo. It cites Elon Musk tweets more than published journal articles.1

So let’s begin with the philosophical framing: the core thesis of IABiED is that society should collectively lose sleep over “super intelligent AI”—whatever that means.

As a professor who teaches machine learning, I’ll spoil the punchline: I don’t think you individually, nor society collectively, should devote meaningful attention or resources toward combating an artificial “superintelligence” apocalypse. There are sensible worries and meaningful policies pursuant to new AI technologies, sure, but they are exactly the kind of boring worries and policies that won’t sell a book like IABiED.

Screeds about Artificial General Intelligence (AGI) and Artificial Superintelligence (ASI) are popular because they peddle the kind of formless anxiety that does well with internet algorithms. Fear sells. It’s a fool’s errand to try to help people feel braver and safer when panic gets better engagement. But lucky for you reader, I’m just that kind of fool.

The Shape of Fear

About two-thirds of IABiED is devoted to establishing a fairly simple chain of reasoning:

• AI has an “alignment” problem: It’s possible to pursue dumb goals in smart ways.

• AI has a “black box” problem: AI systems are trained statistically, so we don’t always know exactly what they’ve learned.

• AI has a “capital” problem: Huge resources are being poured into AI with limited oversight or democratic input.

• Therefore: large, unexpected, and possibly catastrophic things could emerge from AI systems.

This logic is mostly fine. The “alignment problem” points out that there’s a difference between knowing how to drive and driving to the airport. If we build robots capable of driving, they are also capable of running over pedestrians. The “black box problem” points out that statistical models can behave unpredictably outside the conditions we tested. When we optimize over millions of examples we can end up with discarding important details about why the outcome is desirable. The “capital problem” points out that markets routinely make enormous technological bets without much thoughtful coordination. There is a difference between “profitable” and “good.”

To all of these points I say: No shit, Sherlock.

I’m willing to endorse this chain of logic. In fact, I’m willing to go further than Yudkowsky & Soares: every one of these conditions has been true for millennia.

More than ten years ago I wrote a piece arguing that “the singularity” happened in Babylon shortly after the invention of writing. Every generation since humans started living in cities has experienced large, unforeseen technological and cultural shifts. Climate change was preceded by nuclear anxiety, which was preceded by industrialization, which was preceded by religious war, which was preceded by dynastic collapse, famine, plague, and invasion. Peace and stability are aspirational conditions, not default settings.

The AI apocalypse, like most things tech bros try to sell, is a worse version of an existing product.

Consider the alignment problem. The authors suggest we should be worried that AI and humans might “want different things.” But of course they will want different things. Identical twins don’t have identical goals. “Humanity” is a collective noun not a unitary entity with single purpose. Even when people broadly agree on a desired outcome, collective action remains difficult. Perfect alignment is unknown to history. Compromise and realpolitik have ever been the coin of the realm.

Likewise, the black box problem is hardly unique to AI. Humans also act under uncertainty. We routinely make decisions without fully understanding our own motivations or the long-term consequences of our actions. Even physics places limits on what can be simultaneously known. Nonetheless we must act. Simone de Beauvoir placed this tension between ambiguity and action at the center of ethics itself: uncertainty is not a bug in human reasoning, it is the condition under which moral action becomes meaningful.

And the capital problem? Well. Economics is the discipline devoted to explaining how markets allocate resources, and how it can go wrong. Of course AI investment is speculative. Of course firms exaggerate transformative potential to attract capital. This was also true of the tulip bubble.

The odd thing about IABiED is not that it identifies uncertainty, misalignment, or institutional failure. The odd thing is that it treats these as startling revelations unique to AI.

Superintelligences Already Exist

One of the stranger assumptions running through ASI discourse is the idea that “superintelligence” would represent an entirely new category of thing when humans already live inside systems vastly larger and more cognitively powerful than any individual person.

Corporations, states, militaries, religions, markets, bureaucracies, universities—these are all distributed information-processing systems. They aggregate collective memory, coordinate action, allocate resources, model environments, and pursue goals over timescales longer than individual human lives. They are, in a very literal sense, superhuman intelligences.

And they are often badly aligned. Democide is one of the leading causes of human death in history. Wars, famines, forced labor systems, and industrial disasters are all examples of large distributed systems pursuing goals in ways that destroy human lives. We already know what it looks like when opaque, massively resourced systems optimize toward goals that ordinary people cannot easily inspect or redirect.

For example in 2022, Russia’s military and political institutions had a values misalignment with Ukraine that resulted in a protracted struggle. The Russian side had greater resources, but Ukraine had trained more on local data. Both sides struggled to adapt their strategy to idiosyncratic feedback. Both sides deployed tremendous amounts of high-tech capital. The result is a straightforward catastrophe with mounting casualties, loss of civilian lives, and destroyed infrastructure.

Isn’t this exactly what the superintelligence worries are about?

Our world is already lousy with Hobbesian Leviathans: man-made gods and monsters. As Simone Weil puts it in The Poem of Force: “force is a relationship; the strong are so in relation to those weaker than themselves.” If we are to survive against powerful forces—whether from nature or institutions—we must create our own powerful institutions and tools that can push back. If our institutions are weaker than the forces imposed upon us—whether they come from natural forces or other institutions—they cannot defend us.

What’s frustrating about ASI discourse is that it usually ignores these already-existing forms of collective intelligence in favor of speculative future entities with magical powers. It treats institutional cognition as normal and hypothetical machine cognition as uniquely terrifying. But many of the dangers attributed to future AI—opacity, coordination failure, optimization toward harmful goals, runaway incentives—are already ordinary features of modern political and economic life.

The Robot Rapture

So if I broadly agree that AI systems can create real dangers, why do I find IABiED so irritating?

Because the book does not merely argue that AI might produce social disruption. It argues that society should orient its moral attention around a speculative robot apocalypse. The vagueness of this apocalypse is presented not as a weakness, but as evidence for its seriousness.

We are told that it’s impossible to know what the danger will look like and that it could happen in ways we can’t predict. It argues that this is a special, incomprehensible kind of danger. This is self-defeating logic.

Imagine someone wrote a book arguing that humanity should devote enormous resources toward preparing for an alien invasion. A reasonable response might be: “What evidence do we have that aliens are about to invade Earth?”

“Ah,” the author replies, “but that’s the thing. The aliens might travel faster than light. So we won’t be able to observe them until they arrive.”

This sounds scientific because it gestures vaguely toward relativity. But it actually undermines itself. If the aliens violate relativity, then relativity cannot provide a guide about what they can or cannot do. The argument uses science decoratively while discarding the constraints that make scientific reasoning useful in the first place.

Much ASI rhetoric works like this. Constraints are selectively suspended whenever they become inconvenient.

One reason this rhetoric is persuasive is that there have been rapid advances in machine learning. AlphaFold, for instance, made major progress on the protein folding problem and deservedly won a Nobel Prize. But discussions of systems like AlphaFold often quietly smuggle in much larger conclusions than the technology warrants.

AlphaFold worked because protein folding was a tightly constrained, well understood problem. We knew the amino acid sequences. We knew a great deal about molecular physics. We had clear ways of checking whether the model succeeded. AlphaFold does not “understand biology” in any way broader than protein folding. It is also meaningfully specialized for the protein folding domain—it is not the same kind of model as ChatGPT for instance.

But in ASI discourse, examples like AlphaFold are often treated as evidence that machine learning systems are generically becoming scientific superminds. The constraints that made the original success possible quietly disappear. A tool for approximating protein structure becomes evidence for imminent godlike cognition.

When claims are so loosely connected to empirical models of cognition, engineering, economics, or politics, they are difficult to refute. “Artificial Superintelligence” begins to function less like a scientific category and more like an apocalyptic belief structure: a coming revelation that will reorder the world so completely that ordinary reasoning no longer applies.

One of the recurring themes in IABiED is that the stakes are so high that nearly every other concern should become secondary. The implication is that ordinary political, scientific, and social questions are distractions from the singular existential problem of ASI.

But this framing often produces the opposite of effective action. When a book presents an infinitely large, formless, and inevitable catastrophe, it encourages stasis and flailing.

ASI is basically the Rapture for tech bros.

Share

Apocalypse Content

Fifteen years ago, Cracked Magazine published what is still probably the most plausible robot apocalypse scenario. The comedy sketch argued that robots wouldn’t kill humanity through laser warfare. Instead they would just pacify us with distracting entertainment, social bickering, and algorithmically generated nonsense. Why fight humanity when humanity loves watch AI-generated Will Smith eat spaghetti?

You will recognize this as the timeline we are currently living in.

Ironically, Yudkowsky and Soares briefly acknowledge in the book that AI could manipulate humans into destructive emotional states. But I don’t think they’ve appreciated that the call is coming from inside the house. IABiED—and most work coming out of the authors’ MIRI2 institute—actively distracts from the real problems associated with AI in favor of apocalyptic thinking.

My primary concern about If Anyone Builds It, Everyone Dies is not that it overestimates AI capability. It’s that it contributes to exactly the kind of diffuse technological panic that degrades public reasoning. The book treats the extremity of the ASI fantasy as evidence of its seriousness. It encourages readers to imagine themselves standing at the edge of an incomprehensibly deadly future beyond ordinary politics and ordinary history.

But politics and history have always ben full of opaque systems, runaway incentives, distributed cognition, and catastrophic failures. When Buddha said “life is a bumpy ride” he was not referring to a mystical robot eschatology.

There are plausible, mundane ways in which AI trained on internet data is is causing trouble. Social media recommendation systems often optimize for outrage, compulsion, paranoia, loneliness, and despair because those states produce more engagement. Internet search is getting worse. Generative media tools are putting a finger on the scale in close elections. There is even a short-but-growing list of deaths linked to chatbot relationships and other AI-mediated delusions. These are concrete social and institutional concerns that can be addressed with ordinary means: policy, activism, and education.

The future will almost certainly contain dangerous AI systems. They will join a world of dangerous governments, dangerous firms, dangerous ideologies, dangerous markets, dangerous media ecosystems, and dangerous people.

Don’t lose your head. Stay brave, lucid, trustworthy, and cooperative.

Subscribe now

Large attentional transformers like ChatGPT generate text that seems surprisingly human-like. On this episode Katie and Patrick ask which bits of the brain work like large language models, which bits AI still hasn’t figured out, and whether the difference can explain why robots can’t count the vowels in a strawberry.

Subscribe now

Neuroscientists don't believe in Artificial General Intelligence because they don't believe intelligence can be “general!” Instead they think the brain has multiple, specialized systems that work together to solve complex problems–more like a swarm of bees than a philosopher king. On this episode Katie and Patrick talk about the hive of brain systems living in your head and why this makes biological intelligence so robust and effective.

Share

Subscribe now

Hey! It’s summer time!

That means I’ve got a break from teaching, and I’m likely to start writing a few more posts than usual! I’ve really been enjoying working on the podcast with Katie, and that’s going to continue (we’ve got a few more great episodes already in production). But you can also look forward to much more frequent blog updates. Oh look—who dropped this subscriber button here?

Subscribe now

To kick off the summer series, I’ve planned a set of articles called Not-So-Easy AI Problems. In this series, I’m going to describe four problems I think AI is likely to struggle with. I think it’s a useful antidote to the constant barrage of AI-generated marketing slop from companies selling machines that generate AI marketing slop. Large, generative models like ChatGPT and Midjourney are great for churning out internet content, and this has created a big stir on the internet about AI. It’s sometimes easy to forget that out in the world of grass-touching there are a lot of skills that AI is still hopelessly bad at.

So without further ado:

Four Hard AI Problems

There are the four domains that I think are hard for current AI models:

• 1. Embodied cognition

  • Sensing and moving a physical body around in space.

• 2. Social constructs

  • Understanding the boundaries of concepts that are made real by collective agreement rather than physical reality.

• 3. Non-local information

  • Knowing the details of things you haven’t observed.

• 4. Combinatorics

  • Manipulating sets and systems.

I’ve put these roughly in order of difficulty, and in this AI critique, I’ll unpack the first one: embodied cognition.

Embodied cognition

The recent AI discourse is almost all a response to the commercialization of some deep-generative models that have actually been around for quite some time. Here’s Geoff Hinton talking about them back in 2007. These models didn’t have quite the spectacular performance they do now, but OpenAI released GPT2 in 2018, and I used a version of it to make interactive fiction back in 2019. Attentional transformers like ChatGPT are part of a larger technical march toward progressively better generators using clever techniques like RBMs, and GANs.

Importantly, these techniques attempt to capture the statistics of text and images. None of them attempt to model the physical states of affairs to which text and images refer. This is unintuitive: we don’t perceive media as the result of token prediction. We imagine light hitting a camera or a person telling a story. It seems so impossible that math can be used to write a poem that it’s easier to pretend what the bots are doing is analogous to human “writing.” It’s somewhat surprising to realize how closely real physical processes map onto the structure of text or photos.

But it’s even more interesting to look at where the AI disagrees with our intuitions.

Look Ma, No Hands!

Everyone knows that AI struggles to draw hands. But why is it so bad? There are tons of pictures of hands. But we humans don’t just understand our hands by looking at them, we understand our hands by using them.

Much of human (and animal, and robot) cognition is embodied. That is, the reasoning and representations used by our brains (or animal brains, or robot brains) refer to the particular sensory-motor assemblies we operate: aka, our bodies. Each body is different, and each body-moving algorithm needs to be adapted to the context of a specific body. If you try to apply a model trained to drive my body to a different body, the model will fail in predictable ways.

You probably experienced this yourself if you ever went through a major growth spurt. Suddenly you were very clumsy, as your body tried to reach out for things with arms slightly longer than your brain was used to. This is still a statistical problem! The motor transforms your brain uses to operate your muscles have parameters that can be estimated by exactly the same kind of models that currently work for text and images.

However, there are two major stumbling blocks. First, AI researchers are data-poor: the amount of motor data that has been collected and digitized—such as joint position, muscle load, and other proprioceptive information—is minuscule compared to digitized text and images. Second, the motors and sensors required to generate and collect such data are far harder to come by than keyboards and cameras.

Current robotic systems mostly work by running specific macros for reaching and movement, which are incredibly tightly adapted to the specific system. They rarely include feedback sensors beyond the load data coming out of their own servomotors, perhaps coupled with an occasional accelerometer or gyroscope.

This is beginning to change! There are a number of new avenues in robotics research using generative techniques that seem very promising. What’s more, roboticists are rapidly developing new sensors, materials, and tools that provide a better sense of their machines’ position in space. It’s also not entirely clear that this needs to be solved the same way humans do (with lots of sensors embedded in joints and muscle tissue). Many robots use clouds of infrared light to estimate limb position. This all means we’re going to see increasingly sophisticated robot movement—but that rollout is likely to follow the timescale of a big hardware project (ten to thirty years), rather than just downloading some software (one to five years).

And that should be pretty obvious. Do you have a smartphone? Cool, you can use ChatGPT. Do you have a robot? Well, you’ll need one before you can do AI robot stuff.

What a Body’s for

But there’s more to embodied cognition than just “moving your body around in space.” For humans and animals, the most important sensory-motor loops are actually chemical. Olfaction and gustation are the obvious ones, but even more critically important are the internally-facing sensors that measure glucose, blood pressure, hydration, electrolytes, circulating lipids, temperature, and so on. Most critical of all are nociceptors—pain detectors.

I had a colleague at UIUC who got an iCub humanoid robot that he hoped to teach to speak. He spent the first three years just trying to make sure it didn’t rip off it’s own very expensive arms.

As Rodney Books at MIT famously pointed out, all of these embodied sensory-motor loops are hard to build, and they’re evolutionary older than anything cognitive. It’s not an insurmountable challenge, but it took evolution billions of years to make pain sensors, and only a few hundred million to get from pain sensors to brains capable of calculus. The fact that AI has stumbled into competence at calculus and language is not actually proof of sophistication. We should be realistic about the difficulty of the embodied engineering challenge.

Again, I think we’re on our way! The technologies are actively being developed, and the important data and the tools for collecting it are being filled in. I think we are somewhere between a few and several decades from robots with sophisticated adaptable bodies. Although they unlikely to be humanoid. Why stick to one body plan? How many humanoid body plans do you see in nature? Better to be like JPL’s rad SnakeBot.

And this diversity of bodies is itself part of the problem. No matter how good your model is, it only works for the body it learned to control. Each different body requires a new dataset, or at least some fine-tuning. Tiny differences in friction in joints lead to extremely clumsy fingers. And unlike generating language where the bar is pretty low, human manual dexterity is an absolute marvel in the animal world. It took me 20 years to learn and I’m still clumsy.

Building working bodies is a harder problem than generating working brains. While there’s no major roadblock, this is a decades-long engineering problem that will eventually produce results that still have non-zero fine-tuning costs.

Boring Miracles

One of the major reasons that it’s hard to convince folks that rapid advances in software do not imply advances in hardware is that we live in a culture that tends to reward intellectual labor with more than physical labor. But markets reward rarity, not difficulty.

It might seem “harder” to prove the Collatz Conjecture that it is to put a box on a shelf, but stripped to the studs, a math proof is a matter of arranging a small number of ink molecules on a thin sheet of cellulose. Moving the box is a tremendously greater physical change. We just don’t value it because it’s a boring miracle that every kid can do.

Next time: It’s true because I say so.

In the next article in this series I’ll unpack the second hard problem: social constructs.

To the general public it's one of the most profound and enduring mysteries of the human psyche. To specialists it's something we don't worry about much. In this episode, Katie and Patrick explore the semantic gap between broad popular conceptions of consciousness and narrower understanding held by neuroscientists and philosophers.

But on the other hand, cognitive biases sure are a good way to look smarter than other people!

And in the media recommendations: We Stan Kim Kitsuragi!

Bonus content: I talk about my least popular paper! I probably should have titled it something more attention grabbing than “Underlying sources of cognitive-anatomical variation.”

Subscribe now

It’s been a minute since I published because a lot of my creative effort has gone into creating a new podcast with my colleague Dr. Katie McAllister. She’s been an exceptional cohost who covers many of the gaps in my own cognitive science background.

If you’re interested in the neuroscience side, you should definitely check us out on your favorite podcast platform.

Much recent AI-related anxiety is a consequence of AI getting better at generating media. Because ChatGPT and Midjourney excel at simulating the internet discourse on which they were trained it can seem to a severely online human as if these models are generally intelligent. In fact, these models are hopeless at even basic biological intelligence. Show me the ChatGPT powered grass-touching robot!

How to move your body.

Suppose you want to reach out and pick up a pen on your desk.

This is a fairly complex behavior. You’ll need to understand what a pen is, what a desk is, and which parts of your visual stimuli correspond to those different objects. You’ll need to distinguish the pen parts of your environment from the not-pen parts and measure the distance between your hand and the target. You’ll need to be sure to inhibit closely related behaviors, like picking up a cup, or writing with a pen. These processes are complex enough to basically involve the entire brain. However, we’re focused on the final common pathway so we can start in the prefrontal cortex (PfC):

The PfC is where this stew of competing concepts is being represented in an abstract way. Next we’ll need a system that can focus in on the specific target of grabbing a pen. These structures are mostly in the basal ganglia embedded deep in the brain:

These structures contain fibers that turn on or off other brain regions. Using a pretty clever system of go/no-go logic gates they can shut down or activate the different pen/not pen representations in the PfC. Once the right configuration is amplified and the wrong ones are inhibited, the pen-reaching behavior can be passed off to the motor regions of the brain:

The premotor, supplementary motor, and primary motor cortices work out the specific details of what motor program the body should run to actually carry out the reaching.

But we’re not actually done. In fact, we’re still about six or seven synapses from the hand! We know we’re reaching for a pen, and we even know we’re going to use our hand to do it, but we still need an entire additional brain to carry out the computations to make that reach.

Did you know you have a second, smaller brain in your brain that’s just for moving your body? It’s the cerebellum, which you might know as the little wrinkly guy hanging off the back of the brain:

The cerebellum does a number of things, but arguably it’s most important role is translating motor plans into actual algorithms for moving muscles. Think of this processing like doing a coordinate substitution. Your eye looks down and sees the pen at a certain angle relative to your head, but the direction your hand has to move in is at a different angle. So all the math your motor cortex did to figure out how to reach for the pen is wrong unless you have a hand growing out of your eyeball. The cerebellum can do the translation to shift the eye’s frame of reference to the hand’s frame of reference. It will also do all the scut work necessary to change all of the visual angles to joint angles. Your arm is not a tentacle. You don’t just have to reach in the right direction, you have to use your muscles to set each joint to the proper position. Roboticist call these muscle movements “forward transforms,” and figuring out the right ones for mechanical arms is a real bear.

We’re still not done! The cerebellum can get us more or less to the point where our hand is at the right place and our fingers are closing in on the pen, but the precise amount of force applied, and the tiny shifts in tension on finger muscles that allow us to hold on if the pen starts to slip have nothing to do with the brain at all. Those are mostly spinal reflexes. The nerves in the hands and spine automatically make micro adjustments in response to sensory feedback from the touch receptors in the hand. This is not to say the brain doesn’t know about those adjustments—it does! It gets tons mechanoreceptive and proprioceptive feedback and it constantly adjusts its plans in response to this data. But the loop between brain and hand takes a quarter of a second. If the hand waited for the brain to respond to a slippery pen, we’d drop stuff.

And this is basically why AI can’t draw hands.

Why can’t AI draw hands?

Everyone knows that AI can’t draw hands. But why is it so bad? There must be tons of pictures of hands.

This video indulges in the most common, wrong explanation: human brains are especially attentive and sophisticated in how they process hands. This is garbage. It’s cargo-cult neuroscience at best. The brain does devote a ton of real-estate to hands, but in sensory-motor regions not visual ones. Besides, human brains devote even more space to faces and in both sensory-motor and visual processing areas. Yet AI faces are comparatively pretty good—the best way to detect them is to check for Balenciaga cheekbones. In any case this explanation gets us nowhere. Saying that humans dislike AI hands because they are sensitive to depictions of hands is begging the question.

Besides, the explanation of hands fails to answer another critical AI art question: why does Batman have three legs?

AI struggles with depictions of bodies because bodies are highly configurable. It’s valid for Batman’s leg to be in either position depending on where he is in his stride. The model needs to make a tricky judgement call about how to render the image. This is a technique that cartoonists play with as well: it’s valid to draw Batman with what appears to be three legs so long as you add some motion blur to suggest movement.

Hands are even more configurable than feet. There are many valid positions for fingers, and they move a lot. Both stable diffusion models like Midjourney and Transformer based models like DALL-E draw by inferring continuations of edges and forms. First, the model draws the average of many hands—a blob—and then refines that blob by adjusting with data from more specific contexts. In other words, they draw a hand a lot like this:

Another thing to notice is how ordinary language usually does not describe physical body positions. If I prompt DALL-E by saying “draw an artist” I haven’t specified if they are holding a pencil, and I certainly haven’t specified if they’re using a dynamic tripod or palmer supinate grip to hold that pencil. If you do go out of your way to use a specific prompt like “Draw a left hand, as viewed from above. it should have exactly five fingers, the joints of each finger should be held straight.” You get a pretty good rendering. Except that it’s a right hand.

It’s really difficult to describe specific configurations of physical bodies with language! In fact, it would be awfully nice if I had a real physical hand attached to me that I could use as a reference. Even better if I had a huge system of neural processing with tons of specialized processing units dedicated to how to configure that hand.

How to compute with your body.

Much of human (and animal, and robot) cognition is embodied. That is, the reasoning and representations used by biological brains refer to the specific sensory-motor assemblies of our physical bodies. Because we have a body we usually don’t need to learn explicit representations for how bodies work—we can use our own body as a simulation.

If I want to imagine how a hand looks when holding a pen, I can just pick up a pen and use my own body as a reference. I’m so familiar with my body that I usually don’t even need to go that far, the configuration is immediately intuitive. I’m so plugged into the physics of my own body, that I can run physical simulations using that system effortlessly and save all of the computation that I’d normally need to use my brain for. A computer scientists might call my body a “specialized quantum computer purpose-built for physics simulations,” and that would be an accurate, albeit awkward way of describing the state of affairs.

Some embodied processing is pretty obvious: controlling a robot body accurately obviously requires a pretty elaborate representation of that body. But some is more subtle—our bodies are loaded with sensory-motor feedback loops that allow us to do reposition our grip without thinking and perceive subtle textural differences. How broadly this embodied processing contributes to human cognition is a subject of debate. There are folks who argue that nearly every aspect of human thought depends on some kind of embodied processing—One example that particularly sticks with me is the claim that we understand fluid dynamics because we take baths—but even if you don’t go as far as metaphor it’s inarguable that having a body is a really handy for many kinds of real-world predictions.

All of this creates some fairly substantial challenges for AI. It’s one thing to draw a hand by configuring pixels on a screen, it’s much harder to draw a hand using a hand. If AI is going to have meaningful impact in the physical world it will need learn to control real physical bodies. And there are some strong reasons to believe that the way that biological bodies work is about as good as it gets. Superior robot embodied cognition is never going to happen.

“Never” is a pretty strong claim that deserves some unpacking. After all, what about this robot bartender? What about these dancing robots?

Don’t get me wrong, these applications are super awesome, and definitely count as AI controlling bodies under challenging circumstances. The single-foot balance at 1:38 probably made some robotics engineering students fall out of their chairs. This is Boston Dynamics showing off, and they deserve it. But there’s some sleight of hand here. These robots are running a specific list of macros in controlled conditions. They have millimeter precision for this dance routine and this invites the audience to imagine that in highly similar circumstances—with a different dance, with a slightly slippery floor—they would also perform well. When in fact, this performance is extremely brittle and would fail catastrophically if there was a light breeze.

Non-uniform bodies

Think back to the complicated transforms the cerebellum had to estimate to translate reaching for a pen to actual joint angles. Now imagine that the pen was always exactly 10 cm directly to the right of the center of your hand. That would be way easier! You could always run exactly the same reaching program. Sure it would be a complicated list of numbers to translate between eyes and hands, but the numbers would always be the same. There wouldn’t be any calculation, just lookup.

That’s exactly what Boston dynamics is doing with it’s dancing robots. Those movements are big look-up tables that say things like “set the knee joint to 45 degrees for 3 seconds.” Current robotic systems mostly work by running lists of specific movement macros that are tightly adapted to the specific task. These macros often don’t even include feedback sensors other than a measurement of how hard the servomotors are working. Occasionally coupled with accelerometery from gyroscopes. This is beginning to change! Roboticists are rapidly working to develop new sensors, materials, and tools that produce a better sense of the position of their machines in space. It’s also not entirely clear that this needs to be solved in exactly the same way as biological systems do. While we use lots of sensors embedded in our joints and muscle tissue many robots use a cloud of infrared light to estimate the position of their limbs. That’s a clever alternative.

But these sensory-motor advances still have a major blind spot: they assumes that the body doesn’t change.

The brain not only has to make calculate new transforms to cover a huge diversity of circumstances, but it also has to re-estimate whenever the body changes. If you attempt to apply a model fit for one body to a different one the model will make very predictable errors. You probably experienced this yourself if you ever went through a major growth spurt. Suddenly you were very clumsy as your body tried to reach out for things with arms slightly longer than your brain was used too. And everyone experiences this at birth because the brain has no idea what kind of body it’s going to end up in before that body has actually grown itself. That’s a big part of the reason the brain has so many partially redundant systems and sensor-motor feedback loops. When you’re working with a self-assembling system produced by evolution it’s got to work. It’s got to work, buddy or you’re dead. It’s ok if it’s a little clumsy, but it can’t fail catastrophically like this:

As Rodney Books at MIT famously pointed out embodied sensory-motor loops are hard to build, and they’re evolutionary older than anything cognitive. It’s not an insurmountable challenge, but it took evolution billions of years to make body sensors, and only a few hundred million to get from there to brains capable of calculus. The fact that AI has stumbled into competence at calculus and language is not evidence of sophistication in walking. We should be realistic about the difficulty of the embodied engineering challenge.

I think the engineering for embodied cognition is currently happening. There are technologies currently being developed to create sophisticated physical robots. The same kind of statistical, generative models that work for text and images can also work for embodied systems and have been applied in some limited circumstances. There are a couple of serious, but surmountable stumbling blocks. First, the amount of motor data that has been collected and digitized—such as joint position, muscle load, and other proprioceptive information—is trivial compared to the amount of digitized text and images. Second, the motors and sensors that generate and collect such data are incredibly primitive relative to keyboards and cameras. Lidar scanners and embedded kinesthetic sensors will probably have to be common place before we get really robust dancing robots.

But even when we do create the requisite sensors, collect the necessary data, and train the right models, we’ll still be in a position very much like biological systems. No matter how good your model is, it only works for the body it learned to control. A different body requires a different model, or at least some fine-tuning to adapt what was previously learned to the new instantiation. Tiny differences in friction in joints lead to extremely clumsy fingers. After a few months of wear, robots will have to re-learn. Human manual dexterity is an absolute marvel in the animal world and it’s hard to learn, it takes 15-20 years before kids aren’t completely clumsy. And that skill only lasts about 50 years before the joints start to accumulate damage and freeze up.

Real brains for a real world

All of this still only just scratches the surface of what embodied cognition really means. For humans and animals the most important sensory-motor loops are actually chemical. Olfaction and gustation are the obvious ones, but more important are probably the internally-facing sensors that measure blood glucose, blood pressure, hydration, electrolytes, circulating lipids, temperature, and so on.

Most critical of these are probably nociceptors—pain detectors. I had a colleague at UIUC who got a multi-million dollar humanoid robot that he hoped to teach to speak using deep neural nets and natural language processing models. It was a very cute little guy named “Bert.”

Bert spent the first three years of his life trying not to rip off it’s own very expensive arms. The millions of dollars did not go toward an embodied pain sense. And take it from people who don’t experience pain: life without pain is hell.

Ironically, humans extreme body competence makes them somewhat poor judges of AI capabilities. Drawing with a pen is tremendously more difficult than simply coloring pixels (which is why indie game devs love pixel art). Handwriting is more difficult than typing (no one reading this blog could decipher my handwriting). Because generative AI produces high-quality images and text, we tend to assign it some of our own body competence. It’s a strange sort of reverse-Turing test, where human’s ingratitude toward the miracle of their own body leads them to dramatically overestimate the sophistication of an AI without a body.

Moving bodies is a hard problem. There’s a ton of awesome, heavy-duty engineering going on in this space. It’s a great area to get into now because this is likely to be a decades- or centuries-long effort. And at the end of that work we will still have imperfect robot bodies that need constant repairs and fine-tuning, just like real bodies do. Because they exist in the real world and the real world full of rough parts, slippery parts, and sharp edges.

It’s objectively far, far harder to pick up a pen than it is to write code. We just don’t value it because it’s the kind of boring miracle that any healthy kid can do.

There’s a meaningful parallel between Neuralink and SpaceX, in that both are using technologies that were largely developed in the 60s. Yet despite their age, space rockets and brain implants got stuck our collective memory as symbols of “advanced technology.” If you’re an engineer it’s interesting to learn about how these technologies have evolved over time, but you’re not shocked by new launches. Now imagine no one had ever heard of the moon missions and you’ll begin to get an idea of how impressive Neuralink’s announcement actually is. Developing medical technologies is gross and uncomfortable, involving surgeries and animal testing. So ironically, the very fact that there isn’t a lot of public information about brain implants makes Neuralink seem innovative when in fact their work is derivative and commonplace.

Additionally, brain implants are just technology with a HUGE gap between their depiction in fiction and their depiction in real life. In fiction, brain implants can literally read your mind to give you cool cyberpunk super powers. In real life, it is usually faster to type by blowing into a plastic tube than to communicate with a brain implant. This is not a technical limitation. Almost all of the uncritical reporting around Neuralink’s announcement starts with the fictional representation of brain implants. This is great for Neuralink and for the media entities covering the announcements because cyperpunk anxiety drives engagement better than medical device engineering. In real life, unless you’re one of a very specific patient group or a neurologist, you can probably ignore brain implants.

Here are the big mismatches between fictional brain implants and real ones.

Brain implants are old.

Here’s a 15-year-old youtube video of a monkey feeding itself with a robot arm controlled by a cortical implant:

This video from Pitt’s Motorlab. Motorlab isn’t a lab focused on BCI. It’s a robotics lab. This video isn’t showcasing the brain implant. It’s showcasing the arm. People in the know think this is impressive because of the arm. The brain implant is blasé.

Here’s an MIT story about a monkey controlling a simpler robot arm using a brain implant from back in 2000. Again, the big deal here isn’t the brain implant. It’s that the signal is being sent over the internet. Think about that for a second: 23 years ago the fancy new technology wasn’t a computer chip implanted in a monkey’s brain. The fancy new technology was a dial-up modem.

The earliest “monkey controls robot via brain implant” study is probably Fine control of operantly conditioned firing patterns of cortical neurons. Published in 1978, it involves chronically implanted micro-electrodes in the motor cortex of macaques. The monkeys learned to use their implants to operate a switch that gave them a food reward. Over time the firing of the neurons measured by the implants shifted to be better able to trigger the food release mechanism.

Even this 50-year-old study was a straightforward extension of work from a decade before. In Operant conditioning of cortical unit activity (1969) Fetz demonstrated that food pellet release could condition the firing of primate neurons as measured by implanted microelectrodes. The 1978 study just connected a robotic lever.

And this is all in primates! The rodent work was done in the 1950s! There are some technical differences—the primates tend to use microcircuit-based implants while the rodents are using arrays of tiny gold wires. But the primary the reason for this difference is that microchips weren’t invented in the 50’s.

And if you count implanted wires, I think you probably should go back to some of the original EEG studies from the early 1920s that involved shoving silver wires into human scalps. Technically not implanted in the brain, but definitely measuring brain electrical signals. It bears emphasizing that if you’re not worried about biocompatibility (i.e., if you’re ok with killing the patient), brain implants are just wires, and it’s easy to shove a wire into a brain. Because it’s so simple, it’s only a slight exaggeration to say that as soon as we had electrical wires we were shoving them into brains.

So the basic concept of brain implants—measuring brain electrical activity with conductive wires to drive machines—has been working for about a century. And it’s not just an isolated practice in research labs.

Literally millions of people have brain implants right now.

Neurostimulation is a multi-billion dollar industry. Many treatments (like those for depression) are not typically done surgically. But for diseases like Parkinson’s it’s common to implant a neural pacemaker. This procedure is super cool and cyperpunky! It involves doing surgery while the patient is awake and communicating with them to ensure the electrodes are properly situated. Afterward they can use their iphone to toggle their implant on and off. It’s great!

The other common implant is the cochlear implant. Like the 1920s EEG work, this is slightly stretching the definition because cochlear implants are in the periphery, and interact with the basilar membrane to trigger the auditory nerve. But I think it should probably count because it’s an electrode array directly stimulating the nervous system and it’s pretty close to the cyperpunk fiction. Cochlear implants give people cool abilities (there are artists who write music for cochlear implants and pipe the results directly into their listener’s inner ear via bluetooth). Cochlear implants also are a great example of medical capitalism eroding a minority culture with it’s own language. What could be more cyperpunk than that?

There are also a number of familiar peripheral medical implants, like traditional pacemakers and spinal cord stimulators. There are also a number of approaches that involve implants to aid in motor rehabilitation for spinal injuries and strokes. There are hobby applications too. Kevin Warwick had a 100 electrode array implanted in 2002 in his wrist. His wife got one too so they could communicate “telepathically.” Which in this case means that they can both turn their lights on and off and then notice that the other person is doing it.

The specific intracranial approach Neuralink’s press release parallels a fairly common technique in rodent and primate studies for brain recording. While it’s just not super common in humans (for reasons that will show up in the next couple of sections) it’s hardly without precedent. There’s even a specific device called the Utah Electrode Array that was developed for use human patients with extensive paralysis. The UEA is not a prototype—it’s a mainstream product. The prototype was deployed in 2006 in a patient named MN. Most UEAs are in patients with serve pathology like quadriplegia, ALS, MS, or locked in syndrome. These patients need some way for their brain to control devices in the world because their peripheral nervous system is seriously challenged.

So even within the narrow scope of “intracranial implants in humans for directly recording from brain cells and using the output to control an external device” brain implants are old news. So why don’t more people have implants?

The medical complications are daunting

The previous section eluded to the medical complications associated with brain implants, and…oh wow, don’t get one.

So first, the survival rate for brain surgeries in general is about 98%. DBS is a little better (in part because it’s less invasive), but about 6.6% of people undergoing the procedure are readmitted within 30 days due to complications—about half of which are fairly serious things like “brain infection” or “seizures.” There isn’t yet good data on intracranial implants because they are rare, but it’s likely to be in this neighborhood. Getting brain surgery is about as risky to your health as COVID-19 infection is for an unvaccinated 85 year old.

But let’s say you’re a risk seeking early adopter. You live through the surgery and now you’re the proud new owner of a snazzy modern implant like a UEA or whatever Neuralink is pedaling. Your friends may laugh at you, because they know a cool new implant is coming out in like a year, but you’ll have the last laugh. Because you know something they don’t—you’ll need a new implant in 6 months anyway.

Most papers on intracranial electrodes measure the useful life of the electrode in days. The best cases eek out 6 months to a few years. Implants typically work best for only about 30-40 days—about as long you can safely keep butter in the fridge. The total useful life of the implant is about like a cellphone contract.

One of the reasons that most of implants end up in the periphery is that the brain has extreme inflammatory responses to foreign bodies. Unsurprisingly, the immune system treats weird stuff in the brain pretty seriously. Astroglia aggressively encapsulate anything they don’t recognize as brain tissue. This means that scar tissue rapidly forms around most brain implants rendering them unable to measure the neural impulses they were put there to detect.

Most current research is focused on this biocompatibility issue. More flexible, stretchable materials, more stuff on the surface of the brain without wires actually penetrating the surface, more inert electrode materials, and more powerful immunosuppressant drugs.

And this research is helping to push back the time that implants can sit around in your brain. Some of the innovations are super cool! I actually had the chance to work on a grant with John Roger’s lab at the University of Illinois. Materials scientists are basically alchemists at this point and make some really fascinating stuff.

It’s worth acknowledging that the scarification caused by the implant is mostly a problem for the implant rather than the patient. Scar tissue basically acts as insulation and decreases the reliability of the signals measured by the implant over time. The brain is pretty robust to the scars so long as side effects like infection don’t develop, the patient’s cognition and behavior probably won’t be noticeably affected.

But I’m hesitant to write anything that in anyway minimizes the bravery of the people volunteering their bodies for this research. The brain doesn’t really regenerate so patients are volunteering for permanent damage. To me, this makes them among the coolest people on the planet. That said, you can lose a surprising amount of your brain and still lead a relatively normal life. In fact, losing pieces of your brain is sort of what normal aging is. Still, until the implants are more biocompatible they’re not going be able to perform their jobs over a long enough time to be really effective.

But suppose this problem gets solved by new implants that are indistinguishable from neural tissue. Implants still won’t be a great option.

Brain implants are SLOW

Picture someone using a brain implant to communicate. How does it work? What are they doing? Where is the wire? Steer away from the abstract idea of “a chip in the head” and “communicating.” What does it look like? Does it look like this?

Because it looks like this.

This is a still from patient T12, who broke the world record for typing with a neural implant at MIT in 2023. That record was about 60 words per minute, about a half as fast as most people type, about a third as fast as people speak, and about a quarter as fast as the world typing record with a keyboard. I don’t want to minimize this: 60 wpm is a big deal, it’s fast enough to be useful. This is huge progress from the early 2000s when 8 characters per minute was a good pace. Not words, characters.

But it’s important to understand what the patient is actually doing, because it’s still pretty close to what was happening in those early days. Patients look at a screen that gives them visual feedback based on the neural activity measured with the implant. That feedback causes near the implant neurons to change their responses. Then you just plug the implant into a computer and train a statistical model to translates neural changes into an output, such as moving a cursor up or down. This creates a feedback loop that the patient can use to select things on the screen. Through practice, the patient can learn to produce elaborate actions quickly and reliably. This is process essentially creates a race between the patient’s practice, and the aforementioned scarring of the area around the electrodes.

It’s important to emphasize that the patient isn’t thinking the words that they’re typing. What their doing is much more like learning to use a novel input method like typing, or clicking buttons on a controller.

There are basically four ways to make brain-implant communication faster:

1. Develop more biocompatible implants so patients can practice longer.

2. Create implants that have more electrodes that record more simultaneous signals.

3. Develop better decoding software that gets more information out of neural firing.

4. Develop better predictive text interfaces that put the likely words in front of patients so they don’t have to dig around selecting individual letters.

We discussed #1 in the previous section. #2: more electrodes is also interesting. A UEA has 96 channels. That means it can at best, pick up about 96 simultaneous signals coming from firing neurons. By contrast, each hand has about 17,000 sensors and a similar number of motor effectors. I think one of the core fictions of brain implants is that they’ll get “more information” by recording directly from the brain. The peripheral nervous system is super complex and the information that gets there is highly specific for motor outputs. It’s weird to imagine that measuring from the pre-cooked brain activity is going to be better. Still, we could certainly increase our recording electrodes 1000-fold and it would help get more info.

But all of this is sort of a moot point because almost all of the speed-ups in the past two decades are a result of #3 & #4: better software. Compare how fast you can type to how fast you can generate text with ChatGPT. ChatGPT can generate thousands of words per minute—you might not really consider it your words, but compare the sheer amount of text generated to typing unassisted. Furthermore, in most of these world record typing tests such as the one T12 is engaging in have a set script. This makes the “typing” more like playing a rhythm game where one carefully executes the right series of cursor movements. These movements then are plugged into a text generator like ChatGPT.

In other words, almost all of the increase in faster brain implants of the past 20 years come from better autocomplete. If the world typing record was allowed to use ChatGPT, I bet they’d be an order of magnitude faster as well.

Reporting on brain implant communication deliberately muddies “normal human language generation” and “ChatGPT enabled script reading,” because it’s much more impressive to imply that the brain implant technology is doing the heavy lifting.

Why not just use eye tracking?

But the much bigger problem lurking in the background here is once you think about the realities of brain implants you have to compare them to other methods for interacting with the world. You’ll notice that nearly all of the patients with brain implants have some severely disabling motor-neuron disorder like ALS. This is because anyone with any movement ability is probably better off using an eye tracker or a sip-and-puff to interact with devices. If you can signal the external world with any part of your motor system, you are typically going to be way faster at just using that part of your body than by using a brain implant. It is more natural to just look at things with your eyes and blink to click. It is more natural to blow into a tube for yes and to sip for no.

Part of what’s silly about the idea that brain implants are doing mind reading is that it assumes that there’s some kind of abstract mentalese going on inside the brain that is then translated to motor gestures that respond to the affordances of the world. The idea that your thoughts are floating around inside your brain in some kind of magical ghostly form that is then transmitted to your hands is a literally ancient belief. In fact, language processing and sensorimotor systems are always intimately co-engaged. When writing or speaking, you’re doing an extremely careful configuration of motor units in your fingers or larynx. These gestures require constant somatosensory feedback. In fact, phonemes—the sounds that combinations of vowels and consonants make—are easier to detect in the somatosensory cortex than in either motor or language processing regions. Because the brain isn’t a magical-abstract-thought-ghost floating in your skull. It’s a system for configuring your body.

In fact, it’s easier to mind read by looking at the body than it is by looking in the brain. Subvocal recognition systems that decode language from the nerves and muscles of the throat are far more successful than brain implants. And this should be obvious—since speech is muscle movement, it should be way easier to decode speech from the larynx than from the brain. Because speech only exists at the larynx, your brain cannot talk.

Even within the brain, language is highly distributed. To get at the information that eventual gets translated into the movement of vocal chords, you’d probably want to have implants in motor, premotor, somatosensory, cerebellar, and thalamic tissues. All of this information comes together and is integrated in the pharyngeal glossal nerve. That’s the brain’s output for speech.

But all of this is super silly because there’s an even more straightforward way to do speech decoding from a brain: just ask.

As cookie monster taught us when we were 3, asking questions is a good way to find out about things.

It’s not just a technological issue that keep brain implants slow, it’s really a consequence of the biology of nervous systems. That system evolved to move muscles around, and brain implants are just piggy-backing on that function. They only make sense for communicating with patients who don’t have any other way of interacting with the outside world.

The myth is the point

I think what makes me grumpy about the rhetoric of the Neuralink story is how thoroughly it requires the dehumanization and invisibility of actual people with brain implants. It requires an audience ignorant of Parkinsons, ALS, MS, locked in syndrome, epilepsy, major depressive disorder, and a whole host of other debilitating and annoying neurological issues. It also requires the complete vanishing of the real lives of patients who put their brains and bodies on the line to further this research. And of the literally millions of animal lives that came before them.

These folks and fauna are are my favorite kinds of weirdos. They are in the grand scientific tradition of Otto Lilienthal, Leonid Rogozov, and HM. They are sacrificing little pieces of their brain to better the world our children will live in. And the real day-to-day lives of these patients is so sophisticated and fascinating. They have created so many clever ways to interact with a world that has made it so difficult for them to be heard.

Neuralink could have made this a story about these patients, but instead it made an advertisement for no one. Neuralink’s devices will only ever help the most severely disabled, which is laudable. But their business model is selling the mythology of cyberpunk mind-reading brain implants. Neuralink is fundamentally a media company funded by idiot billionaires who think they’re money is going to get them into techno-heaven.

Real life brain implants exist because of the fragility and mortality of the human body. Real life brain implants are a story of blood and bone-saws, catheters and cursors that won’t move even when you beg or look a them crosseyed. It’s a story of monkeys dying in squalid conditions. Neuralink doesn’t have the guts to describe their work honestly in a press release because their company is actually selling a world where their research participants don’t exist.

Most recent interest in generative AI is driven by generative models that make text and images. Good writers and artists are rare in human society so there’s tremendous social cache in creating impressive works of media. Generative AIs enable machine learning engineers to access some of this cache by automating media creation. I’m not going to wade into that debate in this piece, but what I do notice is that “artist,” “writer,” and “engineer” are all identity labels that subsume an entire person into a job description.

If you equate an artist with their artwork and an engineer with their technologies, generative AIs seem like they stand to replace us. But if you have a more complete look at humans, generative AIs are missing some basic bits—like mouths. There’s not much social cache in chewing, so it’s tempting to argue that a robot who can chew like a crab or lobster is a trivial AI. But here’s the thing: robot chewing is actually a more impressive engineering feat than ChatGPT’s writing. High social status has nothing to do with engineering difficulty or biological sophistication.

Lobsters are delicious model organisms

I learned about lobster chewing from Dr. Eve Marder when she gave a talk at my University of Illinois graduate program back in 2013. Dr. Marder’s lab at Brandeis makes computational models of simple nervous systems, and that means getting into the nitty-gritty of how a lobster’s nervous system is actually wired together. Eschewing abstract neuroscientific discussions of vague constructs like language and perception, Dr. Marder’s work is focused on the tangible electrical circuitry that moves the flesh, blood, fat, chitin, and chemicals necessary for running a lobster’s mouth. If you want to understand how brains actually work—both synthetic and biological—it’s critical to understand tangible, real-world tissues you can hold in your hand. Forget ChatGPT’s disembodied floating intelligence: picture Dr. Marder poking lobsters with needles.1

Lobsters and crabs are popular model organisms in neuroscience because:

1. Invertebrates have relatively large neurons without protective myelin sheaths. This makes their neurons easier to stab glass needles into for measuring electrical activity.

2. Decapods are hard to kill and cheap to raise.

3. You can eat your subjects with considerably less guilt than in human research.

The specific ball of neurons in question, the stomatogastric ganglion, consists of about 30 neurons. The STG controls the muscles on the the dorsal surface of a crustacean’s foregut, a kind of combination mouth/stomach like a cow’s rumen or a pigeon’s crop. This little blob of cells is about the size of the head of a pin:

Every connection between these STG cells and the muscle fibers they are attached to is known to researchers. We even understand how various genetic factors are expressed to wire up the thing. There are biophysically realistic computer simulations of the STG’s neuromuscular activity that are so accurate as to be indistinguishable from those observed in the living organism.

What does the STG do? It chews and swallows.

The stomatogastric ganglion is a central pattern generator. More accurately, it’s two pattern generators taped together. The front one—the “gastric mill”—chews by moving the lobster’s lateral teeth back and forth over its medial teeth. The rear one—the “pyloric rhythm”—swallows by squeezing the muscles of the foregut to push ground up food into the lobster’s hind gut for further digestion. These two pattern generators trade off by reflex. Solids in the foregut trigger the chewing reflex, and liquids trigger the swallowing reflex.

Who cares though? You would have to be the most boring person in the world to care about lobster chewing! Two little reflexes trading off? Why are we looking at this thing when we could be prompting ChatGPT to write poetry? You’d think this was too simple and trivial to matter.

And you would be wrong.

The existential watchmaker

Evolutionary biologist Richard Dawkins calls evolution a “blind watchmaker,” and it’s worth pausing to consider the STG as a piece of evolutionary engineering. Here’s a plot of the electrical signals recorded from the two reflexes with swallowing on top, and chewing on the bottom:

An engineer who might want to build a synthetic STG has a simple functional goal: run the chewing, run the swallowing, switch back and forth as needed. It’s trivial to wire up a microcontroller to produce the above behavior in a couple hours.

But how long would that microcontroller have to run? For lobsters this circuit is an existential requirement. If a lobster cannot chew it cannot exist. Thus, the circuit can never fail. It cannot be “turned off and turned on again.” It must have 100% uptime. It will never get replacement parts. There will never be maintenance and any repairs must be performed by the system on itself while operating.

And the circuit must run forever. Lobsters are immortal. They exhibit what biologists call “diminished senescence”: they get stronger with age. The older a lobster, the healthier, larger, and more fecund it is. A ten pound lobster is about 50 years old. A 20 pounder has lived a century. While the core behavior is quite simple, what we’re really trying to make is a fully autonomous, self-sufficient, indestructible, immortal chewing machine.

Oh yeah, and it needs to be built while immersed in salt water, out of materials collected by running the circuit itself.

Faced with such a daunting challenge, one of the reasonable things an engineer might do is to try to simplify the circuit as much as possible. Less moving parts means less things to go wrong. Since we need two patterns we probably need two processors and a switch.

This is not what evolution did! Did you notice how there were 13 cells in the previous chart? That’s because the wiring for the stomatogastric ganglion actually looks like this:

Remember, this whole thing exists only to chew and swallow. Yet evolution made a wiring diagram roughly as complex as Bell’s original telephone.

But actually, just counting cells and wires tends to undersell the complexity of the STG. Like most invertebrates, lobsters are extremely chemically complex. Here is an incomplete list of the neurotransmitters and neuromodulators used by each cell in the STG:

My (probably out of date) understanding is that there are 60 known chemical signals in the stomatogastric ganglion. Between the wires and the transmitters, the number of parameters in the STG is about on the order of the number of parameters used in the famous “Attention is All You Need” paper: the paper that describes the architecture of sequence-to-sequence transformers that you probably know better by their brand names: ChatGPT and stable diffusion.

Surely this is overkill for a lobster tummy? Why on earth would you need so much processing sophistication to chew?

Chaos is the nature of things

The world that lobsters and humans live in cannot be made simple. Entropy is the guiding principle. Health is temporary. Our circuits will fail. It is not merely that time breaks things down. There is no beginning “pure” state from which to decay—due to individual genetic variation, many lobsters don’t even express some of the neurotransmitters required by the circuit. There is no perfect, working version of an STG found anywhere in nature.

And yet the lobster chews.

Systems composed of central pattern generators like the STG are common in all nervous systems. Many behaviors require a smooth, repetitive signal, like a metronome or bass line music. Breathing relies on a pattern generator located in the brainstem. A cicada’s droning wingbeat is driven by one of these. The motion of a cockroach’s legs. The sinoatrial node that beats the human heart. If you want to coordinate any behavior with any exterior physical object you are going to need to drive muscle tissue in time with the external world.

Suppose you wanted to set up a circuit to chew some food. You could put the food in a bag, and set up a metronome so that the swinging arm would mash the side of the bag every second or so. Then you could set up a second metronome that would pull open the bottom of the bag and dump out the food scraps every few seconds. This would work well unless the two metronomes got off from each other. Then the circuit would break. So you add a third metronome, and a fourth, and a fifth. With each new element, the timing becomes more reliable—inaccuracies in the timing due to some of your timers being fast and some slow will tend to cancel each other out. The system fails gracefully.

Every lobster is filled with a vast collection of ticking clocks. Each of these clocks is composed of smaller ticking clocks, and most of the clocks are broken. But the lobster is not broken. The lobster is immortal.

The tree of life is a tangle of living and dead parts. Unlike technology, which always works best on paper, biology survives contact with reality. Our technology is absolutely ephemeral compared to biological individuals. An iPhone lasts about as long as a rat. Imagine how expensive a computer that lived as long as a human would be. There are lobsters living right now that pre-date the lightbulb. There are bowhead whales that lived through the Civil War and Greenland sharks from the Renaissance. And this focus on the individual completely undersells biological durability. Life has been around for billions of years because biology is happy to reboot life forms. That’s what death and birth are for.

Modern AI technology like ChatGPT seems impressive, until you realize it does not have to coordinate its output with the real world. It relies on a biological being to do this: a human reader. AI also depends on biological life support for its inputs. We speak casually about “training a new model” as if that is an process that happens automatically, rather than the result of some machine learning graduate student who might rather work in the lab that gets to eat lobsters at the end of the year. Being a writer is a fine thing! Being an engineer is a fine thing! But having a nervous system—a real one, full of broken bits—is a far finer thing.

Fluid, robust, and natural behaviors require massive assemblies of redundant parts all working together. Generative AI is a robust system that can survive contact with the real world, it’s an attempt to provide narrow economic value in the domain of media creation. It’s a poor picture of a human nervous system, and fears of AI replacement can only thrive in an audience with a poor mental picture of how brains actually work. When you picture a nervous system, imagine an endless tessellating choir stretching off into the distance. Every singer is terrible. Each one of them is out of tune, and out of synch, and some are sick with laryngitis. But when they all raise their voices together it is beautiful harmony.

During the industrial revolution mechanical technology was cool. Fast-food capitalized on this by creating the automat: a fully mechanized restaurant that dispensed drinks, cooked meals, and served customers without front-of-the-house human contact. The automat fad lasted about 40 years. Most automats closed during the great depression because labor was cheap and because having no employees was bad PR. The novelty of being served by a machine kept a few automats holding on. The last one, Horn and Hardart, closed in 1991 having survived for years as a NY tourist trap. It’s a Gap now. Automation in restaurants continues to thrive, but with machines backstage and people out front. Modern fast-food joints are mirror images of the automat.

This is important historical context for Google’s recent presser announcing a partnership with Wendy’s to develop Drive-thru bots:

"Wendy's introduced the first modern pick-up window in the industry more than 50 years ago, and we're thrilled to continue our work with Google Cloud to bring a new wave of innovation to the drive-thru experience,"

Todd Penegor, President and CEO of Wendy's

It’s interesting to see Penegor describing a 1971 pick-up window as innovative almost a century behind the automat. This amnesia continues as the announcement passes through the media digestive system. First there is relatively staid coverage in the Wall Street Journal, describing the announcement, but adding the weasel-y and unreferenced:

“Up to 80% of food orders at Wendy’s are made at the drive-through lane.”

Next The Verge gooses the story up with some snide opinion:

“Hangry people don’t want to hear about your GPU shortage and how it’s slowing down their orders — just saying.”

And finally Reddit—the lower intestine of the internet digestive system—excretes the following:

“AI chatbot to replace human order-takers at Wendy's drive-thru. 13.9 million displaced.”

This hype cycle needs to neglect the historical perspective that the automat provides. It feels really strange for me to read these articles, because as a professor of machine learning who’s worked in both academia and industry developing machine learning systems, I think about the historical context all the time. To make a system that can actually work I need to understand the prior art—both technologically and economically.

Yet I like this case because I don’t think it takes much specialized knowledge to understand the hype that’s going on here. I think you just need to visit a Wendy’s.

Making food is easy, but food service is hard

A fast-food restaurant is already a highly automated system. Specially prepackaged food is placed into custom machines with preset cycles to produce highly consistent outputs. There’s some minor combinatorics to assemble condiments and options so that the customer can “have it their way” but even this is partially automated process. This automation happens to look more like bottles with specific nozzle sizes to get the dose of mayonnaise just right. It’s a different, more functional aesthetic than the spidery chrome robot arms that usually imagine when we think of automation. Fast-food workers are already what Amber Case would call “cyborgs:” organisms in obligatory feedback loops with the machines they operate.

Importantly, the worker who takes the drive-thru order mostly focuses on food assembly and delivery. Drive-thru orders are a task, not a job. The drive-thru worker typically wears a headset so their hands are free to operate machines, clean, and do food prep. Taking the order is actually a small part of what they’re doing. Besides, the ordering system itself is already fully automated. You can go to Wendy’s website right now and order whatever you like. You won’t need ChatGPT to do so: just click on the items you want. Typing your order out for a chatbot would be an annoying unnecessary step.

So there’s little inefficiency in the ordering process for AI to correct. The task of taking orders in the drive thru is already fully automated for customers who order online, and the in-person orders are already taken by workers who are mostly doing other stuff anyway. Wendy’s knows this, and has modest ambitions:

The goal, Mr. Penegor said, is to get more customers through the drive-through in the shortest amount of time, picking up extra sales for every minute or so shaved off the ordering process.

I think even this is unlikely to pan out. Most of the AI projects I worked on when I was at IBM research were exploratory, and it was pretty rare where we could beat a human at an ordinary task. And there’s also little reason to think that advances in large language models such as ChatGPT are what is missing. The main job that the person taking the drive-thru orders is doing isn’t language processing, it’s hearing. The sensory processing necessary to translate what the person in the car is saying is hard! There’s traffic noise, wind, food prep sounds, sentences that get cut off and so on. Filtering all of this noise into text is a far more challenging machine learning task than understanding that “quarter pounder” and “hamburger” are sometimes interchangeable symbols. And again, Thomas Kurian, Google Cloud’s CEO, identifies this as the actual technology they’re working on:

Among other challenges, Mr. Kurian said, there can be a lot of extraneous noise in a family car, such as music or children in the back seat. The software has to cut through these sounds and focus on the person making an order and the kind of language they are using… “It’s a very complicated technical problem.”

I don’t think that this sensory problem represents an insurmountable gap. Speech-to-text systems are pretty good now and are getting ever better due to the large sonic data sets eagerly donated by users of Hey Google, Alexa, and Siri. But speech-to-text remains brittle. Especially in noisy environments or for speakers with accents other than “white man from the midwest.”

But even if the audio problem was somehow solved (hard), and even if this did result in an appreciable reduction in food prep time (unlikely), I still think that it wouldn’t result in Wendy’s replacing any employees—and neither does Wendy’s.

Fresh, made-to-order rituals

Wendy’s slogan is “Fresh, made-to-order.” Their primary appeal is that the customer will get better quality fast-food cooked on a griddle by a person in a paper hat. Wendy’s tries to position itself between McDonalds and diner food/fast-casual. Wendy’s has been happy to adopt the same automated machines for dispensing coke, ice cream, and mayo as everyone else. But they trumpet the value of certain ritual practices—such as not freezing their hamburger meat. Their marketing attempts to convince the customer that this makes their products “better” than their competition.

This performance relies on shared fiction. Do you actually like hamburgers that haven’t been frozen better? Can you tell? Have you every thought about it at all? I haven’t and I’m pretty sure Wendy’s relies on customers not deeply engaging with these questions. Like all companies, Wendy’s profit margin depends on the gap between how much their product costs to make and how much their customers are willing to pay.

This is why I think Wendy’s is turning to AI after their painful post-pandemic restructuring. AI might save them a trivial amount of time and money. But just as with the automat, new technology is cool! Wendy’s is interested in AI is as a marketing appeal.

A good comparison might be how Starbucks maintains their brand value. Consider Starbuck’s directive to baristas to not make more than two drinks at a time. Customers complained when baristas created assembly-lines to more efficiently churn out batches of drinks at busy times. In other words, part of what customers wanted was the experience of having a specific person pay attention to their drink. The ritual of a human interacting with your food is part of what you pay for at Starbucks.

Starbucks knows this! They desperately don’t want you to think about efficiency when you’re buying coffee. Their markup is 80%! The gap between what Starbucks costs to make and what customers are willing to pay signals almost Apple-level brand value.

Getting coffee is a social ritual. Starbucks has monetized that ritual very effectively. Wendy’s tries to create the social agreement that frozen hamburgers are bad. All economic transactions involve embedded social signaling. No will die for want of a latte. We buy a latte to express something about ourselves: our social position, our values, and the general fiction that brings meaning to our lives. Wendy’s inclusion of AI in their drive thru probably won’t save them money, but it may allow them to make the argument that their product is more interesting, smart, high-tech, and futuristic than their competitors.

This is a fine line to walk—it may turn out that customers enjoy the old drive-thru ritual. Being paid attention to—even by a minimum-wage worker for a couple of minutes—is of some real value. I have an acquaintance who’s retired dad goes several times a day to the deli to buy coffee and talk to the staff. His fridge is completely full of half-empty coffee cups. It’s not about the coffee. He’s paying to hang out at the deli during work hours. America runs on this kind of sociocapital exchange.

Technological capitalism has been far more successful at manufacturing stories, ideas, entertainment, and brands than it has been at making real products. It’s nearly always cheaper and easier to make media, even whole new forms of media that promote your product than it is to actually do the difficult technical work of improving efficiency. Changing your customers’ mental “software” is effective, invisible, and need not result in productivity gains to be profitable. It’s a good bet that you’re reading this article at work—that is a form of technologically mediated entertainment that probably actually lowers your efficiency.

Wendy’s AI drive-thru is for brand building. AI is cool. Investors and customers are interested the experience of using ChatGPT to order a burger. Such a system might increase the value of the product that Wendy’s delivers allowing them to charge a little bit more. Or it might create a tiny boost in efficiency allowing them to sell a little bit faster. Or it might just get them a transfusion of cash from investors. But even if it hurled the culture of fast-food back to the late 19th century automat that wouldn’t be particularly scary or sexy.

Pursuit of fictional brand value is also why the media ecosystem needed to juice up Google’s press release. The last century of technical advancement has made it virtually impossible to make information transmission faster or more efficient. You can get first-hand battle footage from the war in Ukraine uploaded from soldiers’ phones for free in real time. So the WSJ, and the Verge, and Reddit need to build brand value by manufacturing gravitas, snark, and paranoia. Media companies—just like Wendy’s—are leveraging AI technology not to increase their efficiency, but to increase their perceived importance.

Love and Robots

The automat was built on branding and the combination of selling both food and an entertainment experience: watching clever little machines deliver the food. It wasn’t meaningfully cheaper or more efficient than other forms of fast-food prep. But it could also pad the bottom line selling visions of a shiny, worker-free utopia.

At least until the emergence of newer, more powerful media technologies. Charlie Chaplin’s depression-era film Modern Times (1936) popularized the idea that new technologies were not cool. For millions of out-of-work people, the idea that robots were going to take their factory jobs was genuinely concerning. Four years later, Chaplin’s “The Great Dictator,” drove home the message by making the automation of genocide in Europe the new mental image of the technological future embodied in the chrome-and-enamel machines of the automat.

Never mind that Chaplin’s factory itself was a commercially successful fiction mediated by new technology. Modern Times was a machine for effectively disseminating the idea that mechanization produces dystopian, alienating, and de-humanizing physical spaces while simultaneously utilizing depictions of such spaces to arouse engagement, interest, and outrage. The story of the 20th century is the story of using extremely cheap, efficient media technology to re-narrate industrialization’s brand value. Media technologies slowly morphed the automat into a restaurant full of of paid actors. And as Chaplin would have known better than most—there’s virtually no ceiling on what you can convince customers to pay to see a theatrical performance.

It’s actually not a surprise to me that the first-generation of AI that seems poised to actually have a major impact on economic life is Generative AI. Most of us in the knowledge economy work acting jobs. Industrialization has ensured that few people need to grow food or build houses. The rest of us can focus on creating entertaining diversions for each other. And of course, if you know how to put on a good show, you probably already possess all of the tools you need to convince your audience to consume more of your particular brand of entertainment.

Don’t forget to subscribe!

I’ve spent my career studying intelligence at every level: publishing papers in machine learning, neuroscience, psychology, and sociology and earning a Ph.D. in Computational Neuroscience. I was lead analyst on the world’s largest brain-training clinical trial, helped build a meaning-making machine, developed new measures for human memory, worked on IBM’s robot tutor, and co-authored a book on how collectives learn. Now I teach machine learning at the world’s most innovative University.

I’m often asked “How do I get smarter?” by anxious students, older family members, and peers from other disciplines. For years I gave the standard research advice for good brain health: “Eat right, exercise, get enough sleep, pay attention to what you’re doing.” But everybody hates that advice!

Luckily there’s another way.

Fake it!

While it’s hard to be smart it’s easy to make other people seem dumb.

Most people know the Turing Test as a computer attempting to fool humans into thinking it’s a human. But that’s not the way that legendary computer scientist and codebreaker Alan Turing described the test in his 1950 paper:

“The ‘imitation game’ is played with three people: A man, A), a woman, B), and an interrogator, C), who may be of either sex… The object of the game is for the interrogator to tell which of the other two is the man and which is the woman…

Now we ask the question, ‘What will happen when a machine takes the part of A) in the game?’ Will the interrogator decide wrongly as often as when the game is played between a man and a woman?

Turing wrote the paper while he was on parole. Despite his critical and brilliant code breaking work for the British government during WW2, Turing was charged with “indecency” for a relationship with another man and avoided jail time only by agreeing to estrogen injections. So when he was asked whether computers could be considered intelligent, he pointed out that the authorities seemed to care far more about passing than intelligence.

The Turing test also showcases its author’s razor-sharp wit—or, at least his ability to make himself seem intelligent by comparison. It plays with gender and deception to stress out its participants. It invites its audience to assume that Turing sort of knows everything about how intelligence works. And, by intentionally complexifying the definition of intelligence through inventing this test, Turing ensured that he could never be proved wrong: only different.

You too can acquire this power simply by reading this newsletter!2

Why Now?

I was inspired to create the newsletter because I felt that the recent discourse around AI—specifically large language models and image generators—is coming from actors who are trying to frighten, bullshit, and muddle their audience into acting foolishly. It’s one thing for a queer intellectual like Turing to use his brains tactically against the state that was brutalizing him. It’s quite another for corporations, politicians, and other assorted jerks to use those tactics to bully the less powerful.

I prefer an audience that’s brave, wise, and creative. And so I’ll cover three kinds of stories in this newsletter: AI critiques, under-the-hood tutorials, and neuroscience parables.

AI critiques pick apart hyperbole about AI, reducing the anxiety that these articles attempt to induce. Under-the-hood tutorials provide some expert knowledge about the inner workings of machine learning systems. Neuroscience parables describe examples from animals, brains, and ecosystems that challenge our ideas of what intelligence is. I intend for this newsletter to show how the AI magic tricks are done, disrupting the ways that the idea of intelligence is used to harm and control others.

Because ultimately, I think Turing’s test is a moral one. I think he wanted to live in a world that was more generous. He did not think that deciding whether something was intelligent was a prerequisite to treating it with decency. His own experience taught him that intellectual fireworks were no charm against brutality.

The goal of these pieces is to consider thinking things—of all sorts—with depth and grace.

—Patrick