For the rest of this article, let’s step back to a technical problem: how do we formalize identification of which agents alignment should focus on? We have one conventional “test case” that is almost too obvious to state: in our present circumstances, this rule should identify people as the agents to focus on. There should also be good reason to believe the rule makes proper decisions in other circumstances. We also want it to be plausible that the same rule is substantially easier to implement in software than any approach based on reverse-engineering human values.

I’ll start with a big-picture consideration of how intelligence has increased in the universe over time, but I promise I’ll return to the original goal.

Who Could Think About Alignment?

The anthropic principle explains the apparently improbable fact that we inhabit a universe capable of developing intelligent life. An awful lot of random chance needed to line up properly to produce us. However, the question “how did we get here?” can only be asked by life intelligent enough to think abstractly and formulate the question. Yes, there may be many possible universes compatible with a broad formulation of physics, and likely only a small minority of them are compatible with intelligent life having evolved. Hence, contingent on someone being able to ask the question of how we wound up here, we must be in one of the universes conducive to intelligent life evolving.

Let me now develop this kind of thinking toward a principle that any agent at roughly the computational scale to support thinking about AI alignment should be defined as a proper focus for alignment. Instead of making a somewhat arbitrary choice up-front about which agents qualify, let’s try to define alignment once and for all to capture all beneficiaries. (We’ll come shortly to a twist that keeps literal application of this idea from taking us where we expect, motivating a fix.)

In this explanation, I’m going to embrace the computational theory of mind shamelessly, because it has always felt natural to me. This theory says that brains are understood as special kinds of computers, so thinking about computation broadly subsumes thinking about (biologically embodied) intelligence. For now, we’ll discuss at an abstract level, but the next section will concretize to societies of humans (not atomized individuals) as critical computational systems.

Some computational architectures can host the kind of abstract thought that leads to asking big questions about the universe. Some can’t. Let’s simplify for the moment and think in terms of one linear dimension of computational power. Consider how much such power is needed to be able to design recursively self-improving AI. Presumably there is some relatively well-defined minimum level. If self-improvement is kicked off thoroughly enough to cause an intelligence explosion, then quickly all the rules change, and the familiar alignment problem no longer applies, or at least its context becomes so different that the ideal rules are probably different. For example, AI designing AI is likely to have better explicit, formal/algorithmic understanding of its own values than we do, if it is successful creating its own successors.

These observations combine to suggest that there is a narrow window of computational power that naturally evolved intelligence can have when it is working on formulating alignment: it must have passed the minimum power threshold but not yet have begun an intelligence explosion. The window is narrow if we assume that solving alignment requires fairly similar tools to designing recursively self-improving systems. And let’s remember that we’re comparing slow biological evolution to faster deliberate engineering (a la Kurzweil’s Law of Accelerating Returns), so a “narrow window” might be, say, 1000 years.

So we can potentially formulate alignment in terms of the minimum level of computational capacity that could think about alignment. The definition wouldn’t literally be recursive – it would just happen to talk about the relevant computational level, which happens to be the one that we as humans inhabit (the coincidence explained in this way being reminiscent of the anthropic principle). Then we automatically “pick up” the agents we meant to detect.

Layered Intelligence

Is there one linear measure of computational power that predicts enough about the values of intelligence? Maybe not, but let’s add one more dimension and see what may get easier to predict. I’ve previously suggested a unifying slogan for this blog, “intelligence depends on organizing computation correctly and efficiently”, that is relevant here. Just measuring some notion of the number of atomic units of computation doesn’t do the job. It matters how those units work together to solve hard problems.

It’ll help to orient ourselves with a concept from biology: evolutionary transition in individuality. A great canonical example is more-primitive parts coming together to form cells, which come together to form organisms. At each step, units that had operated independently and even competed with each other find new ways to coordinate and function as coherent wholes. (To head off any confusion, I’ll note that while I’ve found writing on such transitions in an AI-futurism context, it seems to have gone in very different directions from what follows: humans merging with AI and humans by ourselves becoming larger social macro-organisms.)

The world of manmade computers exhibits similar layering. I argued previously for the opportunity to introduce new abstraction layers when AI is adopted pervasively enough, with the potential to provide similar benefit as from the celebrated digital abstraction that underlies most electronic circuits. A single computer, say one suited to act as an Internet server, is built out of components but is structured such that the computer appears to act with one purpose. Individual logic gates may fail or glitch, but it happens infrequently enough to be an exception to the rule, which we usually avoid thinking about. We then see companies putting many computers together into data centers. Through marvels of engineering, these computers can also be seen as acting together for single purposes. Because of the sheer scale, component failures are more common, but good design of distributed systems typically hides those failures admirably well, just as our human bodies prevent cancer at the cellular level most of the time but not always.

However, consider the step up from data centers to the whole Internet. Different data centers are owned by different organizations that compete with each other. It is emphatically not the case that the Internet acts as though it had a uniform purpose, abstracting away the actors that comprise it. Likewise, while humans may abstract their constituent cells well, a global economy populated by humans absolutely reveals the motivations of individual people. The human brain is thought to be able to perform the equivalent of about 10^15 floating-point operations per second (subject to standard caveats about brains being interestingly different from GPUs). The Frontier supercomputer hits about 10^18 in the same dimension. On top of the apparent lead for manmade computers, there is a critical difference between these substrates. Humans roll up into larger social groups, but we maintain individuality that gives rise to continuing conflict and competition within those groups. In contrast, supercomputers can be agglomerated into data centers with uniform governance, providing useful abstractions of working together on single computations on behalf of single owners. Hence, it remains relatively straightforward to increase the scale of a supercomputer (i.e., wait a few years for the next generation to be developed). While brains retain an advantage in power efficiency, we see that deliberate engineering has achieved a scale of internally well-aligned computation well beyond what biological evolution ever has, and it is plausible that this gap only grows with time.

I don’t mean to say that human brains considered as isolated computers are capable of formulating alignment and developing self-improving AI. Instead, we’ve depended on culture and social structures that agglomerate such brains into larger processes surprisingly close to the scale of frontier data centers. That scale may be necessary to tackle either AI-related problem. So, when we measure evolution approaching a threshold to be able to begin tackling those problems, we are considering not just the units with good integrated alignment (like individual people) but also the larger agglomerations that are needed to get the work done. One conjecture here, then, is that the first computational systems powerful enough to formulate alignment will be societies of relatively loosely integrated individuals: we need fairly powerful well-integrated individuals to get real thinking done, but they also need to work together at a pretty large scale to realize the important questions and work out solutions to them.

This observation suggests that we define a computational-power floor applied only to units with strong internal alignment. We backsolve for that floor by analyzing what seems to be necessary to support the right kind of abstract thinking, but I don’t think it’s necessary to characterize the larger, less-integrated computational systems directly – just to be inspired by examples in choosing constants. This move is, at one level, unsatisfying, taking us back toward an alignment condition that seems arbitrarily biased toward one kind of intelligence. The potential saving grace comes from the previous section: the possibility of one Goldilocks band of computational capacity sufficient to develop recursively self-improving AI, so that it really should be true that any society confronting this lesson has similar computational capacity. It then becomes conceivable to deduce what sophistication of individuals supports that capacity, considering that natural evolution runs slowly, so we’ll tend to get the simplest individuals that are up to the job. The simpler individuals couldn’t team up to solve alignment, and the significantly more sophisticated individuals don’t appear until the intelligence explosion is underway and the rules have changed.

This approach offers pretty elegant protection against the outcome of ignoring individuality at the level where we’re used to considering it. For instance, while corporations work effectively toward a variety of goals and can be said to exhibit higher-order thinking, many commentators would see it as dystopian to identify corporations as the primary agents! What saves us is that corporations are leaky abstractions, e.g. a CEO typically needs to work very hard to repeat company strategy over and over again, in a pithy enough way, and yet still finds inconsistent execution across teams. Perhaps the individuals we really intended by that word turn out to be precisely the sufficiently well-integrated, internally aligned computers, so now we only need to figure out how to describe that concept.

I’ve also bumped into a related (and controversial) concept that I haven’t studied in-depth yet: integrated information theory (IIT), which characterizes mathematically how pieces come together into conscious wholes. I’ve written elsewhere about consciousness and why I’m not choosing to refer to that concept. In the present article, I’ve been dealing just with the power of different systems to compute good strategies for alignment, though perhaps some useful mathematics could turn out to be in common with IIT.

Conclusion

All of the above suggests the following alignment principle, which probably wouldn’t seem at all suitable on first inspection, but perhaps I’ve explained why it’s promising.

Generalized alignment should identify individuals by their capacities for well-integrated, internally well-aligned computation past a certain scale, always choosing the maximal such units in a neighborhood.

The maximality part is there so that we don’t do the equivalent of choosing cells, not people. Though the computational-capacity minimum should be chosen to exclude cells, we wouldn’t want an accident like identifying a person’s two brain hemispheres as separate agents.

Naturally, there would be plenty of work left to do in fleshing out all of those concepts into computer code or formal logic, in addition to doing the same for whatever decisions should follow the identification of agents. For instance, there may be good statistical characterizations of well-integrated intelligence that seem obviously compelling after-the-fact, but maybe not. The task seems easier than for somewhat open-ended formalization of human values, with ourselves as an evolutionarily contingent legacy system that is hard to reverse-engineer as a black box. A powerful intelligence might reason from first principles about what intelligence at different computational scales should be expected to want, or, if you’ll allow me to sneak in a second Star Trek reference, it might follow a kind of Prime Directive of noninterference (or other rules of engagement) with qualifying intelligence, as measured by how the AI’s actions perturb statistical patterns in the agents to focus on.

Some interesting misfires of this rule look possible. On the one hand, we worry about false negatives like not identifying children whose brains haven’t developed enough yet, which motivates ideas like setting the computational lower bound conservatively enough or perhaps adding an element of predictable development of an agent to pass the computation threshold. On the other hand, we may have a false positive of identifying a frontier supercomputer as an individual, though perhaps again forecasting where such intelligence is headed reveals a good destination. The next article will partly address that question, considering how there might be some amount of inherent convergence of intelligence toward certain laudable goals, regardless of the alignment starting point.

First, I think a bit about my personal background is relevant. I have specialized in how to scale formal verification to apply to realistic systems without imposing unrealistic costs on engineers. Just to keep things interesting, I typically add another requirement: we want to minimize the trusted base of any given verification. In other words, imagine someone builds an artifact and formally verifies it. The trusted base includes every line of code where a bug could have led to accepting an artifact that actually doesn’t align with the author’s intent. We can find that the trusted base includes code of the artifact, say for the hardware if we have only proved software; code of the formal-verification tool, in the common case where it hasn’t itself been formally verified; or code of further infrastructure underneath either, like an operating system we trust to run the verification software. I’ll have plenty to say in later articles about the best practices of my community in this regard and how they may be helpful to seemingly new problems around AI.

The important upshot for now, though, is that my perspective is one of an engineer. My projects almost always involve building nontrivial artifacts alongside their machine-checked proofs. These are serious proofs checked into version control and often developed by multiple collaborating engineers. Little conceptual mistakes can lead to the proofs being rejected. It’s an activity that doesn’t seem to map well to mainstream AI-alignment work. On the one hand, we have best-effort projects to align and evaluate current AI models, without any particular theoretical guarantees. On the other hand, we have writing more in the register of philosophy (some to be cited shortly) that tackles the big questions of artificial general intelligence and how it may come about from recursive self-improvement. There are no rigorous proofs about complex implementations, because those implementations don’t exist yet or are diabolically unstructured in our present understanding and thus resistant to proof. As a result, it is easy to miss small conceptual problems that doom certain frameworks. I’m not saying I’ve pinpointed those problems! I just know from experience that the probability is near 100% that any complex system that hasn’t gone through rigorous formal verification is actually incorrect.

So let’s talk what makes for a good specification in the spirit of AI alignment, treated as part of a formal-methods project with familiar engineering considerations. By the way, I’m a newcomer to the world of alignment and will appreciate any dialogue about influential and related writing that I’ve missed. Our first stop in this article may seem like a bit of a swerve from the topic I just promised, but please bear with me. We will return to a formal-methods perspective on why we may not want to center humans in the top-level formulation of alignment, even if we play an important role internally.

Abstraction Layers of Life as We Know It

The field of AI alignment so far has generated some big and nonobvious ideas. What is called the value identification problem is basically figuring out the right specification for AI, from the perspective of some thinking agent that we want to privilege morally. The principle of complexity of value asserts that there is no short description of human values; there is fundamental complexity in what we have wound up wanting. This principle sounds like bad news for writing down human values as a formal specification! A well-known alternative proposal involves a kind of epistemic humility by advanced AIs, through the notion of coherent extrapolated volition (CEV). The idea is that the AI’s starting specification is that it should figure out what humans would really want, if they were augmented in certain ways. Then that derived specification should become primary.

The place where I get stuck is why we assumed centrality of humans in the value-identification problem, and I’ll explain two kinds of objections, in this section and the next, one philosophical and one grounded in the pragmatics of formal verification. These objections stand independently of each other.

I think it’s widely recognized in the field that formulating this CEV objective is very challenging, which certainly sounds right from a formal-methods perspective. Still, I run into an objection to it, even presuming perfect formalization of what it means to learn human preferences. The problem is potential conduct of an AI during the period when it is learning our preferences. Assume it doesn’t yet have perfect models of our behavior, so it can only learn through real observation of humans. What’s to stop it from putting us all in vats, Matrix-style, constantly running us through different VR scenarios, to most efficiently figure out our preference structure? Or, substantially less outlandishly, what’s to stop it from taking over our social-media feeds to try to provoke different emotions, learning our preferences at the same time as triggering social problems as we already see from infinite-scroll social media today? It’s not like we can assume some bootstrap morality that flags those tactics as unsavory; finding the bootstrap morality isn’t obviously an easier or different problem. Even if the AI eventually creates genuine paradise compatible with our values, we might object to the intervening dark ages of manipulation. (ChatGPT directed me to a recent paper by Emmons et al. studying a related problem more technically.) Now, it turns out that the rest of this section can proceed independently of this point, so I mostly include it in case readers have good references to share for prior art.

At some level, human centrality is obvious: humans are designing AI (for now), and humans want what we want. However, there is a familiar historical narrative of progress that depends on seeing ourselves as less and less central in the grand scheme of things. Copernicus argued that the Earth revolves around the Sun, not the other way around. Darwin argued that humans resulted from an evolutionary process that produced all of life as we know it, giving us no special place in the grand scheme of things. If we’re not a natural endpoint of a universal process, why should we give ourselves that status in the goals of powerful AI?

Bostrom in Superintelligence considers other approaches to indirect normativity, which try to avoid centering human values, but to me these formulations seem to run into trouble for assuming the major problem of specifying human morality is already solved, even if one formulation (“moral rightness”) is couched in terms of universal, not human, morality. I personally am convinced by moral anti-realism and see the case for universal morality as weak, as morality arises from evolutionary forces and should be expected to be different in different times and places.

My argument here is related to moral circle expansion, a philosophical concept that explains our potential granting of more moral standing to nonhuman animals and AIs alike. For instance, imagining our ancestors formulating alignment as recently as 150 years ago, it may have been “clear” that only a writer’s own race deserved consideration, a principle that we now find repugnant. We could perhaps say that I’m taking that principle to a more abstract level, where we see, say, humans and cows as representatives of roughly the same abstraction level of the universe, raising the question of what makes that level special.

Let’s consider the same question from more of an engineering perspective. Life can naturally be organized in layers, and we can imagine formulating alignment from the perspective of each layer, which can lead to very different outcomes via powerful AI systems, each outcome looking undesirable from other layers’ vantage points. What makes the layer of individual humans special – special enough to get this treatment in alignment?

I will have more to say on the biological version of this picture next time. I’m not, by the way, implying that atoms have human-style preferences. The point is more that it’s hard to formalize “preference” in some way that isn’t sneaking in a special role for humans. One common answer is that the organism layer is distinguished in supporting consciousness, but I’ve already argued against the big-picture utility of that word and won’t relitigate here.

One related alignment idea is natural abstraction: the idea that well-engineered thinking systems will develop the same abstractions for making sense of the vast physical world, because those abstractions just work, and thus we can assume that AI will evolve toward abstractions familiar to us, and we can specify alignment in terms of those abstractions without worry of (gross) misinterpretation. This perspective doesn’t tell us which of many abstractions to extract values from, though, including in situations as in the diagram with agents nested inside each other. That nesting also figures in the idea of embedded agency, which considers how agents should plan changes to complex worlds that include themselves. One subproblem is how agents should plan the creation of nested agents that do helpful supporting work. This perspective doesn’t seem to answer who deserves to be allowed to specify values: in the case of evolution optimizing humans who optimize Python programs, neither extreme orientation (toward highest-level or lowest-level optimizers) gives us the expected answer.

Engineers are used to thinking of systems at different levels of abstraction, and it’s jarring to call out one level as getting special treatment. Kurzweil in The Singularity is Near grounds the history of the universe in development of increasingly complex patterns, with nesting structure like I just sketched, but overloads the term “humanity” to cover whatever artificial intelligence we design, too. On one level, it solves the puzzle to decide that “human” is a synonym for the whole history of the universe, or at least the upcoming part for our own neighborhood in space! That move seems to me like hiding from the problem, though, if we think in terms of some greater cosmic destiny centered on forms of intelligence beyond our comprehension – and which, by the way, would be very helpful to us if they liked us.

Controlling Specification Complexity

Let’s snap back to the formal-methods engineer’s perspective. Whether we are formalizing human values directly or expressing what it means for a system to seek them out and then honor them, it sounds like one heck of a complex specification, and formal-methods people know that complex specifications create the biggest risk for assurance. Humans become an example of a complex legacy system, absolutely the worst kind to tackle for correctness-critical specifications. We evolved without selection pressure for our values to be understandable to computer code, so there’s no reason to think this problem is especially tractable. Maybe this problem is nonetheless the important one, and we just have to find a way to solve it, but let me continue explaining why it’s worrying (and then the next article will get into considering how the problem might actually be avoidable).

The trouble is that complexity in a specification tends to force complexity in an implementation, as code is added to deal with all the challenges that have been spelled out. Then, in turn, complexity of these two promotes complexity of a mathematical proof of correctness, which naturally references all parts of the specification and implementation. We may find that the specification includes N distinct concerns, and interaction among every pair of them (the feature interaction problem) needs to be treated explicitly in the proof, for N^2 required cases. This phenomenon should make us very wary to put a complex legacy system in the specification.

At this point, I’m imagining objections that sometimes we only know how to build systems including some complex parts. The usual saving grace for formal methods is the one I explained around surprising payoffs of end-to-end formal verification. Namely, sometimes it is easier to be sure we got a specification problem right by doing proof about a larger artifact that includes the original as one component. Why? The new top-level system may admit a simpler specification, and the original system’s specification becomes just part of internal implementation and proof details. If the inner system has a specification bug, either we catch it in the course of verifying the larger system or the bug turned out to be inconsequential for our larger goal.

Maybe the hard-to-decipher “humans” of the programming world are complex, popular programming languages like C. Assume for the moment (though I have argued to the contrary and will come back to doing so in later articles!) that building realistic computer systems requires using these programming languages. Are we out-of-luck, now needing to include formal semantics of the languages in our top-level system specifications? Not if we make the right architectural choices! If we also formally verify the implementations of the programming languages, then choice of language becomes an internal implementation detail, with no representation in the top-level specification that we should submit to careful auditing, testing, and so on. I presented one of my own projects that achieves as much, also crossing the hardware-software interface off the list of top-level specification complexities, by verifying the hardware, too.

Another great example from programming is memory safety. Here we are dealing with what is, in practice, a somewhat-nebulous property, governing how different pieces of code can share computer memory harmoniously, with clear rules for who is allowed to access what. Everyone knows C is not memory-safe, while Rust is memory-safe – but what does that statement really mean? We could try to put together a grand unifying theory of memory safety, but I argue that the real importance of memory safety is that it allows modular reasoning. That is, in the presence of memory safety, it is tractable to divide a program up into modules and give each one a separate specification and proof, without unreasonable interaction across the specifications, because we know modules won’t generally trash each others’ memory. Therefore, the mere ability to prove interesting behavioral theorems modularly demonstrates a desirable quality that subsumes what we really wanted from memory safety – and we avoided needing to define “memory safety” formally. (Incidentally, others have formalized a related intuition.) This observation is related to the one I wrote about where resistance to security vulnerabilities follows almost for free.

So, even if humans are central to the story for what constitutes AI alignment, it need not mean that the “top-level specification” of AI says anything about humans! Future AI systems could construct sophisticated models of humans, and the utility of those models to meet some ultimate goal could be proved from first principles, without putting those models in the trusted base. While we humans differ from programming languages in having evolved rather than been designed, the gap could perhaps be made up by formally verified AI scientists studying humans or whatever other relevant phenomenon, a topic I hope to return to in later articles.

Conclusion

Maybe we just won’t be smart enough to construct an initial specification of alignment that avoids bringing in complex modeling of humans, but I hope I’ve made some kind of case that it’s worth trying. I also want to reiterate that I’m using this article partly to get feedback about existing writing I should check out for its connections to (and potential poking of holes in) what I’ve written here. My next article will give one example of an approach to alignment that genuinely avoids including anything about humans in its top-level specification, while plausibly meeting our informal requirements for alignment. The last article in the trio will consider the infamous example of the paperclip maximizer and some potential mitigations against allowing one to develop.

Just Another Increase in Programming Abstraction (Kind Of?)

Some of the points I made are unusual enough that they deserve repeating, since much of the feedback assumed I was suggesting a flow closer to conventional approaches than I intended. Let me give a highest-level recap that sets up that discussion productively. I made two claims, one that is absolutely orthodox in Silicon Valley today and one that represents an edgy departure. The orthodox claim is that powerful automation of software development allows us to think of software code as no longer expensive to produce, to the point where we can routinely throw away all of our code and regenerate it, following improved requirements for what it should do and how it should work. Almost all of the excitement around that direction leans heavily on LLMs and assumes that we generate software from natural language (which includes formats like EARS that add modest structure on top of natural language but still require natural-language processing to interpret). My edgy claim is that we need to ditch natural language entirely, for periodic regeneration of large code bases to be practical. Unambiguous specification languages are going to be a crucial ingredient well-known from formal verification but not deployed nearly as widely as LLM-based tools (despite decades of head start for formal methods!). (There is some subtlety to that last point where we want to retain some benefits of LLMs to help understand informal requirements, which I’ll return to later in this article.)

Some of the comments argued that a good unambiguous specification language becomes “just another programming language,” making the argument here overblown. Indeed, I used exactly that historical framing in the original article! I used the example of how the software-development process used to include drawing flowcharts, relatively informal diagrams that were not processed by computers. Then human programmers (separate from the analysts who drew the flowcharts) translated them into assembly language. Eventually, however, automatic compilers were developed, to translate from new high-level programming languages like FORTRAN. A program “planning” format was much more pleasant when further implementation could be automated.

I wrote that a move to writing “specifications” instead was a natural progression in that direction. We had just been stuck waiting for the right technology to automate implementation from there! Still, what exactly makes a specification different from a program? They sit on a spectrum, and sometimes it makes sense to use the same languages to write both.

However, I’ll argue that “specification” usually implies two important characteristics that are unusual in “programming.”

• A specification is written to explain what functionality is desired, not how to accomplish it. The most concrete and familiar example is performance, like how quickly a program runs. Much of the complexity of code in practice comes from cleverness in data structures and algorithms to improve performance. I would argue that a great specification should spell out requirements on performance (and indeed a variety of projects have shown how to prove performance requirements of real code, as in one of my own recent papers) but not get into detail on how to meet the requirements. In many cases, specifications can be much simpler and easier to audit precisely because they don’t precommit to optimization details.

• A specification often includes nondeterminism to provide flexibility to implementers. We can again find some of the clearest examples in connection to performance. Think of some process that generates a set of answers, and the user is happy to receive them in any order. Depending on how the underlying data were stored, some order may be the most natural, as in reading off the cells of a data structure in the same order as they appear internally. It would be bad for a specification to mandate a particular order, because then it becomes harder to take advantage of a clever new data structure that changes internal order.

A great example to illustrate both points is SQL, the popular database query language. It is possible to write an SQL query that is close to the most compact, straightforward way to describe what constitutes a correct answer. (Cue here complaining by programmers about the concrete syntax of SQL, yeah, yeah!) SQL also gives database engines freedom to return answers in convenient orders, when order isn’t explicitly mandated in a query. Unfortunately, SQL queries do not typically include explicit performance requirements, which could help database engines make tradeoffs intelligently, but I hope future systems get there.

Now, SQL is also a great example of how “specifications” and “programs” live on a spectrum. SQL power users often spend inordinate amounts of time tweaking queries to improve performance. One canonical example is declaring a new index to the database engine, a suggestion of a data structure that may be useful to speed up queries. The hardcore version involves deep inspection of query plans to find bottlenecks in the code that the database came up with. Nonetheless, “pristine,” high-level queries remain highly effective as specifications: easy to read, leaving much flexibility to explore different implementation strategies. It would not be entirely misleading to restate the central point of my last article as more of programming should become more like SQL (including the possibility to start with relatively abstract code, often get satisfactory implementations from it with full automation, and sometimes invest in more-manual improvement that involves adding details to the specification within the same language).

Note how the last little example is a template for how to handle improvement of specifications in other dimensions beyond performance. Through either predeployment testing (of the specification or generated implementation) or observing faults in the field, we can notice and fix specification problems. Examples of other dimensions particularly easy to misspecify are security, privacy, ease of maintenance (DevOps-style), and human usability. Just as with conventional software development, we should expect to keep improving code concerning these aspects – just trying to keep it succinct and put it in specification instead of implementation.

It turns out this style of thinking has a long history in basic research. To start with, we have the relatively informal (or at least typically disconnected from rigorous proof in practice) discipline of requirements engineering. More relevantly, the idea of refinement of specifications into implementations has been studied since at least the 1970s (see a paper by Niklaus Wirth, inventor of the Pascal language). I’ve been working in that tradition myself, including in the Fiat project, where we even used SQL-style notation as a central example for automated refinement with proof.

Relatively declarative configuration languages, say for cloud platforms, are another example that came up in discussion. Amazon’s Zelkova was an early success for formal methods in this area. I think of such tooling as important, but it also seems to solve a fundamentally simpler problem than general software development, and it’s the latter I want to focus on.

The Software Lifecycle with Strong Automation

I hope this framing already suggests some of the details of a future workflow based on unambiguous specifications, because it will look a lot like programming with SQL looks today. Database engines produce query plans automatically from SQL queries. As a result, we save the SQL queries in version control systems like Git, the canonical way of recording code over time, but we don’t put the query plans in version control. If we need to give the query planner instructions, say by requesting indexes, those instructions do go in version control. The point is that what persists, what is not a throwaway byproduct of automation, is exactly the information that makes remaining code generation sufficiently predictable, allowing it to meet our requirements reliably.

One important point I’d like to make here is that remaining stages of code generation don’t need to be deterministic. If requirements were spelled out precisely enough, we can be happy to receive any final, running system that meets the requirements. It’s OK if we get a different one each time! Programmers are used to mandating more determinism than strictly required by their development and maintenance processes, because they aren’t used to writing sufficiently constraining specifications/programs. The earlier SQL example is a good one: because we don’t trust the database engine to choose the right data structures and algorithms, we worry about changes to the subsystem that plans query execution, but what if we annotated our queries and indeed our whole programs with performance requirements instead? Then nondeterminism becomes desirable because sometimes the programming tools find new and better, faster implementations and drop them in without needing to bother us!

The underlying issue is that programmers worry (justifiably) that some important characteristics of the final system can’t be predicted from the source code alone. Yet, because those characteristics are important, we need to make sure we confirm they are suitable. The most common tool is testing, subjecting the generated code to many scenarios and checking it behaved properly. Over time, developers build confidence in even the parts of generated systems that those developers didn’t write themselves, by running many tests over many versions of the program. If a clever programming tool completely changes up those aspects, we have to restart the confidence-building exercise.

I want to emphasize the two big changes to this flow that I’m recommending. The first is to write specifications so precise that literally any realization of them is acceptable. The thought experiment is for, say, a company building a product to imagine that its biggest competitor gets to write any program that meets the specification. The second important change is to rely on machine-checked mathematical proof that a generated program meets its specification. Now we need not rely on testing to find internal bugs in the generated code, as testing covers only particular execution scenarios, while a proof covers all of them. The risks that remain are in the specification, perhaps most importantly in how we characterize the other systems that ours interacts with – so testing should be focused on that interface boundary, and it can be done against a specification that happens to be executable, not the generated implementation. We no longer need to treat e.g. a tricky concurrent data structure that is purely internal as highly suspicious from a bugs standpoint. This shift is similar to the one that happened with programmers trusting the implementations of higher-level programming languages (though there’s no need for trust, as compilers are a great subject for formal verification).

Another concern came up for having tools choose data structures automatically. How does a program evolve over time if the data format keeps changing, leaving prior databases obsolete? This problem is actually addressed pretty naturally: part of the specification for the next version of a program is that it is able to begin with the old contents of a database, where now the specific format of that database is part of the specification of the new version. The code-generation tool may now either keep the database format the same or generate a migration to upgrade to a new representation. (Note that this concern helps us notice a wrinkle of needing to save generated data schemas in version control or similar, even if they were produced automatically.)

Feasibility of Specification and Verification

Some concerns were expressed about scalability of such methods to very large code bases. It’s important to keep in mind which measurements of size we should worry about. It would be strange for a conventional project to measure code size by the amount of machine code generated by compilers. Instead we measure source code, which is checked into version control. Similarly, raising the level of abstraction, we should be worried about the size and maintenance complexity of specifications. Different programs may have vastly different size gaps between specifications and machine-executable code. We’ll have to see how the specifics turn out, but one orienting principle I’m comfortable asserting now is that such a gap often corresponds to the sophistication of performance optimizations in a program (which shouldn’t also be in the specification), so the gap is likely to be widest in domains with lots of attention to performance.

Some readers were skeptical that the LLM-style experience of getting a pile of complex code written on command can ever dovetail with high confidence in correctness – and I actually agree! I’ve argued that deep learning is better understood as a generalization of search engines than as a “reasoning engine” in the colloquial sense. Coding agents are doing something very similar to copying and pasting from existing code bases into a new one, losing the lineage of vetted good ideas. Instead, I’ve already argued for combining reuse of flexible components with AI code generation, where that generation now just needs to reference components by name. The novelty density of all software worldwide is much lower than the novelty density encountered examining all the parts of a single system, and we should be taking advantage of that low novelty to share carefully vetted code across projects. It should dramatically decrease the program-specific reasoning and proof that needs to be carried out. The effort to build, specify, and verify the components scales well with their potential many uses.

Another dimension of skepticism was the effectiveness of formal-verification tools in checking whatever code has been produced. There is a definite tradeoff between accepting a buggy program or failing to validate a correct program (see our past discussion on undecidable program analysis). In cases where the stakes are low enough, it may make sense to adopt a flow that occasionally lets a bad program through, if it enables more development velocity and creativity by coding agents. However, I personally most favor methods that don’t separate code generation from verification. Correct-by-construction techniques interleave correctness reasoning with all steps of writing code, and I’ll have much more to say about this approach in upcoming articles.

Coming up with a sufficiently constraining specification can be hard and error-prone work. Familiar quality-assurance activities for software may switch to specifications instead, including running specifications against many test cases. Tools for simulation, debugging, code review, and so on can also be very helpful for specifications – and all the above practices are actually already common among users of formal-methods tools like proof assistants. However, I would say most engineers not familiar with formal methods are much too pessimistic about the paths toward high-quality specifications. The reason is that most specifications apply to components that ought to be internal to larger systems, and end-to-end verification of full systems catches specification mistakes in internal components very reliably. For instance, a programming-language implementation is tough to specify in isolation, as we need to legislate exactly what each language feature means, but when that language is used internally in the proof of a larger system, we force ourselves to catch any language-design mistakes that are consequential, falsifying what we hope is a simpler top-level specification of an application, and the language definition becomes untrusted (in the good sense of not being assumed correct). There’s also a handy bonus for catching certain security bugs by proving almost any interesting specification about a system.

We can also keep benefiting from LLMs and similar future tools. Coding assistants can be used to write specifications that are then code-reviewed, tested, and so on, at the level of unambiguous specifications. There is an established research area around programming by example, and such techniques could be used to help write specifications automatically from input-output examples or even, with AI assistance, “user stories” that spell out in English ways that programs should be able to be used. The point is that AI can make mistakes, but the final quality assurance is done against an unambiguous specification (which persists in version control, while natural-language dialogues may not).

Aspects of the pushback about nondeterminism of code generators do indeed sound like trouble to me. I wrote previously about how nondeterministic specifications can promote security vulnerabilities through so-called side channels. It also isn’t hard to see how getting the same answer every time can be helpful, for instance in reproducing data analysis from scientific experiments. In such cases, we can make sure that top-level specifications are deterministic, though it may remain useful to allow systems to be built out of components with specification nondeterminism that is hidden well at the system level. A great example is a low-level memory allocator that assigns memory addresses to different pieces of data, where higher-level code is written carefully not to reveal which addresses were assigned. It pays off to allow new versions of the allocator to introduce clever new algorithms that affect address choices.

There was also a question about whether specifications are just pseudocode, and I think they probably aren’t. That terminology usually implies some use of natural language to explain important behavior, which isn’t compatible with truly reliable automatic programming.

Remaining Questions

One interesting question I received was about the place of undocumented features in software that is regularly regenerated. That is, conventional software ships with incidental features that the developers may never have meant to promise to retain indefinitely, and yet users come to depend on them (as laid out in Hyrum’s law). If regeneration can change such incidental features arbitrarily, users may repeatedly be disappointed. For now, I think my strongest response is that undocumented features are bad, and automation may be able to help us avoid them! However, it’s worth thinking about the consequences of users increasingly becoming AI agents themselves. One of my earlier articles analyzed how the problem of ensuring happy users becomes easier then, for instance because new releases can be checked against the code of known users, to make sure they remain satisfied in all possible scenarios. Achieving a similar benefit for human users remains an interesting software-engineering research question.

Complicated systems must often be debugged in the field, as much as we may hope broad formal verification catches all defects. Some bad behaviors only emerge at scale in real environments, which may include components not yet specified or verified. Generated code can still be provided with, for instance, mappings of running code to the parts of specifications that determined it, along with whatever other annotations best support debugging by humans or machines. Indeed, the quality of debugging collateral could be mentioned explicitly in a specification! However, the details definitely constitute an independent research challenge.

Another interesting comment was about how the world and the connections between its pieces seem on an unstoppable upward trajectory of complexity. Methods based on code generation from unambiguous specifications may hit an applicability wall, past some world-complexity threshold. It’s interesting to speculate about how increased adoption of AI may change this phenomenon. On the one hand, “smarter” AIs may go even further in designing complex, society-scale systems. On the other hand, the selection pressure to enable efficient reasoning by AIs may motivate them and us to create bubbles of improved legibility within which economic systems and reasoning systems are codesigned. The world may seem to grow increasingly complex and illegible to humans while AIs are actually improving their ability to keep up!

We all see how amazingly effective LLM-based coding assistants have become, even if it can be hard to build confidence that they wrote the programs we really want. There is an even older tradition of research in program synthesis. I think there’s a lot of exciting technical work to do to draw on these starting points and create the most effective systems for automatic programming. However, it was an intentional omission not to go into more detail in the prior article, as I’m going to need multiple articles worth of content to do that subject justice; stay tuned. For now, just imagine that some reasonable notion of mathematical program proof is fixed, with ideally one standardized trustworthy checker, opening the door to a variety of methods that produce proofs (in the style of proof-carrying code).

I do want to tease now that I see performance of logic-based automated-reasoning tools as a major bottleneck holding back the area. At a minimum, we need these tools to keep up with deep learning-based tools that may be writing code creatively. A major theme of future articles will be full-stack performance engineering to improve logical reasoning and code generation. In fact, I hold out hope that logic-based alternatives will be able to run even faster than purely LLM-based methods, even if LLMs are baked into custom hardware, because of the inherent performance bottlenecks of deep learning.

Next, though, I want to turn to perhaps the hardest case of complete-enough specification being discussed today: AI alignment, the study of how to build flexible automated decision-makers that we should believe will respect our values and not harm us in surprising ways. My impression is that the conversation so far leans very much on thought experiments not connected to implementations, somewhat justifiably if we may (?) not yet have AI powerful enough to exercise the alignment challenge very thoroughly. I’m going to analyze the problem from more of an engineering perspective, grounded in experience building and maintaining significant proofs of system correctness. The gimmick will be thinking of human brains as a legacy system that is worth engineering around if we can, especially if it must be mentioned in the top-level interface of a system.

My answer is an instance of a pattern I plan to return to occasionally. Let’s think of the global economy as a distributed system solving an optimization problem to organize the world properly. One of its tools is signaling, where people produce costly, hard-to-fake displays of competence, which are used to evaluate them and partly decide where to slot them into the economy. Clearly that evaluation process produces massive downstream value. However, it also produces some pure “heat” with resources expended on signaling without directly moving the primary metrics of interest to the economy. In biology, signals probably evolve in the first place thanks to their connections to fundamental survival and reproductive success, even if runaway selection can take them far from that territory over time.

With our present-day abstract-thinking capabilities, we can choose to realign signaling deliberately so that it not only provides competence information but also generates first-order economic value. I’ll use the productive signaling header for articles that suggest instances of this pattern. Today we’re considering how to help spur widespread regeneration of important software.

Get Them Writing Code That Matters

Among top students worldwide in high school and a bit younger, there is a tradition of participation in olympiads like the International Olympiad in Informatics (IOI) or International Mathematics Olympiad (IMO). The basic form is hierarchical competition in particular STEM areas, with more localized competitions choosing winners to advance to competitions with broader geographic scope. For instance, the IOI challenges participants to write computer programs quickly that implement provided requirements, generally focusing on problems framed in terms of simple discrete mathematical objects, with straightforward specifications but nontrivial solutions. (It’s the subject matter that dominates classes commonly called just “algorithms,” but the real scope of different kinds of tricky algorithms worth writing is much broader than we see in classic competitive programming.)

These kinds of competitions are very useful for identifying stars in the respective areas of knowledge. My own institution MIT seems to lean heavily on olympiad records to make decisions in undergraduate admissions, especially for international students. That is, olympiads produce a simple quantitative signal (based on standing in international contests) that is highly legible even to nonexperts, like people in charge of undergraduate admissions. I believe the signal being measured is also very relevant to potential to succeed in related careers. And olympiads are not just a measurement process: training for them helps upskill students and genuinely prepare them to contribute to important social problems in not too many years. We should also acknowledge that, given the centrality of signaling in our own evolution, it’s extremely common for people to hunger for chances to compete successfully and show off their talents, and we should be careful about letting video games become the default outlet!

Call me greedy, but, even given those benefits, I think it would still be fabulous if this experience also helped solve social problems directly. We expect to see approximately a few hundred competitors at a typical olympiad world-finals event, and they are all solving the same problems. The duplication is even greater for the sum of all more-localized competitions that determine competitors at world finals. We only need one competent team to solve a problem to realize the full social value of generated code for that problem! So the first big problem with this structure is that, from the standpoint of producing useful code, it creates massively redundant work, where we can subtract essentially all teams/competitors but the most-successful for a given problem and achieve the same practical outcome. We can see why this uniformity evolved, as it makes judging tractable, but hold that thought.

The other big problem is that solutions to the problems assigned in olympiads rarely create immediate social value, independently of signals about competitor competence levels. Pretty much by definition, these problems were already solved when the competition started! The organizers would be embarrassed to assign an unsolvable problem, and that pitfall is hard to avoid without solving in advance. There is also the scale of the problems that makes them unlikely to have big social impact. The time limitations of these contests push toward limited scope, considering how important real-world problems often occupy teams of software engineers for years, even before first wide releases. AI coding assistance is changing that equation, but please also hold that thought!

My alternative proposal is a competition structure around creating useful new open-source software. Developing a new and better program in an existing category is explicitly encouraged, and in fact it connects to our initial motivation to kick off widespread regeneration of important software, drawing on new wisdom in cybersecurity and elsewhere. Olympiads produce many variants of the same program, but they are all promptly thrown away, while I’m suggesting a contest that develops variants with increasing real-world value. If we can just judge competitors properly and summarize results in a form highly legible to nonexperts who make decisions like undergraduate admissions, then we are in business: we have harnessed signaling in a way that produces a socially valuable first-order output, not just the handy signal. (As a quick parenthetical tangent, AI is changing the educational landscape so much that it isn’t clear the admissions process we know today will last much longer, but if it goes away and isn’t replaced by other similar talent evaluations, humans have probably stopped being competitive to do knowledge work. Oops!)

I should add that, through showing a draft of this article to an LLM, I learned for the first time about Google Code-in, which brokered between high-school-ish students and open-source projects that provided tasks within their code bases. On the one hand, I frequently interact with students just past this age range applying for research roles, and since I hadn’t heard of Code-in before, it is unlikely to have become a common tool for signaling programming ability. However, the fact that it continued for 10 years, with almost 15,000 students participating, provides some directional evidence that this sort of initiative can scale. Like the Google Summer of Code, which brings students into established open-source projects for quasi-internships, Code-in focused on projects that had been around for a while. Code-in wound down in January 2020 (did they know the pandemic was coming?!), just a little too soon to benefit from the new wave of AI-coding tools. It is already very feasible for the best-qualified students to create their own useful new software packages from scratch, and we should take advantage of that more-granular signal. Maybe it is only in-reach each year for approximately the number of students who participate in olympiad world championships or even earn top distinction in them, but that’s still an important population. Also, while that contributor count may not represent a big-enough workforce for e.g. a wide retrofitting of important software with new security protections, we can hope that advances in AI coding tools grow the population with time, and the contest structure can evolve to take advantage. (In this case, there is a helpful correlation, where when the risks from AI models finding vulnerabilities grow, the tools probably also improve to allow larger contests to maintain quality standards.)

Implementation Challenges

Which details need to be nailed down to make this kind of competition work?

Open-source maintainers are already suffering from firehoses of AI-generated slop “contributions,” and you want to add to the problem? Don’t worry: this proposal is for creation of new software projects and doesn’t require interfering with the normal workings of established ones!

We are talking about creating programs worthy of mass adoption. Do we really want to entrust that task to high-school students? I’m not talking about a standardized test that all high-school students need to take. A very small fraction of high-school-equivalent students worldwide participate in the feeder programs for IOI. It only needs to be feasible for that echelon of students to build programs that at least make good starting points for broader open-source contributions, perhaps overwhelmingly by professionals, after improved fit for an important problem is demonstrated. And they get to use all the latest AI-powered software-engineering tools! We are still getting used to the power and engineering implications for those tools, but I don’t think it’s crazy to imagine small student teams or even solo competitors making really valuable and solid stuff over manageable time periods (which can nonetheless be at least months long, in stark contrast to durations of olympiad contests). Also, naturally, all of these efforts should be associated with formal verification that permits checking of quality for resulting code, but let me not get too sidetracked by that hobby horse.

So we have each team writing its own new program, covering a wide range of domains to maximize social value on success. How on earth do we judge the results effectively? Just getting AI to do the judging is probably a helpful start but also vulnerable to pernicious gaming of the system, so we are going to need human judges. (Again, maybe we don’t need human judges, but then we’ve probably crossed over to the regime where, for better or worse, we don’t need to train humans in programming or evaluate their skills in it.) We can take advantage of the same hierarchy already found in olympiads: a succession of increasingly global competitions, where wider-scope competitions are fair to assume as including competitors with higher average competence. The earliest stages could even keep the conventional programming-contest format, switching to developing novel programs later on, when we don’t need as many judges. Competitors need to earn the reputations that their programs are worth evaluating. However, also think of companies eager to meet the best potential hires early. They may volunteer their engineers’ judging time, even relatively early in the hierarchy of competitions. It may even be possible to crowdsource evaluation massively, using a metric like GitHub stars to judge outcomes, though the chances to game that metric (e.g. with sock puppets) are clear. A proxy for real economic value derived by users would be ideal.

You’re talking about evaluating the program that is produced, but we know that it’s actually AI writing most of the code these days, and the extent of that phenomenon will only increase. How do we know judging isn’t really just evaluating how much token budget a kid’s parents endowed financially? If we get fully automated generation of valuable new open-source software, that’s also a great outcome! Then we probably don’t need mass training and evaluation for programming skills anymore. If an important role remains for human guidance, then capacity in that role is precisely what we want to measure and what a judging process here should be able to get at. That still leaves what seem to me the biggest challenges to resolve, including the difficulty of effective expert review of reams of AI-generated code, to assign scores. Maybe this domain could be a good testing ground for techniques that will matter even more for industrial software-engineering teams? We can also hope costs of AI coding assistance are headed downward (though I have my suspicions that this goal will require moving away from deep learning, which is inherently slow and expensive).

OK, but, seriously, what are the human judges actually doing to determine what skills the contestants have demonstrated? I agree it’s a tough question! We can see clearly the immense payoff from traditional programming competitions giving everyone the same problems and grading based on test cases passed, though the cost is not just lack of production of adoption-worthy code but also practice only on programming tasks that are unrepresentative of most real development in important ways. For now (before neural implants, say!), it should be safe to have face-to-face interviews with students, going through their code and asking the right questions about it. It’s a labor- and time-intensive process, which is why it’s important that relatively few contestants reach this level of the competition. We would also need some kind of rubric, a scoring formula that helps address the massive heterogeneity in what creates value across different kinds of programs, and this rubric also sounds nontrivial to create. In general, I think the saving grace is that this scoring aspect doesn’t need to be very precise; it just needs to identify a cohort of participants who have demonstrated enough competence to be worth inviting into the next-higher level of participation (e.g. university admission or a first job). Also, it should be possible to borrow good ideas from science fairs, which also involve judging within a pretty open-ended design space of projects selected by students and also are already recognized as a somewhat common signaling currency by university admissions.

Are there even enough high-school staff out there who are qualified to help students succeed in this kind of competition? Maybe not, but there’s a decent chance that generative AI provides a good substitute! Students can go beyond just asking AI to write code by also asking AI questions that prepare students to do more work on their own.

How will students find which problems are worth solving with software, anyway? There could be some shared online resource with proposal of problems and voting on which ones matter, by suitably vetted audiences. Companies could propose problems alongside offers to judge solutions, perhaps augmented by something like automated test cases. However, especially with the way things are heading with AI coding assistance, choosing the right problems makes up an increasingly large fraction of effective software development. We may want to measure exactly this skill, even if it is very rare in the student population, even if research to find the right problem comes to dominate time spent on the competition!

How will students know they should spend time on this weird new thing? It’ll take some time to bootstrap marketing, I’m sure. Our brains evolved to expect social contexts like those in hunter-gatherer bands, where sets of skills worth learning stayed remarkably constant (from a present-day perspective) even over thousands of years. Clear guidelines and developmental rituals were provided, to guide young people through skill development. Olympiads have tied into this cognitive machinery today: students look around for what people do to develop skills and get credit for developing them, and they head for the competitions suited to their talents. Schools build up resources to help students and encourage them into the proper paths, motivated partly by the school community’s chance to get some signaling credit for elevating champions. If there are enough/important evaluators out there ready to use this signal, the rest should follow over time.

Conclusion

I sneakily saved a disclaimer for the end. I personally stuck with competition programming only minimally in my high-school days. Instead, I spent my time writing programs and getting people online to use them. Readers should be suspicious that I’m arguing for a change in signaling regime that disproportionately benefits people like me. I hope there’s still enough of a case that the social benefit is large!

It’s interesting to speculate (but I’m less qualified to do it) on olympiads in other domains. An especially intriguing one for me is math. Do we want students all going after different open problems to prove with Lean? Is it even conceivable that this early-career cohort could exhibit collective productivity comparable to what the whole international math community was pulling off five years ago?

There are plenty of logistical challenges left to sort out to implement these ideas; my goal is just to get the conversation started. My next post will return to the idea of frequent, rapid, automation-supported redevelopment of software, trying to fill in some of the holes pointed out by the great reader comments on the previous article (ironically, that link won’t take you to them, but they’re on LinkedIn, GeekNews, LessWrong, and ACX).

I’m going to make the case now for an even broader opportunity. We need to stop thinking of production-ready code as a scarce resource. It may take a few years to get the tools up-to-snuff, but we’ll reach a point where the cost of ongoing reimplementation of significant code bases drops to the levels associated with SaaS subscriptions today. Let’s start preparing ourselves for how software should be built and maintained in that world. Whenever there are innovations in security concerns that all applications’ designs should address, we will be able to regenerate all of their code to comply. When new algorithms for familiar problems are invented, the cost should be near zero to regenerate using those algorithms, even if they require small customizations. All sorts of different requirements dimensions are fair game for incorporating in fairly frequent regeneration of all code in a project.

The idea that legacy code will steadily lose value, as it becomes easier to replace automatically, is becoming pretty mainstream, at least in the Silicon Valley hype ecosystem. In fact, the strong version of the thesis, which I endorse, is that code as we know it will become a throw-away automatic byproduct of the real long-lived artifacts, joining formats like assembly language that we mostly think of in that way today. I’ll start by making my version of that case. However, there will be a twist! I don’t think the story is nearly all about generative-AI methods that are popular today. Instead, formal methods will be crucial to realizing the strongest version of this idea, because they enable truly automatic generation of full code bases without human oversight.

Automatic Programming

It’s not nearly a fringe viewpoint today that programming will soon be done almost entirely by AI code generators that start from natural-language requirements. The goals and methods line up to an extent with the longstanding program synthesis area of computer science, though the details have been shaken up a lot by machine learning relatively recently.

A good book to help understand the present situation is The Computer Boys Take Over. It covers the rise of “automatic programming” and programmers’ indignation at having it commoditize their skilled labor. A quote from a conference report covers the idea that “managers can now do their own programming” (though the quoted writer expressed the concept with skepticism while noting it was in the air).

However, there’s a catch: the quote is from 1962! The subject is the introduction of the first high-level programming languages and compilers, which automate the tedious writing of code processed directly by computer hardware. Programmers today just take for granted that they write in high-level languages, and pushback against AI coding tools has to do with the writing of those programs, not the ones that 1960s programmers were worried about being generated automatically. I would argue that the lesson we should learn is that the level of programming abstraction in software development is constantly rising, and we should not be surprised to see a step change toward an even higher abstraction level today. Indeed, David Parnas made that observation no later than the 1980s.

Let me spell out the framing a little more. There are many different ways to describe a given program. We get used to thinking of some languages as for writing “implementations” (e.g., Python) and some for writing “specifications” (e.g., various flavors of formal logic). However, I’ve found such a hard-and-fast distinction to be counterproductive. Let’s just think in terms of lower- and higher-level ways to describe functionality.

So the overall idea is that we want to describe programs in as concise, high-level ways as possible. Instead of storing conventional code in repositories, we would prefer to store the highest-level description of requirements that we can get away with, where available tools reliably automate the rest of bringing the desired program to life. Some nonfunctional requirements may change, in which case we can relatively simply fold them into the requirements and redo generation. Security is a good example that the last article covered, with examples like taking high-level secrecy policies and applying them on top of newly discovered vectors whereby programs can leak information. Another great example is performance, where I’m actually pretty surprised at how rare it has been for practical programming languages to support annotations that spell out performance requirements, so that it’s a compile-time error if code can’t be produced that is fast-enough, uses little-enough memory, etc.

In general, the game is to capture requirements unambiguously enough that any program meeting them will be acceptable, allowing truly automatic regeneration of running systems after light-to-moderate requirements changes. A thought experiment might be useful to capture the model. Imagine that some government routinely bids out software projects to contractors, which see each other as bitter competitors. What if every project went to two contractors, one who writes the specification and the other who gets to write any program meeting the specification? A specification gets written defensively, assuming the least-charitable reading by an implementer. In fact, the same specification, or its small evolutions, can be handed to a succession of adversarial implementers, which can, in theory, be an entirely safe way to maintain a program.

If it works, then established code bases of the conventional kind shrink dramatically in value. It becomes unclear that one organization can have an advantage over another on the basis of its stockpile of code written in conventional languages. What matters instead is having the right specifications. We reanimate them in new circumstances just like we recompile code today when libraries it depends on have been upgraded. I like that analogy, too, for connecting to how we might mix in new security requirements or new algorithm ideas, to be applied automatically in regenerating lower-level code from a specification.

Natural Language Won’t Cut It

The trouble is that the mainstream conversation is all about generating programs automatically from requirements in English and other natural languages. I’ve written before about how natural-language processing is perversely hard: not only are there inherent ambiguities that our evolutionary environment didn’t force us to fix, but there is even a strong argument that, to the extent we can think of natural language as “designed” by evolution, it was designed to be difficult to process, for its role in signaling. The related problems are so hard that I doubt we will ever have fully automatic generation of production-ready systems from specifications in natural language – and organizations that fall short of full automation will not be able to keep up in the new economy.

I argue it’s a historical accident that we wound up with today’s deep-learning stacks as the only ones apparently up to this challenge of comprehensive automatic programming. Certain features were helpful in computer graphics and wound up built into GPUs for that reason, and then GPUs turned out to be useful for deep learning. Epic amounts of effort on full-stack performance engineering have gone into the domain, but we can make the same kind of effort to support better code-generation systems.

Let me give a brief set of pointers to my past articles making the case for a different, logic-based paradigm for automatic programming. Deep learning is very effective as a kind of souped-up search engine, but the match with the full process of programming is strained. At a fundamental level, deep learning implies serious performance bottlenecks, and it may also be impossible to prove mathematically that deep learning-based systems always meet our requirements. The power of these systems comes from analyzing large numbers of examples, and, for many domains, we happen to have enough examples only in natural language, but we need not restrict ourselves only to techniques that need to learn from examples. A related idea for automatic programming is curating libraries of flexible, reusable software components, which AI can then invoke by name instead of applying via emergent patterns from training. An even-more-powerful idea is codesigning decision-making systems with the problems they solve, where we should be able to avoid some of the most standard challenge problems for AI, leaving mostly problems that we can solve by generation from relatively compact and unambiguous specifications. While the standard retort to so many varieties of formal methods is that the real world is complicated and hard to specify, economic forces will push toward simplifying the world in pockets of activity given over to automation – and over time, the fraction of the economy covered by such pockets will grow, and pockets will merge with each other. We should expect collecting unambiguous requirements to get much easier in that world.

Why do we need to push beyond the spec-driven development techniques that are being built out today via a variety of tools, taking natural-language documents as “the new source code”? One instructive spec format from this domain is EARS (the Easy Approach to Requirements Syntax). It provides an alternative to freeform requirements as in the following sentence.

As long as the vending machine isn't out of product X, when a customer presses the button for X and inserts payment equal to the cost of X, the vending machine should dispense one X.

In EARS, the requirement can be rephrased as

WHILE inventory of product X remains, WHEN the button for X is pressed, WHEN payment equal to the cost of X is inserted, the vending machine SHALL dispense one X.

The all-caps words are keywords, delineating parts of a template that has a rigorous formal meaning. The problem is that all the other phrases remain in freeform natural language, with all its ambiguity. An AI code generator can misinterpret those parts, through either innocent misunderstanding or crafty malice.

Tools for spec-driven development will often translate specs at the level of EARS into finer-grained specs, often with multiple steps of increasing precision, taking up significant human time to detect mistakes. We just won’t be able to realize the full potential of rewriting all the code, all the time if we create so many bottlenecks where human attention must be applied. Regenerating based on new requirements should be as push-button as recompiling programs in conventional languages.

Here’s a closing diagram to sum up the narrative of progress that I’m pushing. It involves (1) changing more phases of software development to use languages with clear formal semantics and, partly as a result, (2) moving automation (as opposed to expert human effort) earlier in the development process. Much earlier in the history of software, flowcharts were used as an intermediate program form with unclear semantics, and they were effectively replaced by high-level languages. Natural language-based spec formats strike me as that kind of hybrid beast that we’d be better-off replacing with a proper, unambiguous language.

Conclusion

I’m the kind of optimist who often argued for throwing away all the old code and rewriting it in a better way, even before new AI-powered tools dramatically lowered the costs. When I started graduate school, “everyone” seemed to agree that the x86 architecture had achieved such dominance that serious work on programming tools had to work with x86 binaries – and then we had the rapid rises of both mobile platforms, which sported alternative architectures like ARM that were more energy-efficient; and web applications, which developed their own parallel universe of JavaScript implementations and no distribution of computer-native code. The applications in these ecosystems (and many of the tools behind them) were created basically from scratch over handfuls of years – and they didn’t even have AI to accelerate the process!

The payoff of a reliable flow for rewriting all the code is no longer having an excuse for not using the latest ideas for software quality everywhere. We should be able to take familiar cadences of, say, new software releases and instead associate them with complete rewrites of all code used at particular companies. The key is setting everything up for absolute minimization of the need for human oversight during a cycle of regeneration – and only formal methods seem up to that challenge. It’ll take reengineering of the stack for high-performance inference to work natively with formal logic instead of just linear algebra (though it may work well for the two styles to cooperate), which I started sketching recently (e.g. in discussion of hardware-software codesign that converges the two sides) and which will occupy many future posts here. There’s going to be a lot to figure out, just as the design and implementation of high-level programming languages became a sprawling enterprise.

My next article will suggest one sociological approach to kicking off the virtuous cycle of rewriting a lot of important code.

The AI-safety community has at least for some time (decades) talked around the possibility of powerful AI finding new security vulnerabilities in important code. The 2023 GPT-4 Technical Report called out that usage mode as a risk. In 2026, Anthropic prepared to release a new LLM named Mythos, which they had become worried was too good at finding vulnerabilities. As a result, instead of starting with a broad release, they initiated Project Glasswing, which only shared Mythos with a modest set of big-name tech companies, to get their help understanding the cybersecurity implications. Eventually, a weakened version was released widely under the name Fable. Then the next twist was the U.S. government basically declaring that even Fable was too dangerous and couldn’t be exported to other countries, a restriction so hard to implement that Anthropic removed Fable access for everyone.

Eventually, Fable was allowed back into the wild, but the whole saga got many more people thinking about cybersecurity risks from powerful AI. There was also a more recent headline-grabbing story, about how an improperly configured model-testing environment led to an OpenAI agent compromising a Hugging Face server. Let’s fight the gravitational pull of sci-fi sensationalism and instead talk through the fundamental changes to how software engineering should be done.

Deep Learning Finds Security Bugs

The problem is actually grounded in a mix of technology and economics. Standard software-development methods have always allowed security-critical bugs to survive unnoticed in important programs. Well before generative AI went big, there was a thriving market in zero-day vulnerabilities: each a security bugs that someone had found but that was not yet known to the owners of a program. A bad actor canonically buys a zero-day and uses it to stage one or a handful of attacks. Markets in zero-days were studied, and bugs in important programs would generally go for thousands to hundreds of thousands of U.S. dollars each.

Lurking bugs in popular software programs are one example of the antipattern of security through obscurity, where the maintainers of a system assume that aspects of the system are obscure enough that no one will figure out their security vulnerabilities. Even an open-source program can benefit from obscurity, where the code base is generally complex enough that few outsiders can decipher it well enough to pinpoint vulnerabilities. The less popular a software package, the more it benefits from obscurity. Obscurity is even more powerful for closed-source programs, which may be distributed only in binary form. Then a process called reverse engineering is needed to recover even the level of detail found in normal source code. The bad news is that generative AI is also proving to be useful for reverse engineering, so the obscurity bonus for closed-source software is eroding.

There used to be a major bottleneck in finding zero-days: the time of a scarce population of security experts. If the market values bugs in a particular program at a low enough level, the experts won’t spend their time finding those bugs. The emerging problem may be clear from that last sentence: if vulnerability discovery can be automated by LLMs instead, it often becomes economical to set them loose on relatively obscure code bases and still come out ahead. We have lost the old defense that most code isn’t economically important enough that its bugs are worth enough to justify the cost of finding them. Roughly speaking, the cost of vulnerability-finding is heading toward becoming negligible, as providers of LLM services optimize their offerings.

On the face of it, this development brings clear bad news for “the good guys,” the well-meaning creators and maintainers of software packages. Innocent coding mistakes frequently create security bugs, and it is no longer safe to assume that those bugs will be obscure enough that no one will invest in finding them. However, this state of affairs pretty directly implies a great way for software engineers to defend themselves.

Assume an attacker has a cheap, automated way to find security vulnerabilities. Then we can transform that method into an effective way for software developers to find vulnerabilities before releasing code – maybe even before allowing it to pass continuous integration (the sanity checks often placed between new code and other software developers on a team).

Now assume that the attacker still has a cost-effective way to find bugs using LLMs or any other readily available automation. It follows that the software developers can adopt the same tooling as part of their quality assurance, and it remains cost-effective. That is, defenders are in good shape so long as they have access to roughly the same scanning tools as the attackers. We might need to assume that legitimate software projects are at least as well-funded as the attackers, which isn’t always true, though it’s increasingly likely to be true enough as the cost of LLM-based methods drops. For maximum benefit, we must also assume that attackers don’t get head starts, with earlier access to new automation. (We’ll come shortly to addressing the further asymmetry that attackers often win over defenders when both sides can only run tools that generate bug ideas without guarantees of completeness.)

If we are somehow able to ensure those two assumptions hold, then (modulo the last parenthetical) we are in even better security shape than before generative AI took the stage. There is a one-time adjustment process to bring these new tools into all existing and new projects, which admittedly is a large cost globally, and we will return to that challenge at the end of this article and especially in the next one. However, we should remain worried about the possibility that attackers repeatedly manage to get early access to new bug-finding tools, due to the same kinds of arms-race dynamics that arise for communication speed in high-frequency trading. Is there any hope for a security check to end all security checks, where we don’t have to worry that future tool generations will find bugs that their predecessors missed? So long as we rely on the mysterious black boxes that are deep-learning models, it seems we’ll always have to deal with rare but consequential mistakes, and security is a domain where even rare mistakes can be quite consequential indeed.

Enter Formal Verification

Readers who have been following along in the series are probably not surprised that I now bring up formal verification as the ideal solution! That is, if we produce machine-checked mathematical proofs that programs have the right security properties, those proofs cover all possible execution scenarios, and attackers can’t find bugs later, regardless of the growing quality of their analysis tools.

Let me be a little more specific. LLM vulnerability-finders are not different in kind from a variety of security-scanning tools that were already standard years ago, including the category of static application security testing (SAST). This whole category of bugfinding is often understood probabilistically, generating “good guesses” about where true problems lie. If any guarantee is present about finding all security issues in some category, it is usually paired with generating many false positives, to the point where developers hesitate to wade through them all and may just refuse to use the tool. (After all, we can write a guaranteed-sound “bugfinder” that just accuses every line of code of looking fishy!) The upshot is that automatic methods are not able to draw attention to all security issues; we should think of them as stochastically missing important bugs every now and then. Over time, as new tools are released (including those based on LLMs), they may converge toward reliably reporting all vulnerabilities. However, it is very hard to become convinced that, at a point in time, there does not remain an important gap of undetected issues, which attackers remain incentivized to find and exploit.

Formal verification can fill this gap, guaranteeing coverage of all instances of a bug category, with no inherent requirement to produce false positives. There remains a common objection by savvy practitioners (beyond the cost of proof-writing, which we’ll save for later posts): are we really so sure we know how to spell out what security means, with enough mathematical precision? The good news here is that there are a number of established counterintuitive phenomena of formal methods to help out. For one thing, proving almost any interesting property of a program tends to rule out some of the worst security problems. For another thing, though it may be hard to come up with the right properties to prove of a component in isolation, integrating proofs of different components is extremely effective at finding specification mistakes. The same techniques also help protect against bugs in all system infrastructure that an application depends on, because infrastructure layers can also be verified, and proofs can be composed across layers.

Still, there’s no doubt about it: writing specifications is a nontrivial engineering activity. Nonetheless, there is a force multiplier in specification-writing: it can contribute much more to security than careful software development, per unit of effort. Consider the following two stylized scenario variants, maintaining a software package with and without formal verification.

The upper scenario represents the mainstream approach today, where new security bugs keep popping up. Many of them are similar to past bugs, but human attention still isn’t up to catching them before they are released into the wild. The lower scenario represents a pivot into formal verification. Yes, there are still surprises of new kinds of bugs that sneak through. However, each time we learn about a new kind of bug, we strengthen the specification to rule out all bugs of that kind. With sound formal-verification tools, it then becomes impossible for those bugs to sneak through, even in the face of arbitrary programmer mistakes.

Even with formal verification, we still have the risk of serious zero-day bugs. However, if we arrange verification properly, then the zero-day window only opens for each new category of bug, not each individual programming mistake. We still worry that the bad guys get access to bug-finding software before the good guys and exploit a zero-day window, but there should just be many fewer windows that open. A window doesn’t open for each slip of the fingers by a programmer but instead only when an interestingly new kind of bug is discovered.

Formal verification has another asymmetric advantage over heuristic bug-finding: it allows us to move beyond having each development team suffer in isolation dealing with security problems, instead enabling significant shared effort toward blocking classes of vulnerabilities across all projects. It’s not that every project needs to discover and remediate a new bug category on its own. There are even opportunities for nearly instant updating of, say, all open-source projects’ security specifications, once a new bug category is discovered and announced by one project.

As an example, let’s use the classic security property of confidentiality within information flow. Some inputs to a system are considered to be secret, and their values should not be able to influence outputs that are considered public. Actually, there can be a lot more sophistication to the security policy than just considering inputs vs. outputs and secret vs. public values. Most generally, software developers define lattices of security levels. I wrote previously about how such concerns can be addressed for the case of compilers. We stepped through a series of increasing twists on top of the same basic confidentiality requirements: permitting nondeterminism in the specification opens up opportunities to leak secrets through choices that are left up to the implementation code, and allowing an eavesdropper to monitor timing can open up additional bad flows.

Just like software implementations, specifications should involve libraries of reusable notations. We can imagine a library for specification of secure information flow. The library can define a language for describing security levels, which each software project will use to specify its security policy, in addition to tagging inputs and outputs with levels. Then the library encodes how to connect that fundamental requirement to all sorts of different channels, including those based on nondeterminism and timing. In fact, the library starts out without coverage of those channels, but once it’s added, it becomes instantly applicable to all applications using the library. The situation is similar to what’s already in deployment with e.g. CodeQL query packs for pulling in new rules for static-analysis tools with weaker formal guarantees. Furthermore, if best practices are being followed through use of end-to-end verification, then infrastructure like compilers and operating systems can have its specifications expanded analogously, and the new application can have its specification stay linked with those of other components, for very efficient rollout of specified protection against a new attack family.

Now, updating a specification isn’t the whole story. Proofs about programs need to be extended to cover the new requirements. I wrote previously about how simplifying programming languages allows AI coding assistants to be more effective, including in meeting security requirements. If we simplify far enough, no explicit proof effort is required! I gave an example from our startup Nectry, which integrates checking of security policies into an agentic loop, so an AI writing software is getting constant feedback on security mistakes that it makes. (By the way, though it may not be obvious from the current Nectry web site both that we are doing this kind of formal methods and that we have a private beta running, in fact both are true, and folks at companies medium-sized and up can contact me if they think they might want to participate.)

I’ll also have much more to say in later articles about automating development of verified software, for broader classes of programs and specifications.

The Looming Challenge

I just discussed two main ways the rise of generative AI need not make software security more precarious than before – and can in fact help make security better (in the case of formal methods, through accelerating automatic proof-writing). The problem is that each method, LLM vulnerability scanners and strong formal verification, requires significant changes to software-development processes. Many legacy systems were developed before such technology was available, and we should worry about their security. Even new projects remain overwhelmingly carried out by teams not trained in these new methods.

What trajectory can we take toward getting these methods integrated into all software projects of nontrivial importance? My next article presents one framework with implications beyond security, considering how software maintenance should change when the cost of writing new code plummets.

We’ll stick to the big picture of design principles behind GPUs, rather than heading into a practitioner’s tutorial of GPU programming. I’m not even qualified to give the latter, considering that the CUDA programming framework that dominates this domain was released to the public in the same year that I finished my PhD, and I never picked up the experience later. We’ll analyze a simplified version of CUDA as what programming-languages folks would call an abstract machine, a somewhat-fictional model of what GPU infrastructure does that is both convenient for programmers and usefully close to hardware reality.

One more piece of grounding is helpful before I get into GPU details. This domain is one that takes great advantage of regular parallelism, where many parts of a computation follow a common sequence of instructions and just touch different data. (Examples are coming when we get into the main part of the article.) The lurking point behind this sequence of posts, and much of my writing here in general, is that making intelligent systems more trustworthy depends on computations that are not regular, as arise in symbolic domains like compilers and formal verification. In particular, it will be hard to upgrade our programming process to produce reliable code automatically if formal reasoning can’t keep up performance-wise with creative generation of programs.

OK, I’m almost ready to present the main ideas behind GPUs and highlight similarities and differences with the spatial-programming framework I sketched. One consistent theme will be that, unlike with CPUs, GPUs expose the right knobs for tuning performance in low-level code, and they largely do it by exposing programming abstractions that are closer to what physics gives us to work with in designing circuits. We will see that every distinctive feature brings some benefit in controlling costs of communication within a chip, which I emphasized as the key obstacle to scaling. By varying hardware and compilers more broadly, we might only be able to improve performance by constant factors, like the difference between processing n inputs taking 2n or 100n time steps. While computer scientists often perform only asymptotic analysis of performance, the constant factors certainly do matter. We’ll also discuss how different programming abstractions are more or less friendly for humans, compilers, or other automatic analyzers or generators of code.

Step 0: Vector Instructions

Before we actually talk about GPUs, I’ll cover a feature that is widely supported by CPUs: vector instructions, otherwise known as single instruction, multiple data (SIMD).

If many cores are running the same code but operating on different parts of memory, there is wasteful redundancy. For instance, the code for each common instruction needs to be copied into each core. Though we can control this overhead by achieving good memory locality via running the same small set of instructions repeatedly, the duplication of code in instruction caches across CPU-based systems is fundamental. Every core also keeps track separately of where it is in the program (its program counter).

In the SIMD model, shockingly, a single instruction operates on multiple data! Concretely, one instruction explains manipulations on multiple contiguous memory locations. Take the example of trying to increment by one every memory cell in a group of four. This diagram contrasts how it works with four CPUs vs. one CPU running a SIMD instruction.

Progress beyond old-school CPUs: One of the biggest obstacles to high parallelism with CPUs is the need to associate each unit of parallel work with a separate processor core, a relatively heavyweight unit of hardware with many parts. Only so many complex cores can fit in a given spatial area, so making the units of parallelism more complicated pushes them further apart and increases communication delays. A vector instruction is nice for allowing the logic of a replicated operation to be relatively simple, taking up little space relative to a full CPU core.

Remaining opportunities: It’s very clear from this model that it only helps directly with very regular parallelism, where exactly the same operation is performed on a run of contiguous storage locations. Even when such opportunities exist in a program, relatively complex code may be needed to find them. Furthermore, many symbolic workloads (like formal verification) often don’t decompose in such a regular way.

Step 1: Many Small Threads of Execution

The original central abstraction of general-purpose GPUs is single instruction, multiple threads (SIMT). Like the non-SIMD CPU program we just discussed, a SIMT program can have many threads (units of parallelism) running a single program. What’s different in GPUs is bundling threads into groups that share positions within program code. Efficiencies very similar to vector instructions are realized in running such a group. For instance, every software-visible execution step only needs to fetch code from memory once and share the information across all threads in the group.

The CUDA term of art for a group of threads, launched together, is a warp. The hardware will generally maintain sets of running warps, picking to execute next one that is ready to proceed. This scheduling hides memory latency: while several warps are waiting for responses from relatively slow memory, another warp can proceed, perhaps generating a memory request itself and suspending until the response arrives. It is also valuable to notice similarities in the memory requests coming out of different threads in a warp, since often a single request to memory can cover what multiple threads have asked for, and memory requests are often the biggest performance bottlenecks.

When a warp contains instructions that make decisions about what code to run next, it can happen that execution diverges, where not all the constituent threads make the same decision. Here’s where we come to the reality of how the simple CUDA model is implemented. It is not simply the case that the hardware monitors thread activity dynamically and, on each execution step, runs just a set of threads that are converged with each other. In reality, the work is split between compilers and hardware, e.g. with compilers inserting extra instructions to note where threads converge again. Also, programs are overwhelmingly written to avoid divergence, since divergence has performance costs.

Progress beyond old-school CPUs: Programs can be written in terms of many more parallel threads, giving both hardware and compilers more opportunities to exploit parallelism. We’re talking about hundreds of thousands of threads exposed at once to latest-generation GPU hardware, as opposed to merely hundreds of threads for the latest CPUs.

Remaining opportunities: We’re still clearly heavily dependent on regular parallelism: SIMT only helps when many threads are doing roughly the same thing, and hardware scheduling of warps is heavyweight-enough that we wouldn’t want it to happen separately for each thread. Whether logic for tracking thread coordination is built into hardware or found in instructions inserted by a compiler, we still pay runtime costs to rediscover structure that may have been very clear in source code but lost during compilation. GPU programs remain sequential (one instruction after another), not spelling out dataflow graphs explicitly, which is ironic given that much GPU code is created from dataflow graphs with systems like PyTorch (though today’s compilers recover much of that information). Hardware could do a better job planning efficient execution with a code representation that avoids incidental ordering constraints between execution steps.

Step 2: Hardware Acceleration for Linear Algebra

The last two steps represent a smooth progression in one design dimension, but let’s turn next to two other important aspects of GPU design that are mostly orthogonal. A critical part of today’s GPUs is custom hardware for two-dimensional matrix operations, principally variations of matrix multiplication, which figure so centrally in deep learning. In fact, a study of BERT training found that tensor contractions accounted for a majority of GPU running time (though see the next step for a twist, as much of the time was actually spent on related data movement).

We see here an instance of the general pattern that performance can often be improved substantially by making hardware less flexible and more specialized. An intuition for the payoff here is that generality requires extra logic that takes up space in a hardware design. A specialized circuit has fewer parts, which can be closer together, which lowers cumulative delays for communication between parts. It’s also true that more of the computational work feels fundamental to a workload, instead of merely representing overhead from explaining to flexible hardware what it should do next.

These hardware units in GPUs are programmable mostly in choosing variations of matrix multiplication, including selecting different number representations and slight variations in how elements are laid out in memory. A matrix accelerator sometimes (especially in the latest hardware generations) includes its own local memory, with input matrices copied into it and output matrices copied out. One way or another, hardware is organized so that matrix accelerators interact with other state purposely situated close to them.

The CUDA term of art for such hardware is tensor core.

Progress beyond old-school CPUs: It’s hard to compete with hardware created for a relatively narrow purpose. For a program heavy on linear algebra, performance is going to be great. These hardware units are often implemented as systolic arrays, in a style related to the spatial one that I sketched last time, and it’s believable that they achieve the best performance possible for their specific hardware-fabrication technologies.

Remaining opportunities: Even programs that want to work with tensors of three dimensions and up face awkwardness in using matrix accelerators. Taking good advantage of them is even more challenging for symbolic processing. If only the systolic arrays were moderately more programmable, we could teach them new symbolic tricks.

Step 3: Explicit Memory Movement

I’ve emphasized that communication costs dominate performance of parallel systems as they scale, and hardware that tries to hide that fact through complex protocols makes it hard for programmers to tune performance. I would argue that this principle is the critical one that GPUs have really embraced, leaning into explicit data movement. Add in the last step’s use of hardware acceleration for common operations, and we have the explanation for GPU domination of deep learning.

While CPUs expose memory just via the fiction of a global memory that all threads can read and write, GPU programming also distinguishes between different kinds of memory, often associated with different levels in a hierarchy of nodes. Program instructions are needed to copy among the different memories, rather than just having hardware-managed caching do copying between CPU cache levels automatically. (It is important to note that GPUs do maintain cache-coherence protocols for data motion touching global memory.) Whether we are talking about “normal” SIMT threads or different kinds of more fixed-function accelerators (principally matrix accelerators), information often needs to wind up in their private memories, where it can naturally be accessed very quickly. (For conceptual simplicity here, I’m counting registers as a special case of memory.)

Progress beyond old-school CPUs: With cost of communication as the most important influence on performance, it’s a big win to be able to manage it explicitly with copy operations. When the programmer works hard enough, optimal use of physical resources can be coded up. Then the work of implementing the abstract machine can be divided intelligently between hardware and compilers.

Remaining opportunities: The latest GPUs still include very-involved hardware-managed caching, so with even-better compiler support, we could hope for better performance. But thinking in the other direction, what happens when compilers aren’t smart enough to plan the right data movement? We might still want runtime hardware support for understanding what is being computed and where, to make smart decisions about copies – with higher-level understanding of program structure than what hardware-managed caching systems embody, where they see every cache line (chunk of memory) as equal and undifferentiated. Also, CUDA and friends should be seen as rather-low-level languages, where programmer-friendliness is low. While indeed they are most-often used today via compilers from higher-level abstractions like PyTorch, those abstractions generally offer relatively weak features for program state that changes during execution. It is a shame if the need to work explicitly with state forces dropping down to low-level CUDA (or even meaningfully higher-level frameworks like Triton, which retains explicit programming of data movement but allows it to be described at a higher level). Finally, distances in physical space (which determine communication cost) are a little opaque in these kinds of hierarchical models, where number of hops in a tree is an OK but imperfect approximation.

One Alternative Architecture, Briefly

Let me also briefly mention the alternative parallel-computing system from Cerebras, because it came up in a reader comment for the last post. Roughly speaking, their programming and hardware stack looks like a simplified, flat GPU, where spatial geometry is exposed directly instead of laundering it through a hierarchy of thread groups and memory. Many threads are tiled in two-dimensional space, with explicit communication only with their immediate neighbors. Instead of any runtime tracking of thread convergence, compilers map work to threads that run independently. The individual threads are still programmed with relatively conventional sequential, memory-based code. Programmers and compilers can still do a really good job optimizing communication patterns on this platform, when they put their minds to it. The direct experience of writing such low-level code is still quite grungy, suggesting that higher-level languages would be good to add to the mix. Finally, as with GPUs, there is minimal runtime help for making smart data-movement decisions, in cases where not enough could be predicted in advance by compilers.

Conclusion

That’s a wrap for this sequence of four posts on programming abstractions and performance tuning. We see that GPUs represent a significant advance over CPUs in exposing the realities of physics, allowing more-but-simpler threads of execution and, arguably more importantly, explicit movement of data across memories. However, much of their efficiency is tied to regularity of data layouts and operations, which is not a natural fit for many domains like formal verification. There is good higher-level-language tooling for describing computations as dataflow graphs that don’t maintain state that changes during execution, but going beyond that model usually requires dropping down to rather low-level languages that are even more trouble to use correctly than conventional CPU-focused languages. We should aim for “have your cake and eat it, too” stacks that compile from pleasant high-level languages down through intermediate forms that make it easy to express spatial optimizations and end up in hardware that has enough information to make good runtime decisions efficiently.

Now we’ve covered some of the basic building blocks for how we could make tasks like code generation very efficient and reliable at the same time (if we can make formal reasoning efficient enough, too). I’m going to step back now and spend three posts on how we should reorient software development to take advantage of that capability. I’ll start with the question of how worried we should be about advanced models finding security vulnerabilities as quickly as we can fix them, transitioning into broader implications for the software-development process.

Many programs aren’t subjected to direct performance optimization. Either they are needed for contexts where performance isn’t critical, or they are fed into automated systems that do the optimization, whether those be conventional compilers or AI coding tools. However, as in the last two posts, my focus here is on languages that we use to express the process of optimizing a program’s performance. Whatever tricks we want to apply for improving speed or memory usage, they need to be expressible as transformations in the chosen language. It needs to be tractable enough to predict a program’s performance from its source code. When we realize a given program has left performance on the table, we hope that the methods to improve performance can be expressed in the language.

The claim I want to make is that languages for hosting such optimization need to be explicitly spatial, representing something closer to hardware circuits than to the software programs we are used to. The last two articles emphasized how such a spatial model is fundamentally what physics gives us to work with, and performance depends critically on communication costs between nodes arranged in space. While we have tried hard to work with software notations that hide the spatial nature and hardware that scurries around understanding software well enough to make good spatial decisions, splitting the optimization problem between these two layers makes it too hard to solve well.

Another slogan I like to use for this message is assembly language is over. That is, going forward, we need to swap out the class of lowest-level software programming languages. It has the two main weaknesses I emphasized about low-level programming languages in general: sequentialized code and global shared memory. It’s a fine abstraction when we don’t want to get serious about performance optimization, but it’s totally unsuitable when we do, leaving us searching for a replacement or several of them that deserve similarly central places in computing stacks.

If we accept this principle, then the inevitable result is that performance-tuned software and performance-tuned hardware will converge, in terms of the kinds of abstractions used to describe them. In my own experience working with computer-science students, it’s very common for those who have focused on software to declare that they “aren’t hardware people” and don’t want to work on hardware projects; I’m sure the opposite pattern is common in the world, too. It’s my view that these two disciplines will need largely to merge, and let me sketch, coming at it from the two directions, first how software can become more like hardware and then vice versa.

How Software Should Become More Like Hardware (Getting Spatial)

Physical reality is pieces of computation and storage scattered through space, communicating with each other at cost proportional to distance. To support productive optimization of software, we need languages that expose enough of the physical reality, at a minimum with explicit communication and a good way of estimating its cost. A commonality we will see is switching from having hardware make spatial decisions at runtime, toward having compilers make spatial decisions in advance. We can afford to have automation “think harder” about spatial decisions when they are made before execution and then reused for many runs (though also some challenges of understanding execution grow, when we can’t simply observe actual values flowing through a system).

There is a long history of trying to boost performance by moving work from hardware to compilers. Very long instruction word (VLIW) systems are an example first developed in the 1980s, where a low-level software program specifies not just a sequence of basic instructions but instead a sequence of groups of instructions that should run simultaneously. It becomes an important part of the compiler’s job to understand instructions well enough to put them in parallel groups. A particularly infamous initiative in this direction was the Itanium architecture family, which Intel was predicting would supersede their conventional CPU designs for server workloads. Instead, the effort failed in the market, and the failure is often blamed on the difficulty of writing effective compilers. However, note that these efforts have generally assumed that software programming languages look just as we’re used to, retaining the weaknesses that I complained about recently, and my personal bet is that such obstacles can be overcome by making more sweeping changes across the whole stack of software and hardware. So much is changing with adoption of AI in software engineering that it’s a great time to imagine other changes, too.

So let me sketch some potential approaches. Ironically for the part of this article labeled as being about software, we’ll mostly be looking at hardware architectures. The point will be what software programming model each one exposes, as the lowest level of software coding for its stack. We will focus on how such software formats allow programmers to work hard enough to make the optimization choices that best exploit what is physically possible, rather than being limited by somewhat-arbitrary program formats. In some sense, responsibility is passed to compilers, but I’ll have to save for later posts some ideas of how these compilers ought to work.

One common approach today, which already blurs the line between software and hardware, is field-programmable gate arrays (FPGAs). Basically, an FPGA has one circuit skeleton built into it – many computation nodes and many links between them. Programming an FPGA involves telling each node which logic gate to become (virtually speaking), using which of its links with other nodes. The nice thing about FPGAs is that they offer close to the optimal level of control over distribution of computation in (two-dimensional) space. One important downside is that they impose overheads over custom silicon, which isn’t surprising when we note that each node of an FPGA is programmed by filling in a literal truth table (though the FPGA term of art is “lookup table”), an exhaustive description of which output bits to generate for every set of possible input bits – which sounds kind of crazy to encode and execute directly. A widely cited study from 2006 establishes that using an FPGA instead of a custom chip usually slows down processing by a few times (referring to the wall-clock delay for the critical path), requires about 20 times as much physical area, and consumes about 10 times as much power. These particulars vary across designs, and the distribution has changed as FPGAs have evolved since 2006, but the quantitative orientation still seems about right. Such relatively modest costs may not be enough to deter usage in many important domains, but then the biggest issue is probably the complexity of programming FPGAs effectively, which we’ll get to in the second part of this article.

One way that FPGA overhead has improved, which is also highly relevant to hardware design in general, is through providing more special-purpose functional blocks. The general pattern is that specialization improves performance. The available programmable units need not have the full flexibility of Boolean truth tables, to provide a useful abstraction of reconfigurable circuits. Hardware dataflow architectures were developed starting in the 1970s, to work with software described explicitly in circuit-like form. However, these original systems didn’t have a spatial element to software programming. Compute nodes spread throughout the spatial area of a chip would dynamically choose circuit nodes to evaluate next, after noticing that their inputs had already been calculated. That dynamic scheduling mimicked the same two disadvantages that I described for the heroic efforts of CPUs (and indeed can be considered to have had direct influence on the modern development of out-of-order CPU execution).

Let’s consider one example of a hardware style that exposes more explicitly spatial programming. A line of 21st-century efforts has developed that kind of dataflow platform, hybridizing FPGAs and old-school dataflow machines. Like FPGAs, we have a grid of reconfigurable gates/nodes in a dataflow fabric. Like prior dataflow machines, the vocabulary of available operations is more similar to CPU instructions than truth tables. Each node is typically programmed with the equivalent of up to a handful of CPU instructions, allowing a silicon node implementation more like a mini-CPU than a sea of truth tables.

In this diagram, note how the CPU approach on the left (associated with the buzzword of a von Neumann architecture) involves physical circuitry that is program-independent, performing complex run-time analysis of a program, effectively building up and tearing down a sequence of spatial dataflow graphs. In contrast, in the coarse-grained reconfigurable array (CGRA) approach on the right, a program is mapped once into a configuration of each compute node in a grid. Then the computation can proceed without changing the role of each piece of silicon during execution.

Two main advantages (which we can imagine also applying to a variety of similar architectures) follow from the CGRA style.

1. We avoid the so-called von Neumann bottleneck, particularly the cost of continually fetching instructions from memory to reconfigure a silicon computation graph dynamically.

2. Probably more importantly, the lowest-level software format can talk directly about the two major factors that I derided low-level programming languages for not supporting. That is, computation can be laid out explicitly throughout space, with attention to the communication distances between nodes that talk to each other; and many memory operations can be performed on local memory nodes accessible only to their immediate neighbors, avoiding global memory with a hidden cache-coherence protocol and its complex performance model.

I found it fascinating to learn that this kind of architecture has been explored to optimize two very different classes of computing: high-performance servers and ultra-low-power embedded systems. High parallel performance motivated the design of the Plasticine academic prototype with an associated software compiler called Spatial, which led to the company SambaNova Systems. Later, targeting small embedded computers dependent on harnessing energy from their environments, for instance in space satellites, an academic prototype called Riptide was developed, which led to the Efficient Computer Company.

The design space is wide open, of the ideal languages to form the boundaries between hardware and software, and the class I’ve just sketched is just one example (though it’s one that I’ve become a fan of); we’ll cover a variety of ideas in later posts. Let me highlight already what I see as one major opportunity. Almost all full computing stacks of this kind maintain global shared memory, one of the two major performance misfeatures I highlighted in the last two posts. Much storage that would traditionally be maintained in shared memory is pushed off into elements of the programmable dataflow fabric. However, typically source code still uses shared memory explicitly, and those memory accesses are carried over into the generated code. As a result, we have the twin downsides of a complex model for software programmers and a hardware implementation with cache coherence that is very hard to scale to high parallelism. I argue that a major pending codesign challenge is creating programming languages and hardware models that allow full coordination across many parallel compute nodes without classic shared memory.

How Hardware Should Become More Like Software (Strengthening Modularity)

Whether we think of spatial programs as describing software or hardware, coding them up in the traditional hardware way leaves much to be desired. Hardware development is traditionally done in terms of register-transfer level (RTL) languages, which basically identify the key pieces of evolving state in a program (registers) and associate each one with a circuit that computes its new value based on prior values of registers. Such a program can be run repeatedly to evolve register values, one clock cycle at a time. This abstraction provides great control over what’s possible to implement with physical circuits, but it’s challenging to program correctly, accounting for a large fraction of the expense of developing new digital hardware.

One alternative that has become popular is high-level synthesis (HLS), which compiles programs in low-level programming languages into RTL. It suffers from the same challenges that plagued the Itanium project and its cousins, though it has been much more successful, probably because engineers who use it have been more willing to change their programming conventions to enable it. Other approaches to higher-level hardware programming include Chisel and a broader family of metaprogramming techniques it represents, where programs in various languages are written to generate RTL code. This style of tool offers weak modularity for metaprogrammed components, making it easy for a bug in the program that computes one module to cause trouble for a different module. I would argue that HLS is too hard to pull off because it doesn’t offer a spatial abstraction, while metaprogramming systems don’t help enough because they expose a spatial abstraction that is too low-level, lacking in both structure and guarantees.

An alternative abstraction that has greatly influenced my research is associated with the Bluespec hardware language developed at MIT starting around the year 2000.

The left pane of the diagram shows the standard RTL approach, where a complex spatial computation is one big circuit with all kinds of winding wire paths within it. The right pane, for the Bluespec style, exposes more of the modularity that software programmers are used to, for instance in object-oriented programming. The circuit is broken into modules, each with a well-defined purpose. Every module has private, encapsulated state, exposed to other modules only through explicit method calls.

In the more-modular style on the right, the modules can be handled separately in a variety of appealing dimensions.

1. Different teams can build different modules.

2. Each module can have its own formal specification, which need not mention anything about other modules.

3. Each module can be formally verified separately against its own local specification.

4. Yet there are streamlined principles for reasoning about what happens when separately verified modules are brought together into one system.

The other crucial ingredient is compiler technology that can turn such high-level descriptions into RTL automatically, reprising our theme of moving decisions about spatial arrangement to happen in advance. Through static analysis of a design, that style of compilation can understand possibilities to run different modules simultaneously and generate circuits to arbitrate the decisions, often coming very close to the performance of hand-written RTL, if not matching it (or exceeding it, in cases where engineers were confused by all the concurrency and missed optimization opportunities). Muddying the terminological waters is the hardware world’s convention of calling that process not compilation but synthesis, not to be confused with program synthesis (automatically writing software from specifications, a topic we’ll spend plenty of words on in later posts). This service of automatically interconnecting modules for performance is very different from what RTL tools can do with the lower-level notion of module that they have long supported, where module couplings are general wires rather than anything so high-level as method calls.

I’ve been involved in a series of projects developing Bluespec-inspired languages with strong connections to machine-checked proofs, including Kami, Kôika, and Fjfj. Some previous posts have used examples from formal-verification projects that have drawn on these frameworks, including for end-to-end integration verification across hardware and software and for proof that hardware systems don’t leak secrets through timing.

Again, the Bluespec style is just one example of raising the level of abstraction in hardware programming. I’ll explain it in more detail in later posts, and I also plan to cover some other abstractions that depart even further from industry practice.

Conclusion

There is always a tension in the design of abstractions for programming, whether by people or AI. I think of the programming process as a structured search through the space of possible programs, informed by a specification (formal or informal). On the one hand, making abstractions higher-level allows single steps in the search space to accomplish more-interesting changes, which accelerates search. It is also easier to apply formal verification to validate new variants against requirements. On the other hand, lower-level abstractions typically provide more control over performance. I’ve argued here that, first, those lower-level abstractions at the hardware-software boundary need to be more spatial to support continued performance scaling; and, second, that we needn’t accept industry’s standard hardware-description languages as our spatial-programming playground, with opportunities to raise the abstraction level while still maintaining good control of performance.

The next post will be the last in this series on abstractions for performance, covering where GPUs line up with these principles and where they diverge, especially when it comes to supporting efficient formal verification. Then we’ll change gears to discussing how higher-level engineering workflows should change with increased automation.

The last post covered low-level programming languages, the ones most-advertised as providing programmers maximum control over the performance of programs. I argued that the popular low-level languages are actually remarkably poorly suited to that task today, as hardware and software evolve together. Understanding the performance of a program in such a language requires thinking about aspects of hardware organization not at all represented explicitly in program syntax. I will use the current post first to go into more detail on the fundamental performance model that the laws of physics give us, and then I will primarily focus on the performance model that modern CPUs expose to software and the ways it makes performance tuning unnecessarily hard.

Computation in Space (Communication Dominates)

Let me start by repeating a figure from the last post.

Computation needs to be accomplished by configuring physical space. The parts of a computation coordinate through sending signals through space, and the speed of light is believed to be a limit on how quickly those signals can propagate. As a result, the distance between physical parts of a computation comes to limit performance. This diagram distinguished between computation and storage, though it is interesting to analyze both and notice that the same physical communication bottleneck limits them.

Take computation, which we can cast as circuits that break up the calculation work. I reviewed this model previously in discussion of inherent performance bottlenecks for deep learning. The key measurement limiting time to get the complete answer of a computation was critical-path length, or the length of the longest path through the diagram of a circuit. Interestingly, this length can be seen exactly as communication delay implied by the geometry and connectivity of a computation substrate. We’re just talking about communication across the regions of space that divide up the work of computation.

Perhaps less obvious is that performance of storage systems follows from essentially the same foundation. Software engineers are used to thinking about storage hierarchies, where computer systems are broken into different kinds of storage, including fast storage with relatively low capacity (storing fewer bits), slow storage with relatively high capacity, and different intermediate points. We can just accept storage hierarchies with this kind of trade-off as part of the facts of life today. Alternatively, we can see that storage is just another way of arranging matter in space, such that elements of the matter tend to preserve certain changes to their state. The more bits we want to represent, the more matter we’ll need to arrange in space, and thus the further-away some of those elements will be from each other. So we see that longer delays to return answers follow naturally from higher storage capacity. For this reason, it is often a good idea to avoid one gigantic storage system in favor of smaller ones spread throughout space, intermixed with the computation that will consult them.

In summary, we can go very far thinking about the fundamental limits of performance by considering only communication costs, in systems of nodes arranged spatially.

The Fiction of Flat Memory

My own career path kept me relatively confined to the software side of things until relatively late. I came (implicitly) to think of the interface that a CPU exposes to software as the fundamental definition of what is possible. Then, early in my time as faculty, a collaboration pulled me into learning about how memory systems actually work, as we did an especially comprehensive kind of formal verification of such implementations (and later work extracted out some of those ideas into a reusable implementation language). Here I’m talking about cache-coherence protocols, the canonical ways of implementing the fiction of memory as a flat sequence of numbered addresses, each accessible from every computation node. This abstraction is very far from fundamental, given the physical reality of computing, and it was eye-opening for me to see the complexity and computational cost that goes into providing it.

Let’s take a quick look at roughly how this kind of protocol works. We have the large and slow main memory at the top of a diagram, and we have individual CPU cores (our parallel compute nodes) at the bottom. In between are two additional levels of memory hierarchy. Each core has its own dedicated L1 (“level one”) cache, which stores memory addresses accessed recently by that core, keeping them accessible quickly. Groups of L1 caches share larger and slower L2 caches. The L2 caches finally connect to the main memory, through a directory that tracks the states of the different caches. Note that none of these ideas appear directly in low-level programming languages, providing the root of their disadvantage for modeling and tuning performance.

Protocol details won’t be too important for our main message. However, it is important to understand that different layers of the hierarchy can effectively lock (claim temporary exclusive ownership of) addresses, restricting the ability of other parts of the hierarchy to work with those addresses simultaneously. This diagram starts with the bottom-right CPU core holding a write lock on some address, meaning no other core is allowed to read or write that address, which saves the overhead of communicating with those other cores to coordinate accesses.

So now imagine what is an especially bad scenario for performance of this protocol. The bottom-right core took a write lock on some address, though now that core is done writing, but the system didn’t bother to release the lock. (After all, that core may return to writing the same address a millisecond later.) The bottom-left core, located furthest-away in space from the lock holder, wants to read the address. It has to route a message to the owning core through the whole hierarchy: up to the bottom-left L1, then to the L2 it connects to, to the directory, and then down a symmetrical path to the bottom-right L1. That L1 learns it has been asked to give up the lock and does so, and it also responds with the current value stored in the address. This response snakes back through the hierarchy, though now it moves in the reverse direction from the request.

We can see that quite a lot of communication has happened implicitly, which is trouble for giving programmers a performance model that is easy to reason about. Eyeballing this diagram as a literal spatial layout (which wouldn’t quite match any real CPU’s but is still illustrative), the total distance traversed by the path is roughly four times the actual physical distance between the two CPU cores involved – and the software code had no say in determining the path. There will be additional delay from the coordination logic that, as each signal is received, decides what further signals (and local state changes) to trigger based on it. As a storage system adds capacity, it often adds additional levels of hierarchy, which ironically can make worst-case communication delay worse, even as they may improve common cases.

The big point here is that important decisions highly relevant to communication overhead are entirely hidden from the programmer, who needs to apply a hardware model mentally to have a chance of predicting a program’s performance. Low-level languages generally don’t include direct abstractions of placement and neighbor-to-neighbor communication, but we need those abstractions if we are to do a good job performance-engineering.

Sequential Low-Level Code and Making CPUs Undo Translation Work

The last post emphasized two design weaknesses of low-level programming languages: sequentialized code and shared memory. We just saw how shared memory is very involved to implement in hardware with good performance. Let me now explain why sequentialized code is also problematic, in a way that applies just as well to programming languages C and Rust as it does to assembly and machine languages, the more direct bridges between software and CPUs.

Here’s a quick summary before we go into more detail. The laws of physics let us compute only with computation graphs arranged in space, with cost based on accumulated communication distances. Sequential programs are very far from this physical reality, yet we need to translate them into reality to run them. Good translation requires effective analysis of programs, which consumes computational resources if we perform it on-the-fly, as CPUs do. In fact, automated program understanding is so difficult that even the expensive translation in CPUs frequently makes (conservative) mistakes that hurt performance. Sometimes CPUs even make mistakes that create security problems.

OK, now back to more specifics. A CPU program is a sequence of instructions, which store intermediate state in registers, essentially parts of a core-local memory that is especially small but also especially fast; plus the shared, flat memory. Literally running such a program one-instruction-at-a-time can be punishingly slow, so the CPU makes an effort to extract latent parallelism and run many instructions at once, which corresponds to mapping parts of programs onto available computation graphs in space. We discussed before how long sequential dependency chains in computations create fundamental bottlenecks, so the CPU is motivated to notice which dependencies actually exist. Actually, it can overapproximate the dependencies and be sure to respect any dependency that it suspects may exist.

The trouble is that dependencies are indicated through the two types of state I mentioned, and they are both tricky to model.

1. Dependencies through registers are complicated by the fact that there are relatively few registers, which are fast but expensive to fabricate. Therefore, one register will often be reused for many different purposes throughout a computation, creating the potential for confusion. What appears to be a dependency may be an illusion, involving a register in two different lifecycle phases, serving different purposes. The extra work that a CPU does to extricate these false dependencies is register renaming.

2. Dependencies through memory are complicated by the fact that a given instruction may access an address computed by a complicated formula. Given two address formulas found in two parts of a program, it can be very challenging to decide if they might actually indicate the same address – and that analysis may even not be computable. As a result, CPUs are often very conservative about ruling out dependencies through memory.

Yet CPUs do a lot of work to best-effort approximate such dependency information. It is important to recognize that, alongside the “real work” of running a program, we constantly pay for the CPU to analyze a program as it runs to approximate its dependency structure. The CPU is repeatedly reconfiguring a physical graph of compute and storage nodes to stand for different parts of a program, minimizing the total communication distances that we focus on in this post. One trick is pipelining, where one instruction is broken up into several steps, which may reveal additional parallelism. Different steps of all of a large bag of instructions may be run simultaneously, so long as dependencies are respected.

The other big snag that arises in this delicate choreography is that sometimes the CPU is not even sure which instructions will be running in the near-term future, so it makes a best guess and prepares itself to undo work whenever a guess turns out to be wrong. This performance optimization can have nasty consequences, as the world learned through the Spectre and Meltdown attacks in 2018, which took advantage of the fact that CPUs don’t undo bad guesses completely enough to erase information (say, in L2 caches) that can later be used to leak program secrets. (See some past discussion here of effective reasoning about this risk from a software perspective.)

The irony of the situation is captured by this cartoon.

The CPU is reconstructing the information that often was already in the programmer’s head and maybe even written out directly in a program! Most clearly, if the original program was written out as a nice dataflow graph, then it is much easier for a CPU or anyone else to understand dependency structure. Yet we habitually use programming tools that translate nice high-level code into sequential, memory-access-heavy code that CPUs interpret natively, making bad decisions thanks to approximation and imposing run-time costs for the analysis that drives those decisions. Sometimes relevant analysis is much easier when performed with concrete values that appear at runtime (e.g. we may know when two formulas for computing memory addresses did or didn’t evaluate to equal values), but where ahead-of-time reasoning is possible, it can pay off significantly in alternative architectures.

Up Next

Some of these gripes can be addressed by smaller changes to existing low-level code formats. One I like is BasicBlocker, which uses compiler support to leave enough hints for CPUs that they don’t need to guess about where programs are headed. However, we should be able to reap bigger gains by making larger changes to standard abstractions.

So let’s think about what those changes should be. The next post will propose one principle for joint design of software and hardware abstractions: that each side should move closer to where the other has historically been, with more hardware-style software and more software-style hardware. We’ll discuss historical reasons why the community hasn’t moved as much in that direction as this post’s analysis may suggest it should. Not coincidentally, AI-champion GPUs already incorporate some of the principles we’ll cover, though we will discuss how they are less-suited to formal verification and related ingredients that will help us trust programs and automate programming more effectively.

Let’s consider in general the problem of automatic generation of code from its specification. Many AI coding assistants are being used today to write code in a “best effort” way, where the result may include mistakes such that the generated program does not follow its specification. I’m most interested (and future posts will most commonly discuss) methods that are correct by construction, where a tool might give up but will never return an incorrect program. An especially interesting approach is structured search through the space of programs, using formal verification as a fitness function to evaluate intermediate and final program variants.

Regardless of what approach is taken to automatic programming, we face conflicting desires to, on the one hand, simplify the space of programs and how we reason about it automatically; and, on the other hand, generate the highest-quality programs in the end. The rub is that higher-quality programs may be longer and more complicated than their lower-quality cousins, requiring more expensive search to find. An important determinant of the effectiveness of search is choice of programming language(s). My main claim in this post is that widely used programming languages are poorly suited for this style of search, especially when we are trying to write high-performance programs – which will be the case when we think about a program-search system recursively improving itself, given how much time and space a complex search can take. To argue this claim, I will start by spelling out a general framework for program search, and then I’ll draw attention to a missing ecological niche within that framework that is underserved by widely used programming languages.

Searching the Space of Programs

Let me describe at a very high level a process of gradually refining a specification into a final program. Arguably, human software engineers work in this way, and we can expect to rely increasingly on programs that automate the process. There is a tradition in computer science that makes the ideas formal, called derivation by stepwise refinement, but I’ll leave things largely informal here. We can think of the exploration as moving from the top of the following schematic depiction to the bottom.

The highest levels of the diagram deal with relatively abstract notation and therefore allow the most functionality to be covered by the shortest descriptions. The lowest levels are closer to the computer hardware we plan to run on, and they require details to be spelled out in excruciating detail.

Every node of this search is a program in some programming language. Every language we choose should pay off in tractability of search, control over details of the final program, or both. The most straightforward model is that we start off choosing languages biased toward simplifying search and eventually transition to languages that provide low-level control, though mixing the benefits throughout is also fine. Regardless, it should be a red flag if we are ever working with a language that scores poorly on both axes… yet I will claim that today, AI code generation is overwhelmingly working with such languages.

Low-Level Programming Languages

Let’s think in particular about languages suitable for the lower levels of the search process. A very dedicated software engineer might work directly with assembly language, almost the lowest-level form of programming undertaken today. However, it would be much more common to bottom out at a programming language like C, which was invented in the 1970s and has overwhelmingly influenced the design of low-level languages. More precisely, such languages are meant to allow fine-grained control over how a program uses available hardware resources and achieves the best performance.

There has been a lot of great innovation recently in this space of programming languages. Probably the biggest success story is the Rust language, which adds formal verification-style checking that programs follow safety rules. In contrast, C has been notorious for how easy it is for programmers to introduce mistakes that compromise safety and security. These safety rules don’t guarantee that a program does exactly what we want it to do, but formal-verification tools like Verus and Prusti add that kind of checking. As a formal-methods zealot myself, I won’t be satisified until some such verification method is considered to be part of the language definition, and programs that fail verification are considered ill-formed. However, for the rest of this post, I want to focus on a different axis of language design.

The problem is that languages in this category are meant to give fine-grained and effective control of performance. I would argue that higher-level languages are almost uniformly more pleasant to use to implement programs where performance doesn’t matter; we only choose a language like C or Rust to let us tune performance. Lisp machines provided a lovely stack for developing all code in a unified high-level language, and they “only” were missing performance capabilities, to keep up with other trends in hardware! The problem is that C was designed for the hardware of the 1970s and hasn’t been updated for the directions the computing world is heading in, with the end of Moore’s law forcing more cleverness in how we arrange computation. Thus, this language family is designed to fit snugly against the hardware model of the 1970s but fails to expose the affordances needed to get good scalable performance today. C, Rust, and friends are too low-level to be programmer-friendly high-level languages, and they are also too high-level to expose the true toolbox of performance optimization. (If we are stuck with particular fixed hardware, then C and Rust may give programmers just about all the performance knobs they have available, but we should think bigger and codesign new hardware with its programming tools.)

Distance and Communication

Computer hardware has always mediated between the physical world and nice abstractions for programming, but that kind of “magic” mediation layer has had trouble scaling. The reality is that computation takes place in three-dimensional space, but let’s simplify down to two-dimensional space for purposes of this blog post (even though three-dimensional chip technologies are gaining in popularity). Then we have a sea of nodes spread through space, and let’s say each node stands for computation or storage.

Some pairs of nodes are connected by wires, and typically signals only propagate along those wires. (Wireless communication creates enough overhead to make it infeasible at the scale we’re talking about here.) Because there is a manufacturing cost per wire, relatively few of the potential direct connections between nodes are built out. As a result, the fastest path between two nodes may take a rather-indirect route, as opposed to the most direct geometric path down the middle of the picture. However, that direct path establishes a kind of fundamental speed limit for communication between a node pair, often called the “speed-of-light limit” when we assume a signal propagates at the maximum possible speed.

I discussed a simplified version of this kind of system earlier in the context of fundamental performance limitations of deep learning. There the point was that a deep-learning system can be seen as a computation graph (system of nodes and edges between them), with edges representing input-output relationships. Computing the final answer requires computing the values of all nodes, which will take at least as long as the longest path that exists in the graph, an important measure called critical-path length. As systems scale, the communication overhead of propagating signals along wires often comes to dominate total execution time, and the major flaw in C, Rust, and their relatives is that the programming model hides these communication overheads and the aspects of program structure that make them tractable to predict and control.

Conventional low-level programming lives in a fantasy world that deviates from reality in two major ways. These big two were also presented in Chisnall’s “C Is Not a Low-level Language”, and the additional point that I’m emphasizing is that these fantasies make it harder to evaluate the performance of a program, which makes effective program search harder.

1. We pretend code looks like sequences of instructions that run one-at-a-time. In reality, all parts of a physical computation fabric are always “running.” Hardware like CPUs tries hard to promote utilization, the fraction of nodes doing useful work at some instant, but it can only go so far, reverse-engineering the structure of fundamentally sequential code even when the related behavioral questions are undecidable.

2. We pretend that there is one big shared memory that all parts of a program access. Yes, this abstraction is painstakingly constructed by hardware, but we can’t think of it with a naive cost model. The time to access a memory location at a particular computation node can vary dramatically based on a variety of factors, and the same node can experience very different access times for the same memory location at different moments. The principles behind predicting these performance factors come down to tracking locality of memory references, a concept that unfortunately often requires global thinking, meaning understanding the performance of one part of a system can require tracking behavior of other parts in detail. Alternatively, we can use established abstractions like MPI that reveal the hardware topology more directly, but these systems are also seen as painfully low-level and difficult to program.

An abstraction of sequential code execution is helpful for puny human brains to keep up in understanding a program, but it’s just too far from reality to be ideal for analyzing scalable performance. Low-level programming languages have usually started from sequential programming features and provided bolt-on features to explain how several such programs should run at once. The inability of language implementations to understand what is happening within the units of parallelism makes it harder to perform significant code rearrangements automatically as optimizations.

The most elite performance engineers can keep the organizations of specific hardware platforms in mind, to look at conventional software programs and “see” what they will cost to run. However, the details are punitively complex, which holds back automatic search through program variants to find the ones with the best performance. Importantly, this level of performance modeling seems to require thinking about programs at a very low level of abstraction that hides very little about hardware organization. At that level of abstraction, even simple-sounding improvements can require changes to many lines of code, perhaps spread across a large code base, and many search heuristics can never pull off that level of synchronized modification. (An example of such a nonlocal modification is dropping in a new data structure that requires a new coordination protocol for shared-memory accesses by all parts of the code touching the data structure.)

Conclusion

Whether we are thinking about human software engineers designing and implementing a program or about AI search through a space of programs realizing a specification, two properties are especially helpful in the notations we use to represent programs under consideration. Some notations allow us to describe complex systems succinctly, and those notations are convenient to understand and manipulate algebraically. Some notations give us the control to describe every last detail of harnessing hardware for maximum performance. The trouble with mainstream programming languages like C and Rust is that they display major shortcomings with respect to both goals. Even partisans of those languages typically agree that programming simplicity is sacrificed for control, making automated understanding of programs and their behavior (performance-wise or otherwise) formally impossible, though additional structural disciplines like Rust’s can help. But then the performance model of code is only implicit, given the massive gap between the realities of hardware and the fictions of parts of a program running sequential code and accessing a shared global memory.

There are enough parts of this argument and its evidence and consequences that I’ll continue over the next three posts. First, I’ll explain the heroic effort of modern CPUs to present the C-style programming abstraction while exhibiting very opaque performance behavior. Next, I’ll suggest a convergence between the programming styles of software and hardware as a unifying principle to chart a better course. Finally, I’ll cover GPUs as an example of an approach that has evolved closer to this ideal but retains some real weaknesses relevant to creating trustworthy decision systems, the overarching topic of this blog. Setting the stage with these details will prepare us for more exploration of our overall slogan that “intelligence depends on organizing computation correctly and efficiently.”

I will now make the case that the broader challenge of settling on system requirements also becomes fundamentally easier. The basic advantage comes from whose intentions need to be formalized in the requirements. Getting requirements out of squishy humans can be arbitrarily challenging, but what happens when the source of new requirements is overwhelmingly other programs already in production?

The Hard Part of Software Engineering

Even before AI coding assistants changed a software engineer’s mix of time spent on different activities, it was a truism in the field that the overwhelmingly time-consuming activity was understanding what users really want, or requirements-gathering. With recent automation of so much of routine coding, we might expect that the centrality of requirements-gathering will only increase. I want to suggest that, leaning even more into what will be possible with automation, we will ironically see requirements-gathering as we know it decrease in importance. To make that case, I should first review the conventional wisdom.

When programmable computers were first developed, the details of who did what to get them programmed were no doubt a little chaotic. Soon enough, though, around the 1960s, a split developed between analysts and programmers. The former had the job of understanding requirements and translating them into relatively unambiguous notation like flowcharts, which programmers could then translate into code. My own formal programming education involved just the slightest brush with flowcharting before that trend fully evaporated, but, in the course of a summer internship, I do remember being advised by a long-time technical employee of a large corporation that “programmer” was a poor choice of career path, representing a kind of cognitive underclass taking orders from analysts. (Perhaps developments in AI coding assistance make that advice more prescient than I realized at the time!)

The discipline of software engineering developed orthodoxies like the waterfall model, where teams are very careful to go through many iterations of written requirements before beginning the expensive process of programming. Writing the requirements is not just a process internal to a team of software specialists. Arguably most important is interfacing with the intended users of a program, to understand fully what functionality will satisfy them. One conversation is rarely enough. Instead, the intended users must be shown repeated revisions of a requirements document, to help them think more abstractly about the full range of relevant scenarios.

Eventually, waterfall went out of style in favor of agile methods, which focus on getting to working programs more quickly, so that users can give more-informed feedback. The problem is that even the software experts find it hard to enumerate all relevant scenarios for a complex program. Many scenarios only become clear as critical by relying on the expertise of the users, yet those users are not trained in the kind of abstract and systematic thinking needed to map out all relevant usage flows, with their steps and subtleties. Eliciting such information to guide a product roadmap remains so challenging today as to motivate the specialty of product management.

The challenge of learning what solution is really best for users is often explained using a quote attributed (perhaps apocryphally) to Henry Ford, about how if he had asked his customer base what they wanted in transportation, they would have asked for faster horses, not the cars that he eventually gave them. There are at least two separate problems. First, a user with a vague idea of the fundamental purpose of a piece of software may have trouble explaining that purpose in enough detail. Second, a user without an understanding of software-development pragmatics may not understand what is feasible to build at what cost.

The good news is that these challenges should diminish dramatically when the users asking for functionality frequently aren’t human. More precisely, the old kind of requirements-gathering challenges remain, perhaps even in a more challenging form, for a minority of the parts of an agent ecosystem, while this activity can largely be avoided for the remaining parts.

Software as a More Knowable User

There is a fundamental reason why it is easier to know what a program “wants” than what a person wants. Our evolutionary journey hasn’t placed much selection pressure on human minds to be easy to model by others. In fact, aspects like signaling have even promoted opacity of our thought processes, say to make it harder for others to figure out what will impress us, so that we can believe that the ones who do succeed at impressing us are unusually competent and worthy of affiliating with. In contrast, a program needs to be explainable to at least one audience: the computer that runs it! That is, the computer needs to have a clear-enough sense of the program meaning to be able to run the program, which we can phrase more technically as a need for programs to have unambiguous semantics. Add on top the need (at least until quite recently) for humans to understand the program as they are writing and maintaining it. There has been plenty of selection pressure for programs to be understandable, which can now pay off in their ability to request writing of new code to help them do their jobs better.

The ultimate setting for this advantage to shine is where significant pockets of the economy include only AI interacting with more AI, rather than with humans. Such a setting can maximize the benefit of environments designed for legibility to automated reasoning. An ecosystem of automated work can recursively self-improve toward a specification set in advance, or a set of competing and cooperating agents can evolve together toward their own specifications. Either way, we have an easily machine-readable characterization of fundamental goals, which can periodically be projected into requirements for new programs.

The next drawing illustrates the principle more visually. The left panel shows the world more as it is today, including many AI agents, almost all of which have to coordinate directly with humans, forcing the gathering of requirements from stakeholders who aren’t great at explaining what they want. The right panel shows the world we may be headed toward, where most of the economy is handled by AI agents, who naturally form into clusters with only rare gatekeepers who need to interface with humans. The interiors of the clusters are places of order and clear requirements. In this world, the great majority of programs are interior and benefit from clear sources of requirements.

It is critical that we will not be coming along later and trying to characterize the behavior and desires of agent ecosystems. There is the potential now to weave formal specification and verification into their design from early days. Our challenge becomes writing a specification covering the whole AI ecosystem, forcing the inhabitants to evolve in ways compatible with our goals and values. This requirements-gathering challenge remains at least as hard as usual! My point is that many other engineering projects within such AI bubbles become easier to frame, and they should become the majority of ongoing software-engineering effort (and source of associated cost).

A more modest version of this phenomenon is already common today. Take the example of a software-engineering team implementing a new web browser. Existing technical standards significantly constrain the new design. There are already voluminous requirements written down in Internet standards, explaining how the code for a web page corresponds to what should be displayed to the user. As more of the economy is handled by AI agents, more of their contexts will be standardized similarly or at least embodied in existing code that could be analyzed.

It is worth noting here that the most popular kind of AI software today, deep learning, has the distinct disadvantage that individual programs (now construed to include learned elements like model weights) have little apparent structure and thus are difficult to understand, say to figure out “what they want.” I would argue this quality should push us toward more use of other methods designed for understandability and ability to bound their behavior precisely with mathematical arguments, though my broader argument remains relevant otherwise. Put another way, we may have a short window to decide if we want to live in a world of inscrutable learned systems or instead push for systems proved to meet explicit requirements. It may be that the second path is critical not just for safety against rogue AIs but also for the efficiency of AI self-improvement that it enables, even if the ultimate goal happens to be better targeting of online ads. Let me now explain the source of that efficiency.

Copying Parts of Your Own Specification and Code is Relatively Easy

Let’s get a bit more specific about how the earliest stages of software engineering get cheaper, as we get programs asking for the writing of new programs.

Assume first that the user programs have been written following good practices and have their own polished requirements documents. We can even look to the most general version of an argument, assuming that the programs that will ask for new programs have their own requirements written out to the standards of quality desired for the new programs. The glib explanation of the advantage, then, is that we can just copy relevant bits of requirements content out of the requirements for the users requesting the new programs. No such option is straightforward for human users, who tend not to publish their life goals comprehensively in machine-readable form.

Consider more broadly how it should happen that programs want to see new programs written. I have suggested that we see an economy engaged in technical innovation as a distributed system in the computer-science sense, with different agents cooperating and competing within a set of overlapping optimization problems. Different participants know at different levels of detail what they are trying to accomplish; the participants that are software should be especially likely to be associated with rigorous formal specifications (or at least, that honor is unlikely for the people!). At some level, such an economic system is a program engaged in repeatedly optimizing itself.

Let’s get more specific with an example where a self-improving program naturally persists some of its own code into an improved variant. The relatively straightforward case is a program for computing a particular mathematical function and for noticing ways to improve its own structure to compute that function more efficiently. I’m arguing that software-engineering projects spinning up in the future will look more like this kind of self-optimization than like what we’re used to. A program with a goal but only a relatively fuzzy idea of how to achieve it will ask for new programs to be written that aim at the same goal but with more specific strategy. A good if fairly technical analogy is to multi-stage programming, which I’ll explain by example (though most of the code details aren’t critical to follow). Here is a (recursive) Python function that raises number x to the power of n.

def power(x, n):
    if n == 0:
        return 1
    elif n % 2 == 0:
        y = power(x, n // 2)
        return y * y
    else:
        return x * power(x, n - 1)

Imagine that an AI agent uses this function frequently for exponentiation tasks. In fact, it notices that half of the uses have n of 13. Through a mechanical process known as partial evaluation, it produces this specialized version.

def power13(x):
    x2 = x * x
    x3 = x * x2
    x6 = x3 * x3
    x12 = x6 * x6
    x13 = x * x12
    return x13

We had one stage of computation to come up with this code, and now we can run the new code repeatedly in a later stage, hence the name “multi-stage programming.” The new power13 function can run more quickly than power (it doesn’t need to run all of those tests about arithmetic properties of the exponent; it doesn’t need to do bookkeeping about how a group of function calls relate to each other). Part of what we give up to gain that speed is the generality of the function, but we can imagine we keep around the old version, too. We still use up extra storage for specialized versions, and we still spent time to come up with them. The same tradeoffs apply to just-in-time compilation as practiced by web browsers for efficient execution of the code embedded in web pages.

My claim, now more technical, is that these examples from mainstream computing today are more representative of the future of software development than are classic software-engineering projects. Yes, new projects will still be started to build software with human users, in which case the classic problems and methods remain relevant. However, for efficiency reasons, more and more of the economy will be handled by AI agents with no direct connections to humans, only other AI agents. Agents and groups of agents will trigger new software development for the same reasons it paid off to specialize the power function above, but these projects will be relatively better-defined, with their specifications naturally derived, often in mechanical ways, from the specifications or at least code of the requesting agents.

The next image tries to illustrate the pattern, with an agent requesting a new program based on a subset of its own code or specification, plus a novel environmental stimulus that the code should now be specialized to.

Previous posts have involved similarly geeky analogies based on interpreters, programs that run programs in particular languages; and compilers, programs that translate programs between languages. At some level, an agent could operate indefinitely as an interpreter. It has some high-level mission with a lot of execution details to figure out, through some kind of search process, perhaps coordinating with others. It can accumulate notes on wisdom gained so far, which are repeatedly consulted by code written to be prepared to act properly on whatever notes are passed to it. However, as the search proceeds, some parts of the code are revealed as irrelevant. Some are revealed to contain significant handling of uncommon or irrelevant cases. With a complex-enough initial program standing for everything that goes into doing a job somewhat like those done today by highly trained professionals, all sorts of software-engineering projects can be seen as specializations of that initial program. The possible scope is even wider as we imagine communities of agents coordinating to kick off software projects.

Conclusion

The argument I’ve spelled out here fills out a three-point recommendation or prediction for how we’ll take advantage of automated intelligence.

1. Formal verification is an invaluable tool for processes that search for better software, allowing up-front confirmation that a program will behave as intended in a variety of circumstances. However, we need sufficiently precise specifications to make formal verification meaningful.

2. If we do manage to kick things off with precise and comprehensive specifications for computer systems, then the searches they coordinate for better software can naturally pass on that specification quality toward the development of the new programs.

3. With that kind of recursive self-improvement loop operating well, it pays off to find ways to cloister units of economic activity away from sources of specification complexity, allowing them to have simpler specifications that enable the search process to operate effectively.

Such a loop depends critically on the efficiency of its implementation. The faster it runs, and the more it can do with a given amount of memory, the more effectively we can find better new strategies, which pays back recursively in the system’s ability to keep improving itself. That concern motivates thinking carefully about what software of the future should look like, as I’ll now cover in a series of posts about design of programming languages. I’ll make the case that the same principles that will aid autonomous systems also make sense for human programmers. I’ll also argue that languages like Rust, often held up as exemplars of the future of programming-language design, are not cut out for the performance scaling we’ll need for important applications.

My last few posts have been reviewing ways that we should be optimistic about the potential to write good formal specifications for important software of the future, enabling formal verification and other ways to increase confidence in those systems. By doing the kind of serious formal verification that includes reasoning about the interfaces between system components, we often get additional cybersecurity benefits for free. More speculatively, we are probably at the beginning of an accelerating curve for reconfiguring our economy so that more of it involves AIs interacting directly with each other, saving the related software from dealing with the complexities of modeling humans.

The upshot of that last trend is likely to be a dramatic complexity drop in the part of specification-writing that has dealt with user interfaces. I’ll present three ways in which it becomes easier to make rigorous arguments that these new software components make their “users” “happy.”

Signaling and Interface Trends

Interfaces to computers have changed significantly over time. On the one hand, a significant part of the change has to do with genuine conceptual breakthroughs in how to design interfaces to promote user satisfaction. Some interface ideas depend on raw computational power, where it may have been clear to earlier designers what they wished they could show to users, but bringing those ideas to life had to wait for compute to scale. Probably most observers would have those factors in mind, thinking back along the history of user interfaces.

This cartoon shows computer users wearing the kinds of clothing associated with their time periods, because I want to draw a connection between computer user interfaces and fashion in clothing. I think this analogy explains only a modest fraction of the cost of creating and maintaining user interfaces today, but it’s a real source of costs that I expect should now decline in importance.

Fashion is widely acknowledged as a great example of signaling and status competition. Sure, some changes in clothing arise from technological improvements that expand the space of what is possible to fabricate at reasonable cost. However, we accept that change in this domain is overwhelmingly driven by aesthetics.

At some level, fashion trends with the most subtle differences over prior ones are the most valuable for signaling. It may be the hardest job to design new fashion whose advantage is only clear to insiders. Then those insiders get to show off their savvy by picking out only those new fashions that succeed at the game. At some level, the participants in this status contest are showing off their working memories, by maintaining mental mappings from fashion trends to their timelines, so they can distinguish between styles that are less familiar because they’re so last decade vs. representing an avant-garde worth getting in on early. The general cognitive load of this process makes it useful to measure intelligence of potential mates and coalition partners, whether we are talking about “book smarts” intelligence or social intelligence to stay on top of group dynamics.

I would argue that computer user interfaces include a similar dynamic, not as centrally as for clothing fashion, but still important-enough to affect adoption trends. Now, I have a selfish reason to make this case, as a specialist in the more “backend” parts of computer systems. When I design user interfaces, they almost universally get panned as archaic. I give the objectors the benefit of the doubt that they are noting some overlooked genuine practical innovations in how to boost user productivity. However, it sure feels to me like there is an element of remembering interface vibes from different eras and wanting to be reminded of only the latest aesthetic, independently of the fundamentals behind each style. The explanation definitely can’t be that I still remember being excited in high school as HTML tables and frames first came out, right?

Whatever the role of signaling in user interfaces, we might hope that AI agents will have better things to do than worry about fashion. What formal analysis we may perform can focus on utilitarian requirements.

The Rise of Minimalist, Structured Interfaces

We don’t need to rely on sci-fi futurist mode to see important interface changes. A 2025 Salesforce article argued that GUIs would decrease in importance as AI agents proliferate. Another article from just days ago makes the case for the increasing importance of the kind of highly technical interfaces like REST APIs, which are convenient to use from programs – software using software. If the “workers” of the future are increasingly software, then naturally they’ll favor this kind of interface.

Let’s get a bit more specific. I’ll argue that human-facing UIs are greatly influenced by limitations in human working memory, what we can hold front-of-mind at once. Take the example of the controls for an airplane.

The airplane is a complex distributed system, with both digital and analog parts, their full, confusing detail visualized in the leftmost panel. In theory, a pilot can accept input from any of the sensors in the system or send requests to any of the actuators. The trouble is that humans, even the elite ones trained as pilots, can’t remember the full interface at that level. We are saving a user from a search process that crawls over a full system description and brings back the parts that are relevant, as the actual cockpit shown in the middle panel. With software design as we are used to it, the developer performs that expensive search up-front, stocking each “screen” of a program with exactly the input and output options deemed relevant to that stage of execution. This framing fits with the theory of distributed cognition, which holds that representations are split across participants in a system.

However, we don’t get the full story, just thinking about what is relevant to the user at each step. A user will often have some mission spread out across multiple steps, and working memory also makes it difficult to track mission status and what lies ahead. So the goal of a GUI in a single moment is more than just displaying the system’s affordances relevant to that moment: it also needs to summarize consequences of past decisions (how we got here) and options for next steps (where we may go). At some level, all of the bits of state throughout the whole system are sufficient to track both aspects, but our working memory can’t hold all of those bits, so we get the interface designer’s guess at what modestly large read-out will remind us where we are in a search process.

AI agents, represented in the rightmost panel, are just much better positioned at simply keeping all relevant state in memory accessible at reasonable cost. There are still differences in accessing different pieces of stored information, exemplified by memory hierarchies and context windows, but even the slowest storage method represents a huge win over what humans can muster. As a result, your canonical AI agent is ready to confront something like a fundamental description of a system’s parts and capabilities, which looks more like a data schema or an API specification than a GUI. My personal view is that designing better abstractions will remain important for increasing effectiveness of these systems, but so much can be accomplished with relative brute force.

Hence, in spelling out what makes a program acceptable, we need not worry about situation-specific details of interfaces and the capabilities of users to understand them. Instead, we can present relatively literal descriptions of what systems can do, an exercise typically much easier to get right. Humans remain important consumers of APIs today, requiring a certain amount of abstraction work by API designers, which may become increasingly unnecessary as AI usage predominates. (And, don’t worry: in later posts I’ll keep arguing for the importance of abstraction is making future systems faster and more reliable. It just seems the argument here is strong enough even continuing to lean heavily into brute force, as is the fashion today.)

The Value of an Open-Source User

There is another interesting shift in the dynamics of software to be used by software. It becomes relatively doable to have a machine-readable model of the user! The result is that the equivalent of usability testing becomes cheaper and can be automated.

The starting point is just creating a test suite for a service, which includes running a variety of common client programs against it. This kind of exercise is already standard today.

However, we can follow a strategy related to the one I wrote about previously, for AI agents coming to trust code provided by their competitors. Consider any user of the service whose source code is known to the service author. That user may be built by the same organization, or it may be released to the world as open source. Now the quality-assurance exercise is to carry out formal verification of the combined system, potentially including the service and all of its known users. Properties may even be checked of emergent behavior that arises from different users interacting with each other. So long as we can formalize, say, some productivity measure for the collective users, we can now (with enough proof effort) guarantee that measure across an infinite variety of scenarios, each lasting for infinite time. It’s a far cry from what is doable within the usability-testing budget for a software program today!

The change in perspective can really be quite a powerful booster to software development. It’s not just that we want to subject a new service to final checking before releasing it into the wild. We can evaluate usability repeatedly throughout development, to shape all the choices we make. Such checks can work like tool calls in AI coding assistants today. An AI searches the space of programs worth considering, repeatedly checking usability to bias its decisions.

Our AI-centric future may, of course, still include a role for services used by many different types of users, who now correspond to different programs, with different goals, written at different times. We can’t check the service’s usability in advance by reference to the specific source code of users who have not been constructed yet. Here, however, we come to what may be deeply satisfying to software engineers used to negotiating with finicky human users: we can impose a formal specification on the users, declaring that the service’s warranty is voided in the face of deviant behavior. The users can choose to invest in formally verifying themselves, in contrast to humans, who probably wouldn’t even bother to read these terms of service, let alone audit themselves for compliance. (This idea is related to design by contract in formal methods, which forces all parts of a software system to follow a common system of specification and sometimes proof.)

Conclusion

Turning over more of our economy to AI, it becomes important to be able to write down exactly what rules we want those agents to follow. If it is too difficult to formalize our requirements, then formal proof that agents meet them is a nonstarter. User interfaces have been a complex part of software as we know it, which might suggest a tough task ahead in formalizing that part for AI systems. However, if as I argue more and more of the activity of AI agents will be interacting with other agents, then there are a few reasons to expect that formalizing their interfaces will be easier than expected (at least for agents operating entirely in the interiors of zones of legibility).

1. User interfaces have varied to follow trends in fashion, which should not be a concern for AI agents.

2. User interfaces have been designed around limitations in human working memory, while interfaces for AIs can just present “the facts” in the most direct way.

3. When the users of a system are themselves programs, it becomes very cheap to test or even formally prove the system against its users, without any of the costs and delays of recruiting people for usability testing. Indeed, a single proof exercise can cover all users within some formally defined category (even an infinite one).

Drawing on these advances, it may become possible to move the equivalent of usability testing into strong formal proof.

The lingering objection I’m expecting at this point, from all you software engineers out there, is that the truly hard part of a project remains figuring out what users really want, which is mostly orthogonal to interface details. I agree: that element is what makes full automation of software engineering in today’s world a truly “AI-hard” problem! But in a world with even more AI centrality, does this aspect remain nearly as difficult? My next post argues why not.

The Cybersecurity Arms Race

Cybersecurity is often described as fundamentally favoring attackers over defenders. The reason is that an attacker often only needs to find one vulnerability in a system to subvert it, while a defender (or author of a system) needs to anticipate all possible attacks and build protections against all. Not noticing just one potential attack vector, or simply delaying too long in patching a known hole, can allow an attacker to just as effectively do damage as if the system were full of obvious security problems.

One of the worst kinds of security vulnerabilities is often called remote code execution: a way that an attacker with network access to a system can trick the system into running new program code provided by the attacker. Regardless of what we thought a system should do, an attacker modifying its code can change the plan rather arbitrarily. This category is a special case of the more-general phenomenon of subverting a system. I appreciate the evocative synonym perversion from A Fire Upon the Deep, applied to rogue artificial intelligences.

There are a bewildering variety of different remote code execution attacks, and engineers need to make sure to block all of them, even as new ones arise regularly. One of the classics is a buffer-overflow attack, where a segment of memory is reserved to hold user input, but a programming error causes the user input to run off the end of the region and into adjacent regions. If an adjacent region was used to hold machine code to be executed, we have given the attacker a way to inject his own code to execute.

The phenomenon of code-injection attacks is a broad one, and often different variants have common names that obscure their similarity. When user input is allowed to provide code in the SQL database language, we call it an SQL injection. When a similar situation happens with the JavaScript programming language, most typically in a web browser, we call it cross-site scripting. At first blush, it looks like we as engineers of would-be secure systems need to conduct careful audits of all programming languages used within, being sure none of them provide code-injection vulnerabilities.

It gets worse, though. Just enumerating programming languages isn’t good enough, because attackers find new ways to implement injection-like functionality within established languages. The first high-profile buffer-overflow attacks involved smashing the stack to overwrite certain places used to store addresses of code to execute. As increasingly effective mitigations against stack-smashing (like address-space layout randomization, which makes it hard to guess which code addresses contain which code, thus making it hard to guess the effect of an attack) rolled out, attackers developed increasingly sophisticated kinds of weird machines, a phrase that evokes finding hidden functionalities within seemingly innocuous programs. One idea called return-oriented programming creates seemingly impossible execution sequences within the legitimate code of a program by stitching together segments of code in surprising ways, producing emergent behaviors out of pieces that are innocuous individually.

And here we seem stuck as diligent defenders. It’s bad enough if we need to realize every place within a complex system that contains a programming language that an attacker could use to inject and run his own code. We also need to imagine all sorts of devious ways such functionality can be built on top of particular languages, including ways that haven’t been invented at the time code is deployed.

Future AI systems, especially superintelligent ones, amplify these worries. To start with, the baseline capabilities of these systems to influence this world may be so high that it becomes even more catastrophic to allow an attacker to subvert one of them and turn it against its owners. Then, as highlighted by recent use of the Mythos AI model to find new security vulnerabilities in widely used software, highly capable AI may mean an end to the method of security through obscurity that was already deprecated. Superintelligent systems may be able to find longstanding vulnerabilities that human hackers never caught onto.

So, where a novice engineer may worry that it’s too difficult to anticipate all attacks against an important system, but where one learning of formal methods may get excited about the potential to rule out those attacks mathematically, we also see how a more-informed engineer may get to worrying that it is impractical even to characterize all of the attacks in a formal specification, let alone prove that particular mitigations address them.

To the Rescue: Functional Correctness Implies Subversion-Resistance

Luckily, at the next-higher level of enlightenment, we see that using formal verification to block all of the above attacks is rather easy, if we manage to prove much of anything at all.

Think of a formal specification as laying out which destinations are legal within the execution space of a system, based on starting points.

The trouble with merely testing a system is that we might run many test cases, find that every one arrives at an acceptable destination, and yet have it be the case that the system exhibits catastrophic behavior in an infinite variety of scenarios that we did not think to test. Formal verification can demonstrate acceptable behavior in all possible executions.

Now assume that we have already invested in proof of functional correctness for a program, which means roughly that we show it truly carries out the intentions of its creator. For instance, a program that is meant to sort lists of integers in increasing order truly does output the sorted version of each input. Trying to define formally exactly which specifications count as functional correctness can be a losing game, but we do expect that such specifications are precise-enough that many small program changes break them. Let’s keep from this intuition an even laxer requirement: imagine any specification that accepts some system behaviors but not all. Let BAD be some behavior that is not allowed but can be expressed in the underlying programming language. We will now pull a trick reminiscent of our prior discussion of undecidable program properties and Rice’s theorem.

Imagine an attacker finds a way to inject code into the system and run that code, giving great freedom in how to perturb the system’s behavior. The attacker could use that freedom simply to upload the program for BAD, producing behavior that violates the specification, and so formal verification of the program must fail, despite not having enumerated any requirements about code injection. In this next image, I’m using the metaphor of an injection attack opening a gate to a world of great flexibility for the attacker, who is allowed to upload and run arbitrary programs in an expressive language, such that it is essentially assured that, once the gate opens, the attacker can find some way (probably infinitely many) to do serious damage.

For example, there may be an Internet service meant to add positive numbers on demand. Its specification may explain exactly which sums should be produced, in response to which requests. Or its specification might just declare that the service only outputs positive numbers. Both are sufficient to guarantee that no injection attack allows installing arbitrary code. If that vulnerability did exist, then we could make the system return zero, which violates both specifications.

In fact, the authors of specifications don’t need to think explicitly about code injections at all. It is only necessary to explain positively what a program is meant to do, rather than enumerate negatively what it must be prevented from doing. We saw an example in end-to-end verification of a simple IoT system, whose specification simply explained how a network protocol was meant to work, nonetheless ruling out, for instance, that the system can be taken over by a well-crafted packet and tricked into joining a botnet.

Conclusion

This observation certainly doesn’t address all of the challenges of AI alignment. We should expect superintelligence to be given such wide latitude in making plans and transforming the world as to defy concise specification. However, it seems reasonable to assume that any useful specification does rule out some behaviors, from which it still follows that attackers can’t gain access to run arbitrary code. Therefore, there is a relatively clear path toward using formal verification to develop highly empowered AI systems that will resist being subverted.

In fact, the principle generalizes to any cases involving top-level objectives that must be realized by lower-level implementation details that we need not spell out in formal specifications. For example:

• If an attacker finds a clever way to overwrite stored state of a system, then presumably either that state wasn’t important, or the attacker has a simple lever to perturb behavior away from what is specified.

• If an attacker is able to interfere with the flow of information from sensors into decision-making, say by taking advantage of a subtle weakness in digital signal processing, then either those sensors weren’t important, or arbitrarily distorting their values can easily lead to violating the specification.

• Conversely, if an attacker can mess with the logic controlling how a system takes action in the world, then it should be even easier to generate specification violations. Imagine an example of an online-shopping agent with a bug that repeats the last order over and over, instead of responding to new requests. Such behavior would violate specifications on the granularity spectrum from “only place orders that the user explicitly OKs” to “don’t place an order that would exceed your credit-card spending limit” (because the repeated order could happen to be a gigantic one, even as the later legitimate orders are for cheap items; remember, with formal verification, we reason in advance about all possible scenarios).

• Even examples of poisoning the training data of systems that learn are covered, so long as the learned components are implementation details not mentioned directly in specifications. Either the learned component is irrelevant to the top-level mission, or arbitrary data control allows breaking the specification without too much effort.

Of course, much of the current deep learning-maximalist approach is built around assuming that learned systems are central to meeting specifications, despite the absence of ways to prove in advance that learned components don’t surprise us, so my reassurance here in the last bullet only applies to examples where learned components are nicely modularized and only raise worries about implementation details like where the training data come from. We should remember that sometimes we realize we overlooked additional dimensions of top-level specification, as in leakage of secrets through timing that we discussed previously. However, there is great power in only needing to pin down the top-level requirements of a system and not worry about vulnerabilities missed in low-level implementation details.

The next posts will cover two other ways we should expect specification-writing to get simpler as more pockets of the economy become dominated by AI agents interacting with each other. We’ll start by reconsidering user interfaces in that world where users increasingly don’t belong to the same species anymore.

Engineers quickly realize a problem: true certainty about the behavior of systems is impossible. Our example from last time of a network-connected lightbulb controller presumes the correct operation of a physical actuator for the lightbulb. If that actuator instead starts a house fire when triggered, we’re in trouble. If a saboteur takes a hair dryer to the CPU and flips critical bits, all bets are off. The assumption that a hardware circuit executes in an orderly way can prove misleading, invalidating all of the theorems we invested in.

Yet it remains overwhelmingly valuable to carry out formal verification of complex digital systems. Why? We’ll review the effectiveness of abstractions that present clear interfaces on top of messy physical reality. Then we’ll generalize to principles for responsibly bringing artificial intelligence into more aspects of our society. It’s natural to think of adding AI into the world gradually, replacing units of functionality previously provided by humans. However, our evolved world is full of complexities that are expensive for AI to grapple with. The incremental path misses an opportunity to create economic enclaves purposely protected from those evolved complexities, allowing reasoning that is both cheaper and more effective.

The Digital Abstraction

I should confess here that, despite working in an academic department whose name starts with “electrical engineering,” I’ve never taken a class in electrical engineering. I couldn’t explain how flows of electrons provide the behaviors we’re used to in electronic circuits. Yet somehow I’ve been able to develop hardware and software components that function well. What’s the trick? All of us depend critically on the digital abstraction.

A naive electronic component can involve a wide range of voltages in a particular wire. Such an untamed wire can be used as analog storage, meaning that it represents a real number, but computing with real numbers, embodied in voltage levels, is a tricky business. We can imagine that a relatively “natural” electrical phenomenon would tend to spread its voltages over a wide range.

So then how have we come to rely on computers that make discrete decisions? The wizards of electrical engineering found a way to bias electronic components so that their voltage levels bunch up at two extremes. We can then draw a line down the middle of that spectrum and say every level above the line is a one and every level below a zero. Now we can construct the basic building blocks of digital computing, logic gates like “AND” that outputs one exactly when each of its two inputs is one.

Crucially, this digital abstraction lets us forget about continuous voltage and think of the “AND” gate as working solely with zeros and ones. Yes, such components do still glitch occasionally, revealing their analog reality. For instance, a cosmic ray may fly by and disturb the wiring. The point is that extensive engineering has pushed the risk of such disturbances low-enough that we can get far while ignoring them.

With this abstraction in place, we can build other abstractions. We can combine many primitive logic gates into more complex circuits. For example, we can build an addition circuit out of gates for “AND” and other simpler functionalities. Now this addition circuit can, in turn, be seen as a building block for higher-level functionality, ignoring not just how it was built out of simpler gates but also ignoring the analog dynamics of voltage within them.

Our end-to-end-verified stack example presumes the digital abstraction at its bottom level. More specifically, it bottoms out in formal semantics of a hardware-description language (a mathematical characterization of what results any circuit could produce), and we assume some accurate way of executing circuits encoded in that language. The digital abstraction gives us that way. An immensely complicated semiconductor-manufacturing supply chain is able to turn circuit descriptions into physical chips, well-enough that we can (usually) forget its details in designing digital systems.

Other Greatest Hits of Abstraction

The importance of abstraction in planning for a future of artificial intelligence has been emphasized in other sources, including The Singularity is Near. Let me just give a few more standard examples, before moving on to suggest a new kind of abstraction.

Natural evolutionary processes have given us many abstractions. Somehow lower-level physics allows subatomic particles to come together into atoms, which become largely reliable building blocks for chemistry. Then those higher-level molecules become building blocks for cells in biology. Cells can be aggregated into tissues and organisms, with cancer as the consequence of misbehavior when cells act more individualistically. We must plan for (prevent and treat) cancer, but, like a cosmic ray disturbing part of a silicon chip, it happens infrequently enough that abstractions like tissue and organism remain useful.

There are also higher-level abstractions that we design ourselves. One of the most basic ones is to have government maintain a monopoly on (legitimate) violence, so that competition shifts to economic activity, on top of strong foundations of property rights. On top of that foundation, a corporation can have legal personhood and enter into contracts with individuals or other corporations. Several corporations can then form a consortium that lobbies for their collective interest. Or individuals as citizens can aggregate into a nation, and then those nations can form coalitions. Geopolitical strategists can get pretty far thinking of coalitions or nations as atomic agents, even as we know it is often necessary to, say, understand a nation by understanding the will of blocs of its voters.

It is probably not controversial to suggest that taking full advantage of artificial intelligence will depend on developing new abstractions, but I’m going to suggest a strategy that tinkers with different parts of our world than most would focus on.

Bubbles of Legibility and Their Interfaces

One major point I worked up to in previous installments was the payoff from reconfiguring the world for greater legibility to intelligence technologies with good properties. That is, some hard problems of AI come from assuming that the basic structure of the world stays the same, and we plug an AI into a spot traditionally occupied by a human worker. If humans communicate with the rest of the world in natural language, we fall into assuming that AIs must communicate in natural language, too. Yet switching to other modes of communication dramatically simplifies processing.

One goal for the present post is to flesh out more of the strategy for reconfiguring the world. I also want to frame that process as introducing an important new kind of abstraction.

Here is the three-step recipe for building a region of legibility.

1. Identify part of the economy where all decision-making can be artificial, minimizing roles for intelligence that evolved rather than being designed deliberately.

2. Codesign that region with the agents that occupy it, optimizing for lowest cost of understanding by those agents.

3. Carefully create an interface with the rest of the world, a promise of the region of legibility about what service it provides.

One important kind of region of legibility is under single ownership. As a canonical example, consider an autonomous factory.

For simplicity, say the factory is a three-dimensional block of space enclosed in walls. Its mission, what led us to create it, is production of particular physical goods. The nature of those goods can be formalized mathematically, in theory allowing end-to-end proof that the factory delivers on its mission. It won’t be easy to get the specification right, as we must characterize physics sufficiently well. Moreover, capturing the nature of the desired product is insufficient, as we must also capture safety properties, e.g. avoidance of toxic emissions seeping through the factory walls. However, following the end-to-end verification approach, we avoid needing to understand how work is divided up and carried out within the factory. The following common AI challenges can be sidestepped entirely.

• As the factory is fully autonomous, natural language is off the table and need not be processed.

• There is also no need to worry about safe cooperation with humans on an assembly line.

• After the factory is constructed, there may no longer be any need to engage with vision or other conventional senses. The factory is laid out so that movement takes place among well-defined paths with easily detected markers of location.

• The software running the factory may be written in languages incomprehensible to humans, relying on formal methods to guarantee that specifications continue to be followed, even in the presence of recursive self-improvement.

Overall, the protected environment of the factory has been created to maximize legibility to the AI agents inhabiting it, minimizing their cost to make good decisions.

It remains a hard problem to capture the rules we want the factory to follow, just as it was difficult to build today’s silicon supply chain. However, the payoff in cost reduction should be enormous. Moreover, we have available the established tricks of end-to-end verification. It can be easier to believe a formal verification of multiple autonomous factories than one. If their outputs are integrated into single products whose specifications are simpler than those of the constituent components alone, then the trusted “exterior” interface of the mega-factory becomes simpler, and there are fewer opportunities for mistakes in formalizing it. The basic rules of factory safety may also remain largely the same across constituent factories. In such a setting, the mega-factory may even have autonomy to build new factories that follow the common rules. It may invent new decomposition of top-level deliverables into components, design and construct new factories to supply those components, and still remain compatible with its exterior interface.

The exterior interface is the place where humans bring requests and receive deliverables. We work hard to formalize it properly, in a way that retains flexibility for the intelligence within. There can in general be many interior interfaces, which no longer need to deal directly with complexities of human interaction. Rather, the consequences of human requirements are devolved down into other layers and their formal interfaces, and effort at simplifying the human-interface layer can pay off in simplifying other layers.

We will never capture all aspects of existing in the world as formal requirements, but the principle I’m proposing is to encapsulate as much of a system as possible away from those complexities, whether they come from human behavior or from imperfectly understood natural phenomena.

Competition and Ground Rules

What about larger systems than factories, where one of the chief complexities is interaction between parties that aren’t fully cooperative with each other? We can consider settings for AI agents both to compete and cooperate with each other, on top of a foundation that simplifies operation for all of them, just like the digital abstraction for computer systems or the rule of law for human civilization.

I wrote previous about how AI agents can trust code provided by others. The value of rigorous reasoning about code depends on having a reliable computing substrate, as the digital abstraction enables. However, it is also important to have rules governing which compute resources are controlled by which agents. Furthermore, as an agent reasons through the consequences of a piece of code, its job is greatly simplified by using streamlined artificial languages over natural ones, also preferring well-designed programming languages. Gaming out strategies for competition and cooperation is simplified by adoption of a streamlined sensory environment. The physical manifestation could be something like a special economic zone where only artificial intelligence is allowed.

There is a significant payoff from avoiding the need to reason about human decision-making, within the interior of the economic zone. Our brains developed through evolution, with its limited ability to escape local optima in fitness landscapes. We especially weren’t subjected to selection pressure for efficient understandability to algorithms. Some pressures have even been toward making understandability worse, as with signaling.

I should emphasize, though, that compatibility with human wishes remains central to the proper design of an autonomous economic zone. It’s just that those wishes are abstracted properly into the interface of the zone, kept separate from its interior. Requests flow in from humans to their agents, and then the agents act on those requests as efficiently as they can, ideally even in a provably compliant way, even as they may recursively improve themselves over time.

It is also important to acknowledge another kind of competition, which may lead agents or their coalitions to try to sabotage each other. These acts of aggression seem impossible to block entirely, but, still, I’ll argue that it makes sense to maintain orderly economic zones as far as possible. One analogy is with the increasingly global economy of today, with common norms of property rights and protection from violence. Various conflicts interfere with those rights and protections from time to time, and yet progress depends on relying on them most of the time. Otherwise, we wouldn’t see the investment in for-profit companies that is upstream of so many important breakthroughs. In the context of autonomous economic zones, an attack could force direct confrontation with, say, the laws of physics instead of nicer abstractions obeyed by reconfigured matter, in which case less-scrutable methods like deep learning become dominant again, but we can still do our best to avoid those situations.

Conclusion

The principle we’ve covered bears an interesting kind of mirror-image similarity to the idea of keeping an AI in a box. The motivation for that idea was to limit the ability of a rogue intelligence to do damage out in the world, and it has been argued extensively that even just a text connection with human users is enough for an AI to use trickery to “escape.” Instead, the abstraction barriers I’m arguing for protect AIs against the human world so that they can be more efficient and reliable. The idea is that the complexities we protect these AI modules against are the ones that force the use of opaque learned heuristics rather than traceable reasoning processes from first principles. Then mechanisms like formal verification can be used to anticipate all consequences of candidate design choices.

Even if our human concerns are pushed into the external interfaces of autonomous zones, we still need to characterize those concerns properly. The next three posts will go through three reasons that top-level specifications should be relatively tractable to write in this future scenario, starting from one observation around computer security that already even applies to today’s computer systems.

The challenge of getting specifications right is highly related to a high-profile problem in AI safety, alignment. Roughly, both problems are variants of the classic hazard of asking a genie to grant a wish. If some poor soul specifies a wish inexactly, the genie might just give him something terrible that matches a literal interpretation of the wish. AI alignment is concerned especially with recursively self-improving systems that modify their own code, perhaps subject to constraints from specifications given in advance – which is little comfort if specifications are inexact in capturing our true intentions, in ways that leave the door open to catastrophe. If a superintelligence is applying itself to find loopholes in our specifications, we could be in a lot of trouble.

However, the situation decades ago was already pretty risky in representative ways. First, human software engineers can make mistakes. More importantly, someone providing an appealing piece of software might actually be out to get us. It wouldn’t be very practical to write all of our own software from scratch, yet dividing up the work exposes us to sabotage by code authors. (And before someone objects that AI coding assistants now make it practical to write all of our own software from scratch, that usage mode just raises the risk of mistakes or sabotage by those agents!) I wrote previously about how agents sharing code with each other forces a solution to this problem, which luckily has already been studied in the old-fashioned setting.

The subject of the current post is a piece of good news about how alignment, or getting specifications right, is easier than it might sound. Across levels of familiarity with the subject, most people would probably agree that we expect alignment to get more challenging as systems grow in complexity. More moving parts means more parts that could contribute to bad outcomes, and it seems fatal to give any part the benefit of the doubt and skip careful specification for it. Luckily, the storied technique of end-to-end formal verification provides a counterintuitive way to harness expanded scope to lower risks of specification mistakes.

End-to-End Formal Verification

This post will take us through a series of steps of enlightenment in system construction. We can start by noticing that software seems awfully hard to get right, so it is worth investing in disciplined quality-assurance techniques. Formal verification, with its focus on rigorous proofs that implementations meet specifications, is an intuitively appealing technique. What could be better than mathematical certainty that a system behaves itself?

We already set the stage for the first objection: proof only helps us when it proves the right theorem, the specification for an artifact. If we got the specification wrong, then we may wind up worse off, thanks to a false sense of security from a misguided but complete proof. Nonetheless, assume for the sake of argument that we have created a formal specification that captures all of our true requirements for a system.

Let’s say that system is a complex piece of software. The gradually enlightened skeptic notices that software doesn’t just run on its own. There is always some programming-language implementation underneath, and what if that component has a bug? Then behavior can be changed arbitrarily, invalidating all of our original proof work. But why stop there? What about the computer hardware that the program runs on? This reductio can go on for as long as we like, perhaps winding up in quantum mechanics.

With end-to-end formal verification, we choose to stop the “what ifs” at some level and then prove all of the implicated components as a unit. Engineering costs are typically most reasonable when this verification is done modularly, which means that every natural component has its own specification and proof, which we later compose into one full-system specification and proof. Here is a diagram of a natural layered system decomposition, which follows a common pattern of successively translating high-level code into lower-level code as we proceed down the diagram. (We will return to discuss the intermediate specification boxes in the next section.)

While thinking about aligning superintelligence has by necessity been almost entirely theoretical and perhaps even philosophical, end-to-end formal verification has been concretely possible for decades, and we can learn about the broader challenges and potential by studying examples. One of my own related projects is the verified IoT lightbulb, with a unified proof of all digital parts of a simple system to control a lightbulb via an Internet connection. The bottom left of the following photo shows a field-programmable gate array (FPGA), a kind of reconfigurable hardware that stands in for fabrication of custom silicon, though the project was structured to be compatible with manufacturing silicon some day. This system includes all of the layers from the prior diagram, proved together as a unit.

A follow-on project confined itself to the software side but covered more-comprehensive functionality, including a nontrivial cryptographic protocol, for the verified garage-door opener. Following the latest fashion, this Lego garage has a microcontroller on its roof, which runs the cryptographic protocol to ensure that only the proper owner of the garage, the sole holder of a certain private key, is able to open and close the garage door over the Internet.

These sorts of projects are very satisfying in pursuing end-to-end formal verification far-enough to respond to many “what if” questions like I indicated above. However, the informed observer notices a problem that seems fatal at first.

Specifications for Intermediate Layers

Let us now turn to the layers of the first diagram labeled as specifications. Such specifications serve as interfaces between layers. Each one effectively lays out a contract that two layers promise to follow in cooperating. The examples in the diagram are:

1. The application specification mediates between user intentions and the behavior of the program.

2. The programming-language semantics (formal meanings of programs) mediates between programs and the language implementation (here a compiler).

3. The machine-language semantics mediates between the software compiler and the hardware.

4. The hardware-description-language semantics mediates between the hardware implementation and the language it is written in.

A typical mid-layer specification of this kind has two qualities that, taken together, may seem deeply troubling. First, a bug in such a specification can wreck the validity of the whole proof effort. Second, these intermediate specifications can easily be much longer and more complex than the specifications of particular top-level programs. Think of a book-length definition of a popular programming language compared against a one-page write-up about a limited-scope but important program.

There is further challenging subtlety here than just the chance to “get the spec wrong.” Often layers of a system will be formally verified using different approaches and tools, which do not share a specification language. A given specification may be written separately in the two languages, raising the specter of inconsistency amongst what are meant to be two renderings of the same specification.

How much harm could come from inconsistency? Let’s take a popular example of undefined behavior in programming languages. The idea here is that a language has rules that all programmers must follow, and breaking the rules “voids the warrantee” and allows the language implementation to behave however it likes. How could such a punitive contract be a good idea? The problem is that language implementors want to include automatic use of optimizations on behalf of programmers, but whether an optimization is safe to apply typically depends on aspects of program behavior that are undecidable, so the language implementation couldn’t possibly check for such a property without accidentally blocking programs that really do meet the requirements. The fix is to declare especially tricky program behaviors as undefined behavior, so the language implementation no longer needs to consider them.

For instance, if a program includes a sequence of 100 elements and the program tries to read the 101st element, that operation would be considered undefined behavior in C and related languages. The point is that reasonable programmers should agree that reading out-of-bounds in a sequence is a bug, and why should the language implementation be created to work around programmer bugs? Of course, the typical programmer rejoinder is that avoiding bugs is hard, and guard rails are appreciated! Nonetheless, languages like C are designed for such a performance-obsessed crowd that they tolerate the risk of undefined behavior.

How does undefined behavior tie back to our risk of specification inconsistency between two adjacent layers? The trouble would be if the compiler and the software above it were proved against different definitions of what constitutes undefined behavior. Then the software above might stick to defined behavior by its own standards, yet some of that behavior might be considered undefined by the compiler, allowing it to misbehave arbitrarily. For instance, the application might assume that reading the 101st element of a sequence of length 100 returns some safe default value, whereas the compiler considers the behavior undefined and allows itself to make arbitrary changes to any code downstream of such an out-of-bounds access – including removing explicit safety checks that appeared in the application code. Moreover, the more layers we add, the more chances we have to make similar mistakes with interface inconsistency. Considering that accumulating layers is a key technique to manage complexity in engineering, it would be quite a shame to pick up a disincentive against it.

Do these risks doom the whole project of coordinated formal verification of different parts of a system?

End-to-End Verification Pays Off

No! In fact, adding additional layers at higher or lower levels can even reduce opportunities for uncaught specification mistakes. The next level of enlightenment is realizing that integrating the proofs of all layers of a system helps us catch all consequential specification disagreements. There could still be bugs in intermediate specifications, but if we manage to prove the system overall, then those lingering bugs turned out not to matter for our top-level objectives. If there were a serious-enough specification disagreement, then either the proof of the provider of that abstraction would fail from below (e.g. a thoroughly unrealistic promise isn’t actually realized in the code), or the proof of the consumer would fail from above (e.g. an assumption about the layer below is too weak to allow proof of this layer’s specification). A diagram can help explain what happens.

The middle layers of the system have become untrusted in the sense of computer security, meaning that lingering bugs can’t spoil the primary guarantees we are after. Not only do we not trust implementation artifacts like the compiler and CPU, but we also avoid trusting the specifications of the programming language or hardware instruction set. In mathematical parlance, they only show up in lemmas used to prove the main theorem about the whole stack. If that theorem goes through, they don’t have fatal flaws in dimensions that it tracks (but might in other dimensions). A skeptical auditor need not even read those specifications to guard against violation of the top-level theorem.

The takeaway procedural message here is that, in doing end-to-end verification of a system stack within a single proof system, only the top and bottom specification layers remain trusted. In this way, we can often formally verify more-comprehensive systems with much less risk of misspecification than for less-comprehensive systems. By the way, this perspective is spelled out in more detail in a position paper from a broader project I was involved with. Another important aspect covered in that paper, and that I’ll return to in later posts, is the value of carrying out all proofs in a common formal system and tool ecosystem.

These lessons may generalize to problems of recursively self-improving superintelligences. Challenges in getting systems’ top-level specifications right will remain, but we can sidestep worrying about properly encapsulated internals. The next post turns to one important potential objection: computing stacks have been formally verified assuming perfect knowledge of the lower-level abstractions they run on, while real-world AI agents will need to deal with uncertainty and even adversarial platforms that they must run on.

It’s no controversial statement that, as artificial intelligence advances, the details of the computation underneath it matter a lot. My goal with this blog is to think through the design considerations from a new perspective. There are two main “thesis statement” framings that will drive the different topics we cover. They both sound anodyne on the surface, but I’ll be taking them further than most commentators do.

Principle #1: the bidirectional interplay between correctness and efficiency (e.g. running time and power usage) should be central to designing the AI systems of the future. I’m arguing for a central role for formal verification in supporting more ambitious optimization of systems, as the details of optimizations grow past the point that humans can analyze reliably. By using engineering approaches that guarantee correctness, engineers are emboldened to experiment more rapidly with new ideas that improve performance; and then the performance of the tools for formal reasoning becomes an important enabler of a virtuous cycle.

Principle #2: we miss critical opportunities for improvement when we look at systems as they exist today and assume that future systems will have roughly the same kinds of parts with the same local requirements as today. I’ll be arguing for significant changes to both computer hardware and programming languages, but I think it would be a mistake to stop there. We can expand the scope of the systems we are designing to what may be best summarized today as “society-level.” Fields like evolutionary psychology help us understand how humans function today as parts of such systems, leading to certain requirements on computer systems that are not fundamental and that can be beneficial to modify.

Let’s take a look at each principle in more detail.

The Interplay Between Correctness and Efficiency

This principle is going to be more of the usual kind that engineers propose to each other, without threatening professional identities as much as the next one does!

We can see all of scientific and engineering progress as one big distributed algorithm, traditionally staffed by people but with AI increasingly taking over the ideation process. The self-referential process of building high-performance AI systems is a good example (and one that I will return to in later posts), with e.g. algorithmic breakthroughs by human researchers, coupled to automatic translation (e.g. by compilers) of those breakthroughs into low-level code. The boundary between the human and machine contributions is already shifting and may shift dramatically more, making it helpful to consider more broadly the process of finding and validating ideas.

In fact, the evolutionary process that created our brains to do most of the work can be seen as just one part of the same continuing optimization problem. The great challenge of an evolutionary system is getting fast-enough feedback on quality of new variants, whether they are organisms or new software programs. The phenomenon of signaling from psychology and economics can be seen as a neat trick to get earlier high-fidelity feedback on individuals, though there are natural downsides to a convention of intentionally costly displays. An alternative way of moving high-fidelity feedback earlier is formal verification, which lets us evaluate variants in terms of mathematical proof about their behavior in all possible scenarios.

Sometimes lack of confidence in a system optimization or other improvement holds us back from adopting it. Cryptography can help us overcome certain trust problems as ideas and code spread. However, I’m more focused on formal verification as a crucial enabler of adopting ideas, even when we have them ourselves but especially when they come from others we have reason to distrust.

In the first category, consider the example of a recursively self-improving, highly parallel compiler. It constantly invents new tricks to make itself faster and thus able to produce better programs within a given time budget, but how do we know it doesn’t modify itself in a way that compiles all programs into “kill all humans” agents? It is possible to prove that such a process maintains the original purpose of the compiler, through any number of improvement rounds.

In the second category, consider the example of mutually distrusting AI agents sharing code libraries with each other. The paradigm of proof-carrying code makes it possible to distribute code with proofs of fitness for purpose, security, and so on. AI agents can adopt new “brain modules” much more quickly and confidently than humans can, thanks to the chance to check them against the most exacting standards instantly, which is likely to be an important enabler of accelerated progress in science and engineering.

Sometimes poor performance of formal verification limits how much we can achieve of the benefits I just sketched. For example, most of the student research projects I supervise at MIT wind up bottlenecked on the performance of automated-theorem-proving tools. There’s been a successful full-stack effort, from hardware through software programming tools, to enable machine-learning computation based on linear algebra. Formal verification deserves the same effort, and every level of the stack probably winds up looking interestingly different than GPUs and CUDA. I’ve written very little so far about the specifics I have in mind, but expect plenty of posts coming up.

The Importance of Holistic Design

Engineers will all agree that sometimes focusing in on one part of an existing system doesn’t leave enough degrees of freedom to meet some target for improvement. My proposal to reimplement the hardware-software stack to accelerate formal verification fits into that tradition, even as it remains speculative and therefore fair to be skeptical about. And, again, I’ll have plenty to write about details at this level, geeking out about hardware-software codesign. However, more controversially, I want to argue that we can realize even more improvement by broadening the scope of system we consider.

Here are some examples covered by full posts so far, moving up a ladder of abstraction from the physical world to broader cognition.

• Controlling self-driving vehicles seems to involve hard problems of computer vision and more. However, we can circumvent those problems by changing the physical environment so that autonomous vehicles have their own dedicated lanes with intentionally simpler geometry.

• Natural-language processing seems fundamental to many compelling applications of AI. However, a future of AI agents can rely on synthetic languages that are much easier to process.

• AI coding assistants solve hard problems in applying today’s popular programming languages to solve new programming challenges. However, we can make the problem much easier by changing the programming languages. (I am going to have a lot more to write in future posts on “the right way” to automate both software programming and digital hardware design, delivering strong correctness guarantees.)

As we go up the ladder, evolutionary psychology starts being more helpful to understand the origins of the AI challenges we’re used to. Natural language is an especially pernicious example that evolution has shaped to be hard on purpose due to its use in signaling; we don’t need to choose engineering abstractions that are hard on purpose! A theme we’ll return to in many later posts is how we can harness an outside view of our own cognition and where it comes from to spot opportunities to simplify intelligence through bigger changes to how the world works.

I’ve also spent several posts on how a holistic-design approach reveals weaknesses of deep learning, the most-popular AI technique today. Its triumphs are based on its ability to look up and combine prior art, to solve new problems in consulting vast training sets of prior solutions. However, it has major weaknesses in both of the system dimensions I emphasized above. The correctness problems are well-known, given the inscrutability of learned models and thus the risks that they will make decisions that go against our wishes. The efficiency problem is right there in the word “deep,” indicating a required structure of computation that guarantees long delays to answer prompts, compounding multiplicatively as layers of additional functionality are added (despite the intensive effort to optimize the whole stack from algorithms to hardware, which has paid off much more for throughput than latency). I’ll be proposing a new approach or two that solve both problems, using the out-of-style symbolic mode of AI (sometimes cooperating with deep learning), in one of its most-visible modern garbs as formal verification.

But wasn’t that style discredited, with deep learning knocking its socks off in so many important domains? I argue that the hardest AI problems are disproportionately like the ones I listed above, where we have paths toward avoiding them through more-holistic system design. There the principle is that we should reconfigure the world for greater legibility to intelligence technologies with good properties. Some of the pushback will come from human nature, as people across the tech industry and elsewhere are reluctant to give up on hard problems and expensive solutions that are helpful to signal competence and wealth. However, it’s worth overcoming those perverse incentives to push for lower-costs systems that deliver more-reliable answers.

Next Steps

My next arc of posts is going to cover the central ingredient of specifications for formal verification. It doesn’t help much to show that a system follows a specification that fails to cover our real objectives. I’ll review some examples from formal verification today, interleaved with scenarios that may feel more like sci-fi today but may be here before we know it.

Put differently: The first level of enlightenment is realizing the potential of formal verification to support rapid development of complex systems so that they are correct by construction. The second level of enlightenment (which luminaries of formal verification have realized and written about for decades) is noticing that systems must be proved against specifications, and the engineering of the specifications is not obviously more tractable than for the artifacts they apply to. The third level of enlightenment is realizing the suite of techniques that make dangerous specification mistakes easier to avoid than we might think, as I will explain in the coming posts.

Afterward, we can come back to considering how extracting more parallelism in computation may be one of the most important activities for the future of intelligent life in the universe, feeding a virtuous cycle of improving the formally assured intelligence that can improve itself.

This perspective is so far from the mainstream that it isn’t even on a menu of possible positions that folks are used to being prepared to argue with. One natural hypothesis is that success in the real world fundamentally requires solving familiar hard AI problems, so we really need to stick with techniques like deep learning that have the best track record solving those problems. That hypothesis is compatible with an engineering mindset, where in building better artifacts, we may need to trade off between dimensions like generality, speed, and reliability – and deep learning’s success in generality justifies weaknesses in other dimensions.

My subject in this post is how such claims may often be after-the-fact rationalizations. There are other reasons, grounded in evolutionary psychology, why we should expect the popular approach, epitomized by LLMs and the systems built on them, to be seductive. By surfacing those reasons, we can better inform ourselves to choose the right technology for each job. The key will be that familiar aspects of large-scale AI deployment are helpful to individuals looking to build status, in ways that create perverse incentives at odds with traditional engineering goals.

Signaling: Expensive Displays

I previously covered the phenomenon of signaling, where animals perform costly displays of fitness, so that observers are convinced that they are worthy as mates, coalition partners, and so on. I presented signaling as a cool trick to accelerate evolution, where observers don’t have to wait for individuals to encounter rare situations where their skills can shine, since instead regular opportunities are created to show off the same skills in artificial circumstances. Having reliable information sooner helps evolutionary processes improve more quickly, by directing more resources to the most-promising individuals. Humans are particularly flexible in being able to learn new signaling games, which might not even have existed at their births, let alone in the bulk of our evolution before we converged into anatomically modern humans only a few hundred thousand years ago. I suggested “programmable evolution” as a useful way to describe that flexibility.

A textbook example of signaling through conspicuous consumption is the potlach, a kind of feast among the indigenous people of North America. At such a party, a tribal leader gives away or flat-out destroys many items considered valuable. The point is precisely that only a very rich leader or tribe could afford to part with so much wealth. In turn, the wealth is considered to have arisen through effectiveness in leadership and planning, producing a highly legible signal of leadership quality. We couldn’t trust a leader to give a speech explaining his triumphs. We all know how easy self-aggrandizement can be, picking and choosing topics. Instead, we need a signal that is expensive to fake, and what better than wanton destruction of objects that are known to require great coordination and talent to create?

This kind of social ritual is far from having disappeared among us. Think of a Great Gatsby-style mansion party, still built around conspicuous consumption and over-the-top hosting competency. A relatively recent study that I enjoyed was Very Important People: Status and Beauty in the Global Party Circuit, which looks at rituals in nightclubs today.

The point is that we find easy analogies between old-school anthropology and wealth-signaling displays today. Signaling behavior around a variety of admirable qualities is pervasive in our world, for instance accounting for art as a way to signal cognitive ability. Now we can turn to the specifics of these phenomena around AI-powered systems.

Difficult Problems and Expensive Solutions

Let’s start by thinking about how elite software engineers build and maintain their status in the global tech industry. Imagine that an engineer has applied for a job, and the cornerstone of his portfolio is a totally new technology stack, from custom hardware on up, that he built to solve an important problem. On one level, we’re suitably impressed by his versatility. However, we also find it very hard to tell which parts of his work were hard – and of course we want to hire smart people who are good at solving hard problems. We know the popular technology stacks of today and which of their aspects require which skills to execute competently. Our job evaluating the candidate would be much easier if he picked one of the established areas to focus on.

Candidates appreciate this dynamic, perhaps not always fully consciously. With AI as the hottest area today, it is not only valuable for engineers to build up portfolios of AI work, but also it benefits them to choose among relatively small menus of “standard” AI problems that are widely understood by people who make hiring decisions. So now we have the most innocuous form of signaling dynamics leading to lock-in for choice of AI problems.

However, the effects are more perverse than that narrative suggests. Signaling games are more effective when they involve harder problems, so the tech industry tends toward elite engineers choosing the hardest problems to work on. This incentive exists even if the hardest problems are not aligned with the best engineering solutions to real-world problems in any dimensions, like cost, speed, or decision quality. Difficulty is valuable in its own right. Just to avoid the impression of taking potshots at one technical community from my own ivory tower in academia, I’ll mention that I have observed a similar dynamic in the world of programming languages, where many engineers in my circles prefer programming languages like Haskell and particularly abstract ways of using them that turn programming into challenging puzzles, applying more-complex code than is needed to solve a problem, for love of the mathematically grounded obstacles that must be overcome. A programmer with this mindset typically isn’t consciously thinking “I want to make my job harder,” but I argue that evolutionary dynamics explain why it would feel natural and fulfilling to seek out hard problems that most people can’t solve.

I also want to emphasize that current frontier AI work is doing amazing things, with highly successful efforts to solve hard problems in new ways delivering phenomenal value to society. Still, we should remain wary of the effect of signaling on incentives. We should also keep an eye out for the possibility to redesign systems at higher levels, so that lower levels no longer contain AI problems that are as challenging – as sad as it may feel to lose the opportunity to solve those problems heroically.

We’ve now gone two steps up the ladder of ways that signaling drives engineering choices, progressing through increasingly less noble-sounding root causes. The signaling we’ve covered so far takes place within relatively narrow specialist communities: engineers evaluating engineers. It can be an uphill battle to convince modern knowledge workers to forsake the valorization of solving hard problems, even if engineers will generally agree in the abstract that it is better to find ways to decompose a goal into subproblems that cost the least to solve. OK, but let me next consider another variety of signaling that stands in conflict with many of our stated values, where, while specialists may generate the signals, the signals can be evaluated by a general audience.

The last examples focused on costs from the labor of rare expert software engineers. Harder AI problems require paying more to rarer experts. However, another proxy for problem hardness is instructive: the cost of hardware needed to implement competitive solutions. Today’s token commodity in that category is the GPU, the cousin of CPUs that is the overwhelmingly popular choice for implementing deep learning. There is a striking convergence of prices for GPUs and luxury watches, with popular models costing tens of thousands of dollars each (see sources for GPUs and watches). It’s probably the case that GPU prices are driven by genuine supply-chain challenges ramping up to serve the extraordinary demand, while high-end watch prices reflect intentional scarcity and extra features added precisely because they increase price. Still, we wind up with GPUs in a secondary role to signal conspicuous consumption, like works of art ceremonially destroyed at potlachs. That last statement can coexist with the great fit between GPUs and implementation of deep learning, which just helps hide the signaling motive, without reducing its potency.

Then we have the data center, overwhelmingly important today for housing GPUs. Naturally, the cost of a data center is significantly higher than the cost of just one computer inside it, creating an even more potent signal of wealth. Data centers also have a significant advantage for signaling to society at large. Only geeks know enough about GPUs to compare them and decide which are most impressive, beyond just looking at the price tag (which isn’t typically hanging off of the deployed GPU on a piece of string!). There can be a similar dynamic with luxury watches, where only a connoisseur can tell the difference between models of radically different price levels. However, a data center is a large facility consuming a large amount of electricity. It is very easy for members of the global elite across industries to understand that such an object must be expensive and indicate success by whatever organization owns it.

Across tech-company sizes, we can see a common dynamic of signaling with AI-hardware deployments. The biggest companies compete to announce larger and larger build-outs of data centers, as well as breakthrough solutions to ever-harder AI problems (connecting to the prior example of incentives to solve hard problems). Scrappy start-ups can go through hard times and then, after new success in sales or attracting investment, signal their success very clearly by buying GPUs. The cofounders feel a sense of relief that, in contrast to earlier periods when no one could tell if they were “for real,” now everyone can tell “they’ve really made it” when their GPU stockpiles are large enough. Release of open-source software or open-weight models can be a good signaling trick, too, if it’s clear that the artifacts could only have been produced by burning enough compute, for instance in training the model using your GPU farm.

By the way, even individual engineers can feel the call of more-expensive solutions in their daily work. There was quite a stir recently around the revelation of tech companies maintaining internal leaderboards of, basically, which software engineers were spending the most on AI tools. There is certainly a correlation between cost of AI services used to build some technical artifact and the inherent challenge (or business value) of building that artifact, but we can wind up in strange places if measuring and incentivizing just the former.

Conclusion

Engineering involves navigating trade-offs between dimensions like cost, speed, and accuracy. However, the signaling phenomenon I’ve outlined pushes tech-industry participants, often without realizing it consciously, to optimize perversely for focusing on hard problems (the spirit of engineering prefers to simplify problems) and expensive solutions (while maximizing cost is rarely part of a classical engineering trade-off). Participants from CEOs to junior engineers can build status, both within their specialist communities and in society at large, by affiliating with expensive solutions to hard problems. The pull is strongest for problems and solutions that are obviously hard or expensive to a broad educated audience, and we should work to counteract that pull.

Evolution has left us with many bad habits – or ways of thinking that are clearly misaligned with modern life. For instance, we learn as kids that it is not a shrewd move to adopt the all-candy diet, even if our ancestors 100,000 years ago encountered sugar so rarely that they always won by consuming as much as they could find. Today’s engineered systems are vastly more complex than anything those ancestors dealt with, weakening the relevance of their instincts to signal success by displaying expensive solutions to hard problems. We should follow the true spirit of engineering and always appreciate a chance to simplify a problem by changing a system design at a higher level. Sometimes simplifying the world is itself an engineering challenge, and we should watch out for perverse incentives in choosing to do it, but often the investment of changing the world pays off in simpler engineering subproblems ever after.

Deep learning and friends are especially likely to excel at hard problems produced by evolution that we can work around today, as I will explore in considering how the compute stack should be different to take advantage of problems with elegant logical structure. If everything goes well, we won’t need to depend anymore on computer-hardware-world equivalents of luxury watches! It won’t hurt that we get more-reliable systems as a bonus, in a world with more of our economy handed off to AI agents in a carefully curated ecosystem.

The straightforward answer is that, so far, they’ve demonstrated dramatically worse answer quality on most problems of high interest today. If we take a quick survey of popular applications of AI, we’ll find deep learning and friends way out in front on each one. But does that summary really imply that the future is centered on statistical machine learning? I’m going to present an alternative framework that justifies an answer of “no,” though first I want to summarize a more mainstream framework.

Predictive Coding

Predictive coding is a theory of intelligence that is popular with people thinking about powerful AI. Roughly, the theory of predictive coding encourages us to go beyond simplistic models that assume our senses perceive the objective truth of the world directly. Rather, instead our brains need to develop fairly detailed internal models of the world, and we can integrate sensory inputs only with respect to models, finding what model-compatible world state best matches the inputs. When we notice bad results from our current models, we update them by considering the details of what goes wrong, much like how training deep-learning models works with backpropagation to reverse-engineer inaccurate decisions into the right changes to model parameters.

This diagram shows the basic idea in a learning loop. The observer fine-tunes his world model to help him better perceive a vase. Cases where the model makes bad predictions at odds with new inputs trigger modifications to the model.

Why we do find ourselves in situations where modeling phenomena of interest is very challenging? I’ll focus on one broad kind of mechanism: when those phenomena are produced by evolutionary processes that don’t optimize for legibility. That is, some evolutionary process provides feedback on intermediate designs, but it doesn’t penalize designs that are hard to understand.

Even if we were optimizing explicitly for legibility, we could find evolution getting stuck in local optima. The reason is that evolution proceeds through small changes, each of which needs to improve fitness in at least some small way, otherwise a variant would be discarded. There may exist radically more legible redesigns that nonetheless can’t be reached through sequences of small steps that all improve legibility and/or whatever other objectives. (Maybe we find a path of gradually increasing legibility that reduces real-world practicality at intermediate points, even though practicality spikes upward by the end.)

It’s not just biological evolution that can exhibit this problem. We can also see it in deliberately engineered systems, through viewing ourselves as a distributed system for finding better technical ideas. I wrote previously about signaling as an optimization for such systems, where participants go out of their ways to show off their otherwise-hidden fitness qualities through costly displays, to provide earlier signal that helps the optimization algorithm prune unpromising paths. The presence of signaling can actually lead toward optimizing (parts of) systems for worse legibility, to provide opportunities for particularly exaggerated displays of competence.

The upshot is that we may very well have wound up with a variety of canonical AI problems that seem, with respect to our current knowledge, to be irreducibly complex: where only the kind of large, unstructured model produced by deep learning can deliver good-enough decisions. We could even take a cue from Kolmogorov complexity and define complexity of a problem in terms of the length of the shortest model that understands it well enough. The latest foundation models are described in terabits of weights – where “tera” means 10 to the 12th. On the one hand, we can celebrate the engineering achievement of training models that have so much relatively unstructured complexity to them and recognize that complex descriptions are probably necessary to understand a variety of important phenomena. On the other hand, more-complex models tend to be more expensive to find and execute. What both predictive coding and the practice of machine learning share is learning via loops that tend to increase complexity: as a model is better-and-better fitted to an underlying phenomenon, the model gets more complex, or it converges to better results because there was complexity inherent in its architecture from the start. What if we extended such loops so that they could also include steps that by design reduce complexity?

Codesign for Legibility

It’s largely taken for granted today that the way to make progress in the face of challenging reasoning problems is to improve AI systems. However, another technique can be even more powerful: changing problems to be easier to solve. The revised problems can be better fits for AI approaches with superior properties in speed and reliability. Classical rule-based systems don’t just take advantage of well-defined logical structure when it exists; they often also fail completely in domains where such structure is not known. Can we redesign systems at a higher level to help structure become clear?

I wrote previously about codesign with the example of autonomous vehicles. The broader idea is that we can make AI problems simpler by changing the contexts they operate in. Using that idea, we can take the predictive-coding world of a learning loop and transform it into a codesign loop that alternates between steps of learning and changing the world being learned, with an eye toward improved legibility, or ease of learning.

Returning to our idea of measuring complexity in terms of bit counts needed to describe the world, we are now combining steps that refine models, which may indeed incorporate additional bits; and steps that simplify the world, which actually reduce complexity of effective models when applied properly. This framing suggests a very different perspective than celebrating engineering that enables learning many bits. Instead, we see progress as coming from changing the world to require as few bits to describe as we can get away with. Such compression of knowledge comes from structure that maps well to the world, which can in turn support crisp guarantees. This next stylized graph shows how alternating these kinds of changes helps us wiggle description complexity downward. Even after investing in more-complex models, we can find paths back to simplicity, guided by what would help the models, through world simplification.

From this perspective, we should change the world to remove challenging AI problems as far as possible, guided by experience building earlier generations of systems. Here are a few examples of opportunities of this kind, most of which I covered in earlier posts, where I highlight examples of evolutionary dynamics leading to environments that are harder than necessary for intelligent agents. Each example identifies a hard problem associated with AI and finds a way to replace it with a better-structured alternative that streamlines automation. I’m going to cover these suggestions in increasing order of controversy or effort to reconfigure the world.

1. One good example has already been adopted widely in software engineering. The first web applications were designed just to be used directly by humans with browser GUIs. However, soon-enough some users wanted programs to access web applications on their behalfs. Initial efforts involved unpleasant engineering approaches like scraping, which required software to understand both natural language and visual layout. Eventually the owners of many web applications started offering web APIs, ways of accessing the same services in ways friendlier to automated understanding, requiring solving exactly no problems considered “AI.”

2. Software programming is a domain where we are in control, with humans having deliberately designed the programming languages and other tools. Programming languages have always been designed to make programs easier for humans to understand, though inertia leaves in-place some bad design ideas. There is even some signaling going on, where language designers sometimes include complexities because they introduce puzzles that programmers enjoy solving. Instead of sticking with languages that happened to have many examples included in the first big LLM training runs, we should change the way programming works so that AI has an easier time spotting bugs and so on.

3. We should change the environments that autonomous vehicles inhabit to simplify vision and other problems relevant to effective control. Instead of roads full of unpredictable phenomena, in some settings, a model more like subway tunnels is appropriate. Today’s road network evolved with human drivers in mind, not considering the possibilities for cost savings through standardization and simplification.

4. Now gradually ramping up in speculative nature of proposals, moving away from use of natural language will simplify many relevant problems. As more of the economy is dominated by AI agents, they will have options beside natural language for coordinating with each other. Natural languages are creaky machines that were shaped by evolutionary processes that didn’t select strongly for lack of ambiguity or simplicity of processing. In fact, signaling pushes toward more-complex language that can be used to show off cognitive ability. Protocols for communication by AI agents needn’t maintain any of that baggage. (Actually, the prior example of adoption of web APIs is an early case of this principle reduced to practice very effectively!)

5. I can’t resist dropping in one more idea that is quite speculative, which you can take or leave, independently of buying into this broader framework of codesign. Biology is full of mysteries, which are mysterious to our puny brains because, with the possible minute exception of the most-recent past, there hasn’t been evolutionary pressure towards organisms being able to understand their own workings. We can keep working on reverse-engineering our evolved mechanisms, but the long run may see even better results from replacing parts of ourselves. For instance, artificial replacement organs may follow engineering best practices and be easier to understand and optimize than natural organs.

The general approach, of an improvement loop that combines learning and modification of the phenomenon to learn, is an underappreciated secret weapon on the path to effective intelligent systems. We’re considering the word “system” in a broad sense that keeps the above codesign examples in-scope.

Consequences for Feasible Automated Understanding

It may be that today’s canonical AI problems require learning mathematical functions of seemingly irreducible complexity, making logic-based methods fundamentally inapplicable. However, by changing the world so that it exposes different decision problems, we can make logic-based reasoning competitive. By giving the world legible structure, we enable methods that thrive on structure, potentially enjoying benefits for both performance and explainability and mathematical guarantees broadly. Two of the major categories for upcoming posts are (1) more specifics of how the world can and should be different to support effective automated reasoning and (2) the right architecture of those reasoning systems that take advantage of structure, looking across the whole stack of hardware and software. Pulling on these threads will take us in a pretty-different direction from the industry consensus.

From Training Data to Parameters

I suggested previously that deep learning is better-understood in analogy to search engines than to reasoning engines. It typically succeeds in proportion to its ability to find examples it already knows about that are related to a request. However, it is not just memorizing examples. We covered how it generalizes solving for m and b in the equation y = mx + b from algebra class. The difference is that instead of just those two parameters, foundation models can get up to hundreds of billions of parameters today!

The double-edged sword is that, while so much information or even insight can be encoded in so many parameters, it is an uphill battle to extract structure from learned parameters, an obstacle to deriving mathematical guarantees. This general pursuit for AI is called explainability. A good intuition is that, when the training methods that learn parameters aren’t designed around deliberate structure, more work is required to find structure (and thus convincing explanations) after-the-fact.

One category of explainability for deep learning analyzes which parts of a prompt influence the final answer significantly. This kind of analysis already doesn’t explain how the relevant parts of the question informed the answer, but it’s a good start. However, such analysis suffers from classic challenges of software testing: when we only evaluate a program on a particular set of inputs, it is hard to build confidence about possible behavior on all inputs. We may have neglected to test a corner case that matters in some important scenario (or even many and frequent scenarios that escaped our imagination). That risk is serious-enough in a setting where all users have good intentions. In a cybersecurity setting where an adversary is doing his best to drive the software to bad behavior, we must assume that the adversary finds exactly the input that will trigger the worst behavior.

In a deep neural net, we can imagine corner cases in the form of weights (parameters) that have little influence in the questions being asked during evaluation but that turn out to matter for other important questions. No matter how good of a job we do explaining what parts of a question influenced the answer, we can’t be sure that future answers won’t have rather-different, problematic explanations. This example is inspired by adversarial examples and backdoor attacks in machine learning, where realistic failures would involve many weights being moderately off, though I simplify in the diagram to a single problematic weight.

Continuing the march through analogues of classic software-quality techniques, we find interesting analogies with compiler verification as I discussed it previously. With that subject matter of compilers, programs that translate between computer languages, the goal is to confirm that translation preserves behavior/meaning. I explained that one approach is certifying compilation, where every run of the compiler outputs not just a translated program but also a certificate of some kind that translation was carried out correctly. One very-flexible kind of certificate is a mathematical proof. That style of certifying explanation is being explored by many teams now not just for its “obvious” application in getting AI to do math but also in getting AI to write correct code, which could be accompanied by proof of correctness. It remains a niche approach today (e.g., there’s no similar mechanism applied by LLMs to explain arbitrary responses), though a variety of more-targeted projects involve different notions of certificates and their checkers.

I can give one U.S.-centric analogy for the nature of certificates. U.S. taxpayers need to submit annual tax returns where they don’t just declare how much they owe in taxes but also lay out, sometimes in excruciating detail across many forms, the calculations that justify their answers. A certificate is like that kind of “showing your work,” referencing mechanized rules instead of the United States tax code. A given tax return impinges on just a tiny slice of the total tax code, allowing relatively cheap auditing of any given tax return, though a search for a taxpayer’s ideal tax-return strategy could explore many parts of the law that turn out not to be relevant (i.e., some could be used but not in ways that reduce tax owed).

The following diagram shows how a single defect in the model leads both to a wrong answer and a flaw in the certificate. A checker that flags the latter helps us avoid proceeding with the former, though in a way that leaves us without a clear alternative.

I should also briefly mention LLM-associated techniques called chain-of-thought, where models are guided through spelling out intermediate steps in finding their answers; and self-consistency, producing multiple chains of thought and choosing as “winner” the answer appearing most frequently. To the extent these techniques write natural language and remain subject to the vagaries of LLM randomness, they still don’t help produce strong guarantees. Variants using formal languages converge toward generating formal proofs.

Certifying algorithms (and we could charitably view chain-of-thought as a special case) have common strengths and weaknesses. The key strength is that it is often much easier to demonstrate that a single answer is correct than to demonstrate that a system only ever outputs correct answers. This benefit can translate into lower engineering costs to build a decision engine. A counteracting weakness is the inherent potential for unpleasant surprises: a given run of the certifying algorithm may eventually output an invalid certificate, which fails independent checking. For instance, a certificate may be a math proof carried out in a series of steps, and one of the steps turns out not to follow from the previous deductions, after all. Then what is the user to do? The result is practically indistinguishable from the decision program running forever or crashing, neither of which tends to go over well with users. The system winds up in an indeterminate state, unable to commit to an answer. Considering AI broadly, it is not always the case that we have a safe default action to take, when a decision engine fails to make a recommendation. For instance, we may be driving an aerial vehicle in tricky weather, dependent on constant clever decision-making to avoid a literal crash. In those settings, reaching an indeterminate state is itself a failure.

For many reasonable notions of certificates, it remains unclear if deep learning will evolve to produce them at all, for problems that are sufficiently complex and novel. Sometimes finding the certificate is itself the hard part, e.g. looking for proofs of longstanding mathematical conjectures. Even if deep learning can be scaled to produce a certain kind of certificate, it may take unreasonably long to do so. For instance, one easy upgrade for such an engine is to add a loop that checks certificates coming out, restarting the system every time checking fails (perhaps with new prompting about what went wrong last time, to help avoid a repeat). In the worst case, such a system will run forever on hard-enough inputs. Even if it does find a certifiable answer eventually, computational cost can easily balloon with this approach, and these retry costs should be considered a special case of what I highlighted in a previous post about how top-level “agentic loops” can lead to very long delays in producing final answers. Our earlier example of control software for an aerial vehicle is a good one, where we may be able to afford neither long delays nor fallback to simple defaults; and it is not hard to find other domains where faster specialized answers at least provide clear economic value.

The counterpart from compiler verification is certified compilers, where a mathematical proof shows that a compiler behaves properly for all possible inputs. Constructing such a proof can be much harder, but it saves us from worrying about unpleasant surprises on new inputs. Compiler verification has a decent analogy to mechanistic interpretability for AI, which reverse-engineers machine-learning models into human-meaningful explanations, but the foundations of this field are still being established. In the mean time, we might also wonder about how other approaches to decision-making are more suited to proven guardrails.

Our goal will be to avoid needing to catch reasoning errors after-the-fact at all. Instead, we want to take advantage of system structure to make reasoning errors impossible.

Good Old-Fashioned AI

Deep learning and its closest relatives are contrasted with good old-fashioned AI (GOFAI), a cheeky name for symbolic AI. Whatever we call it, this older approach is centered on formal logic and other symbolic ways of representing reasoning, more in the style we’re used to from working out math derivations on blackboards. A particularly influential style is expert systems, which apply domain-specific logical rules to solve new problems. It turns out that style can avoid the challenges we just surveyed around getting stuck in indeterminate states or facing expensive checking of answers. You’ve probably heard about how thoroughly out-of-style rule-based systems are today, and we’ll come back to that infamous history, but let me review first what kind of technology we are talking about. Later posts will get into relevant and interesting developments since expert systems fell out-of-favor, including advancements in programming tools and hardware, plus the chance to take advantage of statistical machine learning where its weaknesses are less relevant.

This kind of expert system is based on rules that deduce new facts from those already known. For instance, here is a set of rules that could be applied by venture capitalists to decide on valuations for AI startups.

RULE: Add $1M to the valuation for every occurrence of the word "agentic" in the pitch deck.
RULE: If person P has a LinkedIn profile that lists employment at company C, then consider that P worked for C.
RULE: If a cofounder has worked at a major AI company, add $5M to the valuation.
RULE: If the product has probability R of destroying humanity, add $10M/(1 - R) to the valuation.

To run such a logic program, we can start with a set of known facts (like the contents of cofounders’ LinkedIn profiles) and keep deducing new facts via our rules until no more follow. At the end, we sum up all the incremental amounts that have been deduced.

It is readily apparent how a simpler explainability approach applies to this kind of reasoning system. Every rule can be scrutinized by experts, in advance of deploying a system. In practice, the rules would be written in some programming language rather than English, for extra avoidance of potential ambiguity. Notations of formal logic are typically used. However, the narrative above of how one “obviously” executes a logic program works pretty well to explain how the real thing operates.

One way to trust outputs of a rule-based system follows certifying compilers: have every execution output a trace for how the rules were used to come to a conclusion. The example I’ve used here is a little unnecessarily complicated from this perspective, but let’s simplify by saying (1) the goal of the system is to certify a minimum valuation for a startup, and (2) assume we have no rules that add negative amounts to the tally. Then a certificate is a list of rule instances: each element is one rule from the knowledge base, along with a further trace for each one, explaining how we deduce its premises. For instance, we can justify a use of the rule about cofounder-from-major-AI-company by including a trace demonstrating that the cofounder worked for a particular company. That nested trace may itself invoke another rule and need to include further traces, but eventually the process bottoms out in referencing only facts that we assumed are true. The point is that, while in general it may be complex to make a conclusion and produce its trace, it is simple and cheap to check a trace.

The story is quite similar to what we’ve considered for my own specialty of formal verification and its applications to recursive self-improvement and code-sharing by agents. Indeed, formal verification, including automated theorem proving, is an example of an area that has been so successful as to be ejected from the popular conception of “AI,” following the old maxim that the “AI” category only includes reasoning tasks that seem sufficiently unsolved today.

Thinking back to our discussion of performance bottlenecks inherent to deep learning, we can see an appealing advantage for rule-based systems beyond explainability. A key principle we relied on was that the latency, or time it takes to get the complete answer to a question, is proportional to the critical path length of a system, measuring the longest sequence of steps that necessarily must occur in sequence. Some conclusions of a rule-based system can be justified with shallow traces, meaning that we don’t need to use rules whose premises are justified with rules whose premises appeal to rules… to too extreme of a depth. In principle, with a good parallel implementation, the critical path is determined by the depth of the trace. In other words, answers with shallow proofs should be findable quickly, with convincing evidence for their truth. The same expert system could still be ready to run for longer on more complex questions, taking time proportional to that complexity.

A Discredited Approach?

Now I can return to confronting how thoroughly expert systems have gone out of fashion. A major AI winter starting in the 1980s was centered on disillusionment with expert systems. Then we had a period when roughly all AI approaches were widely viewed with skepticism, followed by the explosion of deep learning in the 2010s, putting expert systems at even more of a relative disadvantage in the popular imagination. Maybe I should feel sheepish, then, to reveal the truth about this blog: a major theme is going to be the increasingly compelling argument for returning to this style of automated decision-making (sometimes in cooperation with other techniques like deep learning). There are two major apparent obstacles in the way of achieving good results.

First, there has been massive full-stack investment in deep learning and related techniques, leaving competitors with quite some catching-up to do. Consider:

• Foundational work on algorithms in the domain

• AI hardware accelerators like GPUs, as well as the programming tools that complement them, which together provide amazing parallel performance

• Wide distribution of expertise in this kind of computing

• Even wider familiarity with how to get good results prompting generative-AI systems

Second, statistical machine learning has shown amazing results in so many domains where approaches based on symbolic logic stalled. If we randomly sample populations like business users for their most-important problems where they want help from artificial intelligence, we mostly get answers where generative AI is way ahead of other methods today. The burden of proof is on someone boosting another approach, to show how it can be at all competitive.

I’ll take a first shot at that argument in my next post. The key is to zoom out and apply a codesign approach, looking beyond the normal scopes in which AI problems are defined. Later posts will return to performance engineering of whole computing stacks for systems based on symbolic logic.

Our take-away message is going to be that systems based on deep learning introduce sequentiality (certain computation steps that must happen after others) that is often not inherent in the questions that we ask them to answer. Instead, it arises from how we choose to organize computation to answer them. The result is fundamental delays in getting those answers back.

Latency and Throughput in Parallel Systems

We need to think in general terms about the performance of computer systems. Let me explain the basics using an example of courses and their prerequisites.

Imagine that every student in a certain department needs to take every course shown as a rectangle in this diagram. An arrow from one box to another indicates a prerequisite: the course at the source of the arrow must be taken before the course at the destination of the arrow. A longest path in this diagram is highlighted along the top, with courses that are not grayed out.

Even just from the perspective of one student, a process of completing course requirements is parallel: many activities can be happening at once, here taking multiple courses. Following terminology from running-time analysis of parallel algorithms, we will call the longest path in such a diagram the critical path. We use depth to refer to the length of a parallel workload’s critical path, and we use work to refer to the total number of steps (courses in this example), within a taxonomy introduced by Blelloch and collaborators a few decades ago

An important theorem in this domain is that a given parallel workload must run for a number of steps at least equal to its depth. This conclusion should be intuitive after digesting the terminology: a long prerequisite chain in degree requirements indeed implies a minimum number of terms to complete the degree.

In general, we run many instances of a parallel workload at once. For our courses example, this phenomenon corresponds to having many students enrolled at once. The time from the beginning of one execution of the workload to full completion is called latency. The number of workload instances finishing per step is called throughput. Our example connects latency to how long it takes one student to graduate and throughput to the number of students graduating each term.

Clearly both latency and throughput matter in the running example, but latency is more important from the student perspective. It is no comfort to learn that thousands of students graduate each term if it also takes 100 terms to graduate.

Critical Paths of Deep Neural Networks

Most people encountering the term “deep learning” naturally assume that “deep” is only good news: the neural network is doing something hard and complicated. The term refers more specifically to how many layers a neural network has. This diagram shows roughly the kind of structure of a deep neural network, with layers arranged horizontally, each layer including many artificial neurons, spread out vertically.

Each neuron has inputs coming from (in general) many others from the previous layer. Every neuron is like one course in our earlier example: one unit of work that needs to be performed eventually, to generate a complete answer to a question, with dependencies on some previous steps (here from the previous layer). Note that, as the highlighted path shows, the critical path now runs through all layers, so depth and thus latency are proportional to the layer count. Since “deep” refers to the number of layers, here we have our illustration of how “deep” can mean “slow.” There are many ways that a deep neural network could be evaluated as a parallel workload, but, without some higher-level rearrangement of computation, generating an answer will take time at least proportional to layer count.

Of course, there remain all kinds of tricks for boosting throughput. Most obviously, we expect that roughly all neurons in a single layer can be evaluated at once. The standard computer-systems technique of pipelining is also used, just as we would expect from our earlier example of courses: many executions of the neural network can be run simultaneously, where, at any moment, each layer is working on a different user request. However, these optimizations only boost throughput and, in fact, can introduce extra coordination across steps that even increases latency (e.g., pipelining breaks a computation into pieces that incur extra overhead communicating with each other through queues). While AI companies are motivated to control costs by improving throughput, an individual end user’s experience is more dependent on latency, even in the presence of optimizations like batching (executing requests from different users simultaneously). That user can pay lower fees thanks to throughput-improving resource sharing, but it’s hard not to notice that, say, a chatbot takes a long time to finish returning an acceptable answer, and latency can matter even more for emerging use cases.

It can also be argued that the diagram above is oversimplified. Cutting-edge neural networks don’t literally feed neurons with just outputs from immediately previous layers. An especially common technique is KV caching, which feeds each neuron a somewhat-complex summary of past steps. However, these variations don’t change the fact of long sequential dependencies (critical paths), with lengths proportional to layer counts. Two other common techniques worth mentioning are attention, which manipulates dependencies within layers but basically maintains critical-path structure across layers; and residual connections, which effectively allow some connections between nonadjacent layers but maintain the normal kind of connection, too, which generate roughly the same critical paths.

Recapping, by virtue of how deep learning is organized with long critical paths, it is unavoidable that some steps just need to happen after others, and in fact long chains of such dependencies develop, forcing delays before final answers can be delivered.

Deep All the Way Down, Multiplying Latencies

Now let’s consider other elements in agentic systems being built on top of LLMs today.

First, an LLM generates a full answer by calling a neural network repeatedly, once to generate each token (which is roughly a single word). The input to the neural network is a representation of the tokens that were produced previously, which leads to a long critical path stepping sequentially through all invocations of the neural network. In other words, when the neural network has depth d and we are generating an LLM response of t tokens, the critical path at this level of system detail has length proportional to dt.

How exactly do we arrive at depth about dt? At the level of detail in this last diagram, we easily trace out a critical path of length t, starting with the first neural-network invocation, whose output flows into the next, and so on until the last neural-network invocation. However, each box labeled “DNN” (for deep neural network) is a copy of the prior diagram, where we traced out critical path of length d. Dropping in those copies, the path segments connect together to reach length proportional to dt. Intuitively, the system makes t queries to the DNN in-order, and, for each query, we have to pause and wait for the d sequential steps of the DNN.

A few techniques allow deviating from this kind of workload diagram. For instance, speculative decoding uses cheaper neural networks to guess batches of answers, which are then checked in single shots by more-expensive neural networks. However, this technique only improves latency by a small constant factor – not enough to offset the multiplicative compounding of latency. As a result, latency remains proportional to depth of neural network and length of LLM output.

Now consider how a coding assistant like Claude Code works. It can take on many tasks in parallel, but some objectives still depend on relatively long sequences of steps – where many individual steps are copies of the whole LLM flow we just covered. Also, some steps make arbitrary tool calls to external programs that introduce latency of their own. This diagram shows the coding agent calling a compiler to get feedback on relatively shallow problems it finds in code, as well as a test runner to see which inputs from a test suite the program fails to generate the right answers for.

By now you may be used to spotting the problem for latency: every LLM node in this diagram has critical path length proportional to dt. Now consider the longest sequence of high-level steps in the new diagram, with length s. We are up to depth at least dts, before even taking into account the latency of the tool calls. The time it takes to get a complete answer to one top-level question is being multiplied by a further factor for every level of additional sophistication that we introduce. The reason is again that every LLM box expands into a full copy of the last diagram, and the path segments visible directly in this picture connect with the s copies of the critical path of length about dt from before. Latency has been compounding through all layers of the system, from neural network to repeated next-token prediction to higher-level agentic workflow. Adding functionality doesn’t add to latency but instead multiplies it.

Conclusion

The way that mainstream generative-AI systems are being designed today exhibits a fundamental phenomenon of multiplicative compounding of latency, with intensity proportional to how many levels of additional functionality are added. Latency isn’t everything, but getting full answers to questions sooner is clearly better. For instance, the programmer using an AI coding assistant must typically wait out the full latency before doing code review and testing to finalize a code contribution.

The best rejoinder to these observations about current generative-AI systems is that we don’t seem to have come up with other ways to solve the same problems with nearly the same level of solution quality. Perhaps the latency penalty is unavoidable for those problems. Two posts from now, I’ll present a framework addressing that response from an unusual angle. Zooming out even further than what this post covers, with other architectural changes, we can avoid such high latency by shortening critical paths.

First, though, in the next post, I want to present the challenge of explainability for machine learning, contrasted with an older style of artificial intelligence.