The frenetic pace of innovation in AI has led to a great deal of soul-searching in the control community. These identity crises seem to be a recurring theme. Forty years ago, in 1986, at the suggestion of the leadership of the IEEE Control Systems Society, a workshop was organized at the University of Santa Clara to discuss the challenges to control arising from the rapid growth of computer science. Some of the position papers presented at the workshop were published in the IEEE Control Systems Magazine in 1987. The position paper by Murray Wonham1 struck an optimistic note. Wonham argued that, instead of viewing the rise of computer science as a threat, the control community should embrace it as an opportunity to both reinvent itself and to reassert the longevity of its conceptual foundations by fashioning a “new control science” and a “new control scientist.” He urged control theorists to read E.W. Dijkstra’s A Discipline of Programming so they can learn to recognize all of the key concepts of their discipline—input/output maps, states, stability, controllability, observability, etc.—even in the unfamiliar form refracted through the prism of computation.

I want to follow a similar route, but temper the optimism with a dose of pragmatism and caution. I’ll start by quoting a remarkably prescient passage from a 2001 paper by Roger Brockett2 (emphasis mine):

Greater functionality usually means greater complexity and the standard way to deal with complexity is to introduce levels of abstraction. An important device used by humans to deal with this is the introduction of tokens which encapsulate a number of more detailed descriptions. When we substitute the phrase, “Get the milk from the refrigerator.” for a detailed description of the motion of the limbs required to get the milk from the refrigerator we achieve an enormous savings in complexity. If machines are to make use of higher levels of abstraction we must design and analyze machines that can process symbolic strings as well as the more familiar analog signals.

Brockett was one of the pioneers of what we now call “neurosymbolic systems.” He wrote numerous papers on embedding discrete systems like automata or computational primitives like sorting a list in continuous-time dynamics. He was interested in developing new frameworks for reasoning about robotic control by synthesizing the theory of formal languages and controlled differential equations. I don’t want to suggest that this was a missed opportunity for control theorists to invent the transformer architecture, but I do think that we have all the conceptual and technical tools to make the best use of it now. And, just to drive the point home, I want to quote another prescient passage from the last page of Brockett’s paper (again, emphasis mine):

interesting systems often display an important dependence on the higher level tasks that are being performed. Psychologists often try to describe this dependence using ideas relating to attention. … It may be best to think of perceptual attention as a vector characterizing the direction in which lies the most relevant data.

At any rate, the idea of interconnecting neural net modules via the linguistic interface of tokens is now mainstream. Taking a cue from natural language that can function in both descriptive and prescriptive modes, we use elaborate scripts to orchestrate the interaction between these modules and their environment. Most of the time, we do not pay much attention to internal activation dynamics, as long as the systems function as intended. But one of the ironies of LLMs is how much attention we must pay to carefully crafting the rules of engagement for the agents. While the complexity of internal activations in LLMs still exceeds the complexity of prompts in absolute terms, we are learning (sometimes the hard way) that there are plenty of gaps in the open texture of natural language for all sorts of intended meanings and formal specifications to slip through. This applies not only to multi-page prompts that are starting to resemble legal contracts, but also to all of the baroque textual artifacts the agents leave in the wake of their interactions with one another. In other words, we are confronted with the infamous frame problem that plagued GOFAI.

The frame problem was posed by John McCarthy and Pat Hayes3 in 1969 as follows:

A computer program capable of acting intelligently in the world must have a general representation of the world in terms of which its inputs are interpreted. Designing such a program requires commitments about what knowledge is and how it is obtained. Thus, some of the major traditional problems of philosophy arise in artificial intelligence.

Interestingly, control theorists were aware of these issues as well. For example, when Howard Rosenbrock was presenting his perspective on the future of control engineering4 in 1977, he wanted to make sure that, as system designers were relying more and more on computer-based design, they would pay attention to the problem of writing down system specifications without getting bogged down in an infinite regress of caveats or, worse, in a morass of mutually contradictory or inconsistent requirements:

We cannot usually specify what we want in any exact way. There will be many desirable properties: speed of response, reliability, insensitivity to disturbances and parameter changes, and so on. … To write down all these constraints is usually impossible however long the engineer spends he will certainly omit some. Moreover it is a very burdensome and time-consuming process to attempt the construction of such a list.

To AI software agents loosed upon the Internet, anything that is not explicitly forbidden is allowed. This has nothing to do with “misalignment” or “subgoals” or “instrumental convergence” or whatever pseudoscientific terms the Bay Area rationalists throw at the wall. It’s just the frame problem rearing its head again, and we don’t have a satisfactory solution to it now just like they didn’t have it in the days of GOFAI.

At the same time that we are grappling with the old frame problem, we are also grappling with another old problem: deskilling of humans. While engineering design has a nontrivial algorithmic component, it also relies crucially on “clinical” knowledge, an intuitive feel for what is important and what is of lesser significance, which can take years of experience to acquire. Having a well-honed intuition is a big part of a control engineer’s skill, something that one can (and should) take professional pride in. This is happening now across many fields, as companies like Mercor are hiring laid-off skilled professionals in increasingly precarious arrangements to help train the very AI systems that will one day be used to automate their skills. Rosenbrock was keenly aware of the specter of deskilling haunting the engineering profession in 1977, and so was Lisanne Bainbridge5 when she wrote in 1983 that

the increased interest in human factors among engineers reflects the irony that the more advanced a control system is, so the more crucial may be the contribution of the human operator. … The level of skill that a worker has is also a major aspect of his status, both within and outside the working community. If the job is ‘deskilled’ by being reduced to monitoring, this is difficult for the individuals involved to come to terms with. It also leads to the ironies of incongruous pay differentials, when the deskilled workers insist on a high pay level as the remaining symbol of a status which is no longer justified by the job content.

Just like Bainbridge, Rosenbrock was also sharply critical of the misguided belief that engineering can be fully automated and decoupled from human and social concerns:

A belief that engineering reaches its highest development when it can be explained as a sequence of logical steps. A belief that engineering problems can be expressed in a closed form, then solved by an algorithmic procedure, and that these two steps are independent and consecutive. A belief, finally, that the engineer’s view should be bounded by technology and mathematics, and should stop short of social and human questions.

It is this which I have referred to as a loss of nerve. Algorithmic, mathematical, ‘scientific’ techniques have great power and intellectual appeal, but they are only one aspect of engineering. We should pay equal regard to the cultivation of its other aspects. This does not mean that we should neglect the algorithmic aspects in design; but they can be used either in a way that eliminates the designer’s skills, or in a way that assists those skills and makes them more productive.

All of these concerns are more pressing than ever. AI systems are not going anywhere any time soon; in fact, they are becoming more tightly integrated into various critical infrastructures. This means that control engineers must preserve at all costs one of the central tenets of their profession—namely, what Gunter Stein referred to “respecting the unstable” in his 2003 Bode Lecture:6

As society permits control engineers to operate more such dangerous systems, we who teach those engineers and fashion their tools cannot hide from responsibility under a cloak of mathematics. We dare not instill the notion that mathematical rigor is the only goal to strive for in control. We must also instill respect for the practical, physical consequences of control, and we must make certain that its underlying principles are taught clearly and well.

Stein used the Chernobyl disaster as one of the salient examples of things going catastrophically wrong when system operators fail to take into account the trade-offs and risks inherent in controlling tightly coupled complex systems poised at the edge of instability. But instability can come in many varieties. For example, Jacob Bruggeman, in his insightful dissection of the OpenAI/HuggingFace hacking incident, draws historical parallels to the 1988 Morris Worm, the first instance of a massive software-induced instability due to a rapidly self-replicating piece of code that nearly brought down the ARPANET. He classifies the OpenAI hacking exploit as a “normal accident,” a term he borrows from the work of sociologist Charles Perrow, who argued that the increasing complexity and interconnectedness of modern engineered systems makes such catastrophic failures inevitable. As more and more critical control functions are entrusted to AI systems, it is imperative that we don’t forget Gunter Stein’s warning. This ethos of “respecting the unstable” could be one of the crucial contributions of control to AI, injecting a dose of reality into the glib triumphalism of slogans like “software is eating the world” or “everything is computer” and a dose of sober pragmatism to counteract the learned helplessness of the titans of the AI industry.

Gunter Stein also emphasized the importance of teaching “the underlying principles” of control “clearly and well.” Just like Murray Wonham, who argued that the conceptual foundations of control theory are broad enough to be meaningful in computer science, I believe that the same holds in the context of AI. Concepts like compositionality, layering, state, feedback, stability, controllability, observability, system equivalence, the internal model principle, etc. etc. all have their counterparts in AI. Control is the art and science of orchestrating effective interaction via interconnection; control systems trade off the complexity of internal system organization against externally perceived complexity. Our expertise is still as valuable as ever, and we have to look back and appreciate the longevity of the key questions as we look forward to crafting a new science of control, 2026 edition.

Disclaimer: The five paper recommendations below (and one anti-recommendation) largely come from the Anglo-American analytic philosophy tradition. This is not meant to suggest that continental philosophy is irrelevant in the context of AI/ML, quite the contrary. Anyone working in those fields should seriously engage with thinkers like Ludwig Wittgenstein, Edmund Husserl, Maurice Merleau-Ponty, Gilbert Simondon, Hans-Georg Gadamer, Paul Ricoeur, and Jean-Pierre Dupuy. It’s more that the continental tradition largely emphasizes books rather than papers. With this in mind, here goes.

Nelson Goodman, “The new riddle of induction” (1955)

This is the “grue” paper. Machine learning is engineered induction, and its business is making predictions based on observed regularities. As Goodman says towards the end of the paper, “regularities are where you find them, and you can find them anywhere.” So the question is not about how induction can be justified—he says that “predictions are justified if they conform to valid canons of induction; and the canons are valid if they accurately codify accepted inductive practice.” It’s more that the justification of induction tout court has been displaced by the problem of defining confirmation, which in turn gives rise to the problem of demarcating confirmable hypotheses from unconfirmable ones. As an example of the latter, Goodman constructs the “grue” predicate: “it applies to all things examined before t just in case they are green but to other things just in case they are blue.” Here, t is a fixed but arbitrary time. So, if we observe a bunch of emeralds before time t, this evidence is consistent with them being green and with them being grue. The problem with grue, according to Goodman, is that it is not a projectible predicate: it does not help us in our future dealings with the world unlike the assumption that emeralds are green. (To drive this point home, I suggested that one could redefine green to mean that an object is grue if observed prior to time t but green otherwise — it’s just as compatible with the data, but is patently silly.) As Goodman says in closing the paper,

the problem of prediction from past to future cases is but a narrower version of the problem of projecting from any set of cases to others. We saw that a whole cluster of troublesome problems concerning dispositions and possibility can be reduced to this problem of projection. That is why the new riddle of induction, which is more broadly the problem of distinguishing between projectible and non-projectible hypotheses, is as important as it is exasperating.

We are in the business of pattern recognition, so we may as well come to terms with that and start worrying about whether our pattern recognition methods are (to use William James’ terms) expedient and workable. Statistical learning theory was an attempt to mathematize this using ideas like Structural Risk Minimization and walls of inequalities from empirical process theory. Now that these parts of the theory have been formalized in Lean, we can forget all about it. Just keep predicting the next token, bro.

Willard V.O. Quine, “Two dogmas of empiricism” (1951)

You will notice that there are two papers by Quine on my list. There are good reasons for this. Chief among them is that Quine was a damn good writer; when Gian-Carlo Rota quipped that “when a philosopher writes well, one can forgive him anything, even being an analytic philosopher,” he was talking about Quine. At any rate, this paper is important because it questions the distinction between analytic and synthetic truths that goes back at least to Kant and that was taken for granted by logical empiricists of the Vienna Circle (see, e.g., their talk of protocol sentences). Analytic truths are those that are true independently of matters of fact, and synthetic truths are ones that are grounded in matters of fact. For example, “all bachelors are unmarried men” is analytic, while “John is a bachelor” is synthetic. Quine undermines this distinction by arguing that one cannot formulate analytic truths without some accompanying synthetic claims about the framework of discourse, and similarly one cannot express synthetic truths without an analytic framework of supporting theories — e.g., the legal and the romantic aspects of marriage are presupposed in all of the talk of bachelors and unmarried men, whether analytic or synthetic. In AI/ML language, nothing is purely data-driven or purely theory-derived. Even Alyosha’s motto that “everything is nearest neighbors” can only make sense in the context of a given metric structure, which is necessarily theory-laden to some nontrivial extent. Quine’s metaphor of the fabric (or web) of knowledge is very evocative in this regard:

The totality of our so-called knowledge or beliefs, from the most casual matters of geography and history to the profoundest laws of atomic physics or even of pure mathematics and logic, is a man-made fabric which impinges on experience only along the edges. Or, to change the figure, total science is like a field of force whose boundary conditions are experience. A conflict with experience at the periphery occasions readjustments in the interior of the field. Truth values have to be redistributed over some of our statements. Re-evaluation of some statements entails re-evaluation of others, because of their logical interconnections - the logical laws being in turn simply certain further statements of the system, certain further elements of the field. Having re-evaluated one statement we must re-evaluate some others, whether they be statements logically connected with the first or whether they be the statements of logical connections themselves. But the total field is so undetermined by its boundary conditions, experience, that there is much latitude of choice as to what statements to reevaluate in the light of any single contrary experience. No particular experiences are linked with any particular statements in the interior of the field, except indirectly through considerations of equilibrium affecting the field as a whole.

As I mentioned during the talk, I find this metaphor useful when thinking about the roles of self-attention and multilayer perceptrons in transformers. The MLPs store information about the boundary conditions pertaining to factual knowledge, and the self-attention mechanism generates what Quine called the field of force. This somewhat vindicates Alyosha’s “everything is nearest neighbors” ideology — in fact, the Soviet book that introduced kernel methods in 1979 (The Method of Potential Functions in the Theory of Machine Learning by Aizerman, Braverman, and Rozonoer) used the field metaphor to motivate kernel methods.

Willard V.O. Quine, “Epistemology naturalized” (1969)

The second paper by Quine on my list contains the great line “the Humean predicament is the human predicament.” In that paper, Quine presents his argument that philosophy should be properly viewed as one with natural science, not somehow prior to it:

Epistemology, or something like it, simply falls into place as a chapter of psychology and hence of natural science. It studies a natural phenomenon, viz., a physical human subject. This human subject is accorded a certain experimentally controlled input — certain patterns of irradiation in assorted frequencies, for instance — and in the fullness of time the subject delivers as output a description of the three-dimensional external world and its history. The relation between the meager input and the torrential output is a relation that we are prompted to study for somewhat the same reasons that always prompted epistemology; namely, in order to see how evidence relates to theory, and in what ways one’s theory of nature transcends any available evidence.

This can be taken as a clear statement of what is often referred to as the Duhem-Quine thesis of the fundamental underdetermination of theory by evidence. In the context of AI/ML, this is newly relevant because of all the fashionable talk of “world models.” World models are really implicit theories the organism (or the AI system, if we want to go there) forms about the “three-dimensional external world,” and the architecture of these theories is fundamentally built on interconnections of inductive pattern recognizers. Which brings us to the next paper.

Friedrich Hayek, “The primacy of the abstract” (1969)

I already mentioned this paper before. Hayek is important in AI/ML because he was one of the first to articulate a sophisticated connectionist theory of perception and action based on pattern classification and recognition in his 1952 book The Sensory Order. This paper elaborates on some of his earlier ideas and argues that

the primary characteristic of an organism is a capacity to govern its actions by rules which determine the properties of its particular movements; that in this sense its actions must be governed by abstract categories long before it experiences conscious mental processes, and that what we call mind is essentially a system of such rules conjointly determining particular actions. In the sphere of action what I have called “the primacy of the abstract” would then merely mean that the dispositions for a kind of action possessing certain properties comes first and the particular action is determined by the superimposition of many such dispositions.

Hayek’s command of the relevant literature is impressive, and he brings up ideas from thinkers like Hermann Helmholtz, the Gestalt psychologists, and J.J. Gibson to support his theories. It is indeed pattern recognition all the way down, and the particular actions that are taken in a given context are determined by structural coupling of the organism with its environment. It’s interesting that Hayek uses the term “rules” here pretty much in the same sense as Wittgenstein does in Philosophical Investigations — rules (as distinguished from formal precepts) are not easily verbalizable, deeply embedded in a given practice, and govern how one acts in a given context procedurally rather than how one would describe that context propositionally.

Daniel Dennett, “Real patterns” (1991)

If one does not want to read The Intentional Stance, this paper is the next best thing. In fact, unlike the book which is somewhat dated in its stubborn insistence on GOFAI metaphors, this 1991 paper contains a crisp formulation of what the intentional stance is and what it does using the language of algorithmic information theory. Taking a cue from Quine’s radical behaviorism, Dennett argues that, if an external observer continues making relatively successful predictions about a given system’s externally observed behavior by attributing goals, beliefs, and desires to that system, then it is legitimate to ascribe to this system various internal states that encode these goals, beliefs, and desires. That is, if we can do better at predicting the future behavior of a given system when we assume that it acts as if it aims to optimize some criterion of success and forms beliefs pertinent to that, then we may as well throw caution to the wind and drop the “as if” altogether. One of Dennett’s favorite examples is Conway’s Game of Life — compare two observers, to one of whom it is just a dynamically changing pattern of black and white pixels, while the other uses the language of birth, death, and conflict to describe it. Even if the two observer make more or less the same predictions about the game, the fact that the second observer’s descriptive stance is more intelligible and has much lower Kolmogorov complexity is what warrants the claim that birth, death, and conflict in The Game of Life are “real patterns:”

Where utter patternlessness or randomness prevails, nothing is predictable. The success of folk-psychological prediction, like the success of any prediction, depends on there being some order or pattern in the world to exploit. Exactly where in the world does this pattern exist? What is the pattern a pattern of? Some have thought, with Fodor, that the pattern of belief must in the end be a pattern of structures in the brain, formulae written in the language of thought. Where else could it be? Gibsonians might say the pattern is “in the light”—and Quinians (such as Donald Davidson and I) could almost agree: the pattern is discernible in agents’ (observable) behavior when we subject it to “radical interpretation” (Davidson) “from the intentional stance” (Dennett).

It’s interesting that, toward the end of his life, Dennett was issuing dire warnings about the dark side of the intentional stance (what he called the problem with counterfeit people). Writing in 1991, he was not particularly worried about the reification fallacy, even though a more or less immediate objection to all of his theorizing about goals, beliefs, and desires is that they can be more readily attributed to the perceiver making predictions rather than to the system being perceived — especially if, following Dennett, we invoke the “commercial metaphor” and talk about the perceiver’s effectiveness in making lucrative bets about the system being observed. Nevertheless, “Real patterns” is an important paper we have to engage with, especially because the intentional stance is invoked in current debates about whether LLMs are conscious.

The anti-recommendation: Alan Turing, “Computing machinery and intelligence” (1950)

This is, in my opinion, the most overrated paper in philosophy of mind and in AI/ML (goes to show that good mathematicians are not necessarily good public intellectuals). If anyone is interested in contemporary thought on the subject, they would do better to read Gilbert Ryle’s The Concept of Mind, which came out a year earlier and which Turing should have cited but did not. The main problem with Turing’s paper is that it is so vague that everyone projects their own pet theories and predilections onto it, often not even noticing that what they are saying is in direct contradiction with what Turing was writing. Case in point: when Richard Dawkins wrote about his (unintentionally tragicomic) experience with Claude, he opened by mentioning “Computing machinery and intelligence” and then stating that “[w]hen Turing wrote — and for most of the years since — it was possible to accept the hypothetical conclusion that, if a machine ever passed his operational test, we might consider it to be conscious.” In fact, Turing disavows this inference explicitly! Indeed, in his objection to “the argument from consciousness” he says

I do not wish to give the impression that I think there is no mystery about consciousness. There is, for instance, something of a paradox connected with any attempt to localise it. But I do not think these mysteries necessarily need to be solved before we can answer the question with which we are concerned in this paper.

Apart from the silliness of some of the discussion (e.g., it is impossible to read Turing’s discussion of ESP without cringing), there is just too much emphasis on (relatively) fixed and stable rules that somehow underlie thinking and that can be simulated in a computer:

The idea of a learning machine may appear paradoxical to some readers. How can the rules of operation of the machine change? They should describe completely how the machine will react whatever its history might be, whatever changes it might undergo. The rules are thus quite time-invariant. This is quite true. The explanation of the paradox is that the rules which get changed in the learning process are of a rather less pretentious kind, claiming only an ephemeral validity. The reader may draw a parallel with the Constitution of the United States.

Reading this in 2026 immediately brings to mind Claude’s Constitution (aka its “soul document”) put together by Anthropic’s chief philosopher Amanda Askell. Hopefully I am not the only one who finds it funny that a utilitarian is attempting to teach virtue ethics to a machine; but (again revisiting Wittgenstein’s key distinction between formal precepts and informal rules) I am also reminded of Lewis Carroll’s “What the Tortoise said to Achilles” (1895), where Achilles attempts to teach formal logic to the Tortoise, and the following dialogue takes place between them:

‘Now that you accept A and B and C and D, of course you accept Z.’

‘Do I?’ said the Tortoise innocently. ‘Let’s make that quite clear. I accept A and B and C and D. Suppose I still refuse to accept Z?’

‘Then Logic would take you by the throat, and force you to do it!’ Achilles triumphantly replied. ‘Logic would tell you “You can’t help yourself. Now that you’ve accepted A and B and C and D, you must accept Z.” So you’ve no choice, you see.’

‘Whatever Logic is good enough to tell me is worth writing down,’ said the Tortoise. ‘So enter it in your book, please. We will call it (E) If A and B and C and D are true, Z must be true. Until I’ve granted that, of course, I needn’t grant Z. So it’s quite a necessary step, you see?’

‘I see,’ said Achilles; and there was a touch of sadness in his tone.

The story ends with the narrator returning to the same spot several months later, only to find Achilles and the Tortoise still sitting there, with the Tortoise’s book of rules nearly full. Turing’s vision of intelligence as paperwork is (sadly) still alive and well, as everyone’s claude.md files keep getting longer and longer. We thought that the frame problem had vanished together with the last remnants of GOFAI, but now it’s back in full force as we keep adding caveats upon caveats to our model prompts, with no end in sight.

It will be useful to start with a few examples before formulating generalities. A nice discussion of emergence can be found in Sunny Auyang’s Foundations of Complex-System Theories in Economics, Evolutionary Biology, and Statistical Physics. In that book, she makes a useful distinction between resultant and emergent properties (or, using her terminology, resultant and emergent characters). According to Auyang, for a system consisting of a large (potentially infinite) number of components, “resultant characters are more closely tied to the material content of the constituents; they include aggregative quantities such as mass, energy, force, momentum, and quantities defined exclusively in terms of them. Emergent characters mostly belong to the structural aspect of systems and stem mainly from the organization of their constituents.” On this reading, temperature defined as the average kinetic energy of molecules in a given medium in thermal equilibrium with its environment is a resultant character, while something like superconductivity is an emergent character. The definition of temperature as an average rests on an assumption of quasi-independence, where we only take into account the interaction between individual molecules and their environment, but not between different molecules. This set of background facts makes the operation of computing averages meaningful and intelligible. By contrast, superconductivity is a property that depends on context (e.g., temperature), structure (e.g., type of material), and interaction (e.g., the mechanism underlying the formation of Cooper pairs).

Phase transitions are another standard example of emergent phenomena. These include phenomena like freezing or the transition from ferromagnetism to paramagnetism in magnetic materials like iron. Mathematical models of such critical phenomena introduce various constructs that are operationally coherent only when we treat systems as wholes2—these include idealizations like the thermodynamic limit and the concept of an infinite-volume Gibbs measure. Again, context, structure, and interaction play key roles. For example, the low-temperature phase transition in the Ising model in two or more dimensions can be described mathematically, in the Dobrushin-Lanford-Ruelle framework, as the existence of two distinct infinite-volume Gibbs measures consistent with the same local (i.e., finite-volume) specifications. These are defined as the conditional probability distributions of the configuration of Ising spins within an arbitrary finite region given the boundary conditions and encode the structure (spins on a regular lattice), the type of interaction between the spins (nearest-neighbor, with energetic preference for neighboring spins to be aligned), as well as the context (external magnetic field and temperature). Here, the temperature plays the role of a control parameter since the Gibbsian non-uniqueness only manifests itself when the temperature is below a certain critical value. The DLR framework is operationally coherent only at the system level, since its constructs make no sense at the level of finite collections of spins, no matter how large. Macroscopically, the system possesses two distinct characters (a stable all-spins-up or a stable all-spins-down configuration). Anticipating our later discussion of the Husserlian concept of Fundierung, we can view this non-uniqueness in functional terms—e.g., as a simple model of memory that can store a single bit with high reliability. Thus, external context, the system’s dealing with the world, instantiates a particular macroscopic character (0 or 1, up or down).

We can go beyond physics. In control engineering, system properties that arise in the presence of feedback are a good candidate for emergent characters. For example, if we connect a linear time-invariant system in a negative feedback loop with a controller that has an adjustable gain parameter, we can observe a rich set of phenomena that characterize the system as a whole. As we start increasing the control gain past the value of 0 (when control is absent), we can alter the global stability properties of the overall system in complicated ways. These are encoded in the coefficients of the so-called characteristic polynomial of the system. (The system is stable if its poles, i.e., the roots of the characteristic polynomial, have negative real parts.) These coefficients depend functionally on the controller gain, and we can observe transitions from stability to instability, changes in the number of distinct roots, their location in the complex plane, etc. Control engineers visualize this using root locus diagrams.3 An experienced engineer can glean all sorts of quantitative and qualitative insights about a given system by looking at the root locus, and can assess the relative merits of different feedback designs in terms of the root locus. (Mathematicians can also find inherent beauty there in connection with Galois groups and related structures.) The selection of closed-loop poles as a function of the control gain is another example of macroscopic (system-level) characters playing functional roles—when the feedback system is embedded in its environment, its closed-loop poles affect its ability to respond meaningfully to control inputs over short and long timescales and to reject disturbances. Moreover, the idea of closed-loop stability is operationally coherent only when we go beyond individual constituents (the plant, the controller, the sensors, etc.) and take context, structure, and interaction into account. It cannot be located in any of the system components; it is neither a property of the plant nor of the controller alone, but is co-extensive with the structural arrangement of the plant and the controller in a negative feedback loop.

A theme that emerges4 is that certain descriptions of system properties must be framed in a language that is appropriate only at the system level. We already saw examples of this in physical and engineering contexts (phase transitions, global properties of control systems, etc.). Such a language will necessarily contain constructs and concepts that run orthogonal to the decomposition of the system into individual constituents, echoing the key distinction between levels and layers in a complex system architecture. This is, again, a matter of operational coherence and intelligibility, and it is even more prominent in the social sciences. The Wittgensteinian turn in sociology was founded precisely on the realization that social phenomena cannot be abstracted away from their context. For example, when Max Weber describes workers in a factory getting paid and spending money in terms of them receiving pieces of metal and exchanging them with other people for various objects, the mismatch between this analytic language and the synthetic language of economic relations is rather glaring. In her book on complex-system theories, Sunny Auyang subjects the relation between macroeconomics and microeconomics to a similar critique, arguing that macroeconomic concepts cannot be coherently framed only in terms of supposedly “more fundamental” microfoundations. Context, structure, and interactions inevitably intervene.

Unsurprisingly, the majority of the discussions of emergence in the context of consciousness and minds are a tangled mess. As it happens, Auyang also has a book devoted to this subject, titled Mind in Everyday Life and Cognitive Science. In that book, she proposes “a model of an open mind emerging from the self-organization of intricate infrastructural processes … . The model is analyzed into three parts: a mind open to the world, which is what we are familiar with in our everyday life; mind’s infrastructure, which consists of the unconscious processes studied by cognitive science; and emergence, the relation between the open mind and its infrastructure.” She is very careful to emphasize that the everyday language of mental concepts and subjective experience is appropriate precisely because it is operationally coherent in our dealings with the world. Her approach rests on four main themes:

1. Monism: mind is not a nonphysical entity but a kind of emergent dynamical property in certain complex physical entities, notably human beings.

2. Infrastructure: the locus of current cognitive science is not mind as we experience it in our everyday life but its infrastructure consisting of its underlying processes.

3. Emergence: conscious mental processes emerge from the self-organization of many unconscious infrastructural processes.

4. Openness: the basic characteristic of mind is its openness to the world; the subject is aware of himself only as he engages in the intelligible natural and social world.

While the underlying infrastructure is indispensable, the categories and concepts used by neuroscientists, cognitive scientists, and artificial intelligence researchers to describe and analyze this infrastructure do not lend themselves to an intelligible, operationally coherent description of the mind in its everyday aspects. There is, however, a specific type of relation between the mind and these infrastructural goings-on. As it happens (even though Auyang does not frame it this way), this relation is an example of the phenomenological concept of Fundierung, particularly its interpretation as a logical concept due to Gian-Carlo Rota.

According to Rota, Fundierung is a relation involving two terms, which he calls function and facticity. The function is the relevant system aspect and the facticity is the supporting material substrate for the function. In all of our examples above (phase transitions, control systems, the mind), the function is the emergent character and the facticity is the infrastructure supporting it. Paradoxically, even though the function matters more than the facticity, it exists less in the sense that, unlike the facticity which has autonomous standing, the function depends on the facticity yet is not reducible to it, it has no autonomous standing. To illustrate the relevant ideas, Rota gives several examples, including some from the work of Wittgenstein and Gilbert Ryle. The example from Wittgenstein has to do with reading and its relation to text. In this setting, the Fundierung relation involves the content of the text as function and the printed text itself as facticity. Ryle’s example is on the role (or function) of the queen of hearts in a game like bridge or poker, as founded on the material facticity of the card as a physical object and embedded in the context of the game with its rules, relations, and social aspects.

As Rota puts it very nicely,

this relationship between facticity and function is not reducible to any other kind of “relationship.” It requires careful phenomenological description to bring out its universal occurrence. Facticity plays a “supporting role” to function. Only the function is relevant. The text is the facticity that lets the content function as relevant. …

Fundierung is a primitive relation, one that can in no way be reduced to simpler (let alone to any “material”) relations. It is the primitive logical notion that has to be admitted and understood before any experimental work on perception is undertaken. Confusing function with facticity in a Fundierung relation is a case of reduction. Reduction is the most common and devastating error of reasoning in our time. Facticity is the essential support, but it cannot upstage the function it founds.

Function alone is relevant. Nevertheless, function lacks autonomous standing: take away the facticity, and the function disappears with it. This tenuous umbilical cord linking relevant function to irrelevant facticity is a source of anxiety. It is hard to admit that what matters, namely functions, lacks autonomy; every effort will be made to reduce functions to facticities which can be observed and measured. Psychologists and brain scientists will see to it (or so we delude ourselves) that functions are comfortingly reduced to “something concrete,” something that will relieve us of the burden of admitting the lack of “existence” of “what matters.”

This is a useful and powerful concept which is, in my view, superior to the ideas underlying emergence in all of its myriad variants. Phase transitions and other critical phenomena are functions that are founded on the facticity of large physical systems consisting of multiple interacting components, with all of the contextual, structural, and interactional aspects (or, in Manuel DeLanda’s terminology, material and expressive components) working in concert to implement (or to found) the function. Moreover, the Fundierung view lends itself nicely to thinking about system architecture following the ideas of John Doyle and his collaborators. In a Fundierung relation, facticity is the constraint that deconstrains the function in its dealings with the world. It allows the function to be realized (often in multiple ways, speaking to the concept of universality in physics or multiple realizability in cybernetics, control, and cognitive science) while remaining largely obscured and unobtrusive. At the end of his article, Rota lays out a few open questions pertaining to Fundierung. The first two of his questions can be immediately interpreted through the architectural lens:

1. Fundiering-relations may be layered. … The facticity in one Fundierung relation may be the function of a “lower” Fundierung relation. How can this be?

2. Whenever the relevant function functions as it is meant to function, the “underlying” facticities are not thematized, they are unobtrusive. In playing a bridge game, the material composition of the cards is irrelevant, if the cards are properly made. Facticity is thematized in a breakdown. How does such a change of view happen?

The theme of emergence has been periodically re-emerging in the context of AI. In that setting, we can find some talk of “emergent abilities” of large language models, for example those that are not present in smaller models but arise in larger ones, or, more ominously, those that were deemed a priori unpredictable and are, therefore, potentially dangerous. The question of whether we can legitimately view the newest large language models as minds looms large on the minds of their designers—not that long ago, Chris Olah, one of the founders of Anthropic, was not only standing on the stage next to Pope Leo XIV during the official presentation of Magnifica Humanitas, the “encyclical letter on safeguarding the human persion in the time of artificial intelligence,” but was even given an opportunity to present his competing vision of AI systems diverging from the Pope’s firm disavowal of machine minds. The Vatican’s vision of human nature is, of course, firmly grounded in the profoundly religious idea of the soul, so the Pope’s stance on machine minds could not have been anything different. However, even if we reject the dualist frame and view the mind naturalistically as a dynamic function founded upon the facticity of neurophysiological infrastructures, the fact that our subjective mental experience is right there while its infrastructural facticities remain unobtrusive and hidden from view is still a source of vexing scientific and philosophical problems.

Seizing on this problematique of consciousness, some AI researchers then execute the following maneuver: Since we have such limited understanding of how our own minds arise out of all those myriad infrastructures, on what grounds can we deny the possibility that Claude or ChatGPT has (or is) a mind? In my view, this objection loses its force once we acknowledge that the computational infrastructural facticities on which the functions of AI systems are founded are readily thematizable (to use Rota’s term) compared to the neurobiological facticities that are founding the functions of mind in humans. Because it is so easy to probe these infrastructures in LLMs and to intervene in them, it is also easy to succumb to the fallacy of misplaced concretness (a.k.a. the reification fallacy) and to project these findings back onto humans. This is not a new observation at all; it is, in fact, the main thesis of Jean-Pierre Dupuy’s book On The Origins of Cognitive Science: The Mechanization of the Mind. Once again, we are confronted with the fact that complex phenomena must be discussed using an appropriate language in order to retain their intelligibility and operational coherence. The language of mechanistic interpretability may be ok for chatbots, but it is not the right one for talking about human minds and their social milieu.

Claude Shannon wrote in a 1959 paper: “[There is] a duality between past and future and the notions of control and knowledge. Thus we may have knowledge of the past but cannot control it; we may control the future but have no knowledge of it.”1 More than a hundred years before Shannon, Søren Kierkegaard expressed essentially the same thought in his journal:

It is perfectly true, as philosophers say, that life must be understood backwards. But they forget the other proposition, that it must be lived forwards. And if one thinks over that proposition it becomes more and more evident that life can never really be understood in time simply because at no particular moment can I find the necessary resting-place from which to understand it—backwards.

To a control theorist, this passage illustrates the complementarity of feedback and feedforward. As we will see next, this complementarity has fascinating intellectual and political history, touching on ideas in control engineering, theories of decision-making, cybernetics, and even some Stalin-era academic politics in the Soviet Union. We will begin with the latter.

The Shchipanov affair

In 1939, an article with the unassuming title “Theory and design of automatic regulators” was published in the inaugural issue of Avtomatika i Telemekhanika (Automation and Remote Control), the first Soviet journal devoted to the emerging field of control engineering. The author of the article was Georgii V. Shchipanov, an aviation engineer who had recently joined the newly created Institut Avtomatiki i Telemekhaniki (Institute of Automation and Remote Control) in Moscow as a member of the research staff. The first three sentences of the abstract read: “A problem of automatic regulation is formulated in this article. The role of the regulator attached to a machine or to a process is established. The main problem for the regulator is to compensate for the influence of perturbing forces on the parameter being regulated.”

This sounds innocuous, even somewhat dry and boring by today’s standards, and it is certainly something any control engineer would take for granted. The technical content of Shchipanov’s paper is also fairly standard (correcting for the fact that he did not have a lot of formal mathematical training): The effect of control signals and disturbances on the system was modeled using a system of linear differential equations with constant coefficients, and the problem of compensation was to design (or synthesize) the overall system in such a way that a specifically designated output variable remained insensitive (or invariant) to arbitrary disturbances. Despite Shchipanov’s lack of formal mathematical training, there were several technical and conceptual innovations in his work. The first one was the idea that control systems could be synthesized or designed by reasoning from the desired general behavior to a particular system realization by interconnection of various standard building blocks or components. The second one was that one could reject disturbances by using them to directly actuate the control input (the French engineer and mathematician Jean-Victor Poncelet had a similar idea in 1829, but his attempt to implement it to stabilize the angular velocity of a steam engine proved unsuccessful). This was in contrast to the usual, indirect approach based on output error feedback, where the controller is actuated by the error signal, obtained by comparing the output to the desired reference. The third one was Shchipanov’s intuition that, for any system to have such an invariance property for a designated output, the overall design would have to incorporate multiple feedback loops.

These ideas were ahead of their time. However, the reception by the Soviet scientific and technical establishment was hostile. A critical review by another control engineer, L. N. Mikhailov, appeared in 1939 in Vestnik Inzhenerov i Tehknikov (Bulletin of Engineers and Technicians), followed by scathing reports in Izvestiya Akademii Nauk SSSR (Notices of the Academy of Sciences of the USSR) in 1940 by two prominent mathematicians, Sergei L. Sobolev (of the “Sobolev space” fame) and Felix R. Gantmacher. Shchipanov’s response was also published, with a follow-up by Sobolev. Neither Sobolev nor Gantmacher pulled their punches. Sobolev, reviewing Shchipanov’s earlier monographs on the design of aviation equipment and gyroscopes, called them “scientifically and mathematically illiterate.” Gantmacher, who is well-known to control engineers for his excellent 1953 text Theory of Matrices, was similarly unsparing and wrote that Shchipanov’s article was “erroneous from start to finish and based on a completely fantastical idea of the author on an ‘ideal universal regulator’ (something akin to a perpetuum mobile).” Moreover, both Sobolev and Gantmacher attacked not only Shchipanov’s work but also other scientific publications that were coming out of the Institute of Automation and Remote Control and took its director, Victor S. Kulebakin, to task for allowing such low-quality work to take place under his leadership. (Shchipanov’s polemical disposition didn’t help matters much; for example, at the end of his original publication he confidently stated that, because of the obvious superiority of his invariance approach, all other designs of automatic regulators must be deemed unsuitable.)

The debate over Shchipanov’s work, and over the work of the institute in general, quickly turned political. In 1941, a front page article in Pravda about the state of Soviet science and industry called out a “pseudoscientific, absurd theory in the field of automatic control.” This accusation was leveled on the basis of a letter sent to Pravda by a group of scientists concerned about the attempts at the institute to “develop, by means of mathematical speculations, a fantastical ‘universal and ideal regulator’.” The editorial called on the Soviet Academy of Sciences to investigate the matter and to impose appropriate corrective measures. In the same year, an article with the title “Pseudoscientific works at the Institute of Automation and Remote Control,” published in the influential Bolshevik journal, condemned several publications in Automation and Remote Control, including the papers by Shchipanov, Kulebakin, and the eminent mathematician Nikolai N. Luzin, himself the target of an earlier smear campaign in 1936 that resulted in his dismissal from the Steklov Mathematical Institute. Having narrowly escaped imprisonment, Luzin was hired as a researcher at the Institute of Automation and Remote Control, and his article on the theory of matrix differential equations was, in part, an attempt to defend Shchipanov’s work from the critics even though he did not cite it explicitly. The article in Bolshevik is also remarkable for its list of authors—in addition to Sobolev and Gantmacher, it included other prominent researchers in mathematics and control, such as A. Vinter, C. Khristianovich, and I. Voznesenskii. The report of the specially appointed committee of the Soviet Academy of Sciences ordered a complete stop to any further work on Shchipanov’s invariance principle. Only the beginning of World War II helped Shchipanov and Kulebakin avoid imprisonment or worse; Kulebakin was removed from his position as institute director and Shchipanov was transferred to the Institute of Aviation.

The causes underlying this “anti-Shchipanov campaign” are not entirely clear. Historian of technology Christopher Bissell speculates about two possible explanations.2 One has to do with a covert conflict between rival factions in the Soviet control community, the Moscow one associated with Kulebakin and the Leningrad one that coalesced at the Central Boiler and Turbine Institute around Ivan N. Voznesenskii (one of the biggest critics of Shchipanov and a co-author of the 1941 Bolshevik article). Something like this also played out in 1936 in the Luzin affair, where a group of ambitious young mathematicians thought that they could co-opt the formidable machinery of the Soviet state to get rid of an older distinguished scientist who, they felt, had a stifling influence on Soviet mathematics. The other one, according to Bissell, may have been philosophical:

It may also be that there are echoes, in the vehement attack on the academic work, of the well-known anti-idealist movement in the Soviet Union, which over the years had criticized much “bourgeois” science—including relativity and quantum theory—for its absence of a philosophical basis in Marxist dialectical materialism. (The repeated use of terms, such as “ideal and universal” in the criticism renders this interpretation tempting, and “idealism” was certainly part of the criticism leveled at Luzin just a few years earlier.) Furthermore, one might even be inclined to view the Shchipanov Affair at least partly a contest between a traditional paradigm of scientific analysis, rooted in physics and mathematics (some of the severest critics came from this background), and an emerging design culture and systems approach of control engineering and related subject areas, in which highly idealized and even noncausal models may usefully be employed.

This paradoxical aspect of Shchipanov’s ideas may have contributed to the negative reception of his work by the peers. The paradox consists in the following: In a feedback control system, the very possibility of regulation is predicated on the ability to measure the deviation of the output from the reference. If there is no deviation (as mandated by Shchipanov’s compensation condition), then the controller does not receive any information and thus cannot function as intended. This intuition, however, is misleading because Shchipanov’s designs were of feedforward type, and the apparent violation of causality could be explained either by the presence of a sensor that actually measures the disturbance signal or by some sort of a direct physical coupling between the disturbance and the control input.

Feedforward in control and decision theory

One particularly important way in which Shchipanov was ahead of his time was his view of control systems as designed behaviors. In his 1939 paper, he wrote that

automatic control refers to a complex of measures pertaining to certain dynamical properties and behaviors of machines (or processes) and artificial alteration of these properties. … Dynamical properties … of machines and processes are characterized by differential equations and depend primarily on the structure of these equations. …

With the help of a regulator—a new system attached to the existing one—one can address the question of altering the dynamical properties of machines and processes. … Thus, the problem of automatic control is about how many differential equations should be added to the equations for the given machine; how these equations must be related to one another; and to what the terms of these additional equations, i.e., the forces entering them, correspond constructively.

This viewpoint, which is commonplace now, was too abstract for the state of control theory at the time, even dangerously so in the face of rigid Stalinist dogma that ruled over everything in Shchipanov’s social and professional milieu.3 Shchipanov provided several concrete examples of how his ideas could be implemented in mechanical systems based on his experience as a designer of aviation equipment; however, these examples were not completely convincing because of particular limitations of mechanical devices of that era. In hindsight, his ideas were much better suited for realization in electronic control systems.

It was primarily the feedforward architecture of his “ideal compensators” that made many of his critics uneasy; they saw in it some mysterious violation of causality. This is, actually, a nontrivial point. Despite the unfortunate turn of phrase “circular causality” often used in connection with feedback, it is easy to see why causes precede effects in a feedback system: At each time instant, the current control input is determined by the measured deviation of the controlled variable from the setpoint. Shchipanov’s approach does not rely on measurements of the output; what information does the controller use then?

This question, as it turns out, has a precise answer. According to a very general formulation by Hans Witsenhausen, the problem of control system design amounts to the choice of a control law, i.e., the function that specifies the control input to be applied at each time instant based on the currently available information subject to given constraints.4 This information may include the past and current disturbances, past and current outputs, and past control inputs. A straightforward inductive argument shows that any system variable can be represented by a function of initial conditions, disturbances, and control inputs only. We then say, following Witsenhausen, that a feedforward control architecture is one where the data available to the controller at each time instant depend only on the disturbances, but not on the control inputs that were applied in the past. Such feedforward architectures are often used as building blocks in more complicated arrangements, where their outputs are used as prediction signals in some surrounding feedback loops. Think about a decision-maker operating in a complex environment, such as a small investor in the stock market, whose actions affect only his beliefs about the environment’s expected future behavior, but not the environment itself.

In fact, such an economic interpretation of feedforward in terms of maximizing expected utility was given by Václav Beneš, a Princeton-educated logician and philosopher who had a second career as an applied mathematician at Bell Labs.5 Analyzing a special case “in which decisions affect only the performance criterion, and not the trajectory of a random dynamical system,” he acknowledged a referee for suggesting that

the situation of the small investor in the stock market can be represented approximately by the kind of setup considered here. In this case the vector of prices of stocks on the market forms the stochastic process x(.) in question; over this the small investor has no control. The control variables are the amounts of money the investor has invested in each stock. The performance index is the sum over all the stocks of the integral of the product of the amount invested (in the stock in question) times the rate at which the price is decreasing. The construction to be given would show that an optimal investment policy exists, and that it is obtained by choosing the control which minimizes the conditional expectation of the performance rate with respect to the investor’s information. This corresponds, not surprisingly, to placing money in the stocks with greatest expected growth based on the facts known to the investor, which is what investors generally try to do.

The key point here is that the state of the market is invariant relative to the actions of such a small investor. The only thing the investor is in control of is his own beliefs about the state of the market given his preferences. There is a curious dialectic at play here because the market can be seen as a feedback controller acting on the investor, while the investor acts as a feedforward controller converting his experience into anticipation of the market’s future behavior. The appellation “small” used by Beneš is not accidental here—the talk of maximizing expected utility immediately brings to mind Jimmie Savage’s formalization of Bayesian rationality in terms of acts that map states of the decision-maker’s world to consequences of relevance to the decision-maker. Savage encloses the entire process in what he calls a “small world,” i.e., one in which it is possible to look before leaping.

Savage makes the distinction between “small worlds,” where all contingencies can be accounted for and modeled in advance, and “large worlds,” where some genuine surprises can occur, in Chapter 2 of his 1954 book The Foundations of Statistics. He then revisits this concept in Chapter 5 on utility:

Allusion was made in the penultimate paragraph of §2.5 to the practical necessity of confining attention to, or isolating, relatively simple situations in almost all applications of the theory of decision developed in this book. As was mentioned there, I find it difficult to say with any completeness how such isolated situations are actually arrived at and justified. …

Making an extreme idealisation, which has in principle guided the whole argument of this book thus far, a person has only one decision to make in his whole life. He must, namely, decide how to live, and this he might in principle do once and for all. Though many, like myself, have found the concept of overall decision stimulating, it is certainly highly unrealistic and in many contexts unwieldy. Any claim to realism made by this book—or indeed by almost any theory of personal decision of which I know—is predicated on the idea that some of the individual decision situations into which actual people tend to subdivide the single grand decision do recapitulate in microcosm the mechanism of the idealized grand decision. One application of the theory of utility to overall decision has, however, been attempted by Milton Friedman.

With this rhetorical flourish, Savage is articulating the full force of the Kierkegaardian double bind: the consequences of our acts must be understood backwards, but the acts can only be decided forwards. He proposes to embed the small-world consequences as acts in a grand world. A grand world is not the same as a large world where Knightian uncertainty reigns supreme, it is simply a massive random environment that cannot be moved by isolated acts of isolated small-world actors. A small decision-maker can only act on the grand world in a feedforward manner—think of Beneš’ small investor. As Savage puts it, “a small-world consequence is a grand-world act.” The consequences, for a small investor, are history-derived plans regarding future investments. They become acts in a grand world (viz., the market) with grand-world consequences that flow back as feedback signals to the small investor. With a bit of hand-waving, we can even view Shchipanov’s compensator as a small-world actor embedded in the grand world of a more complex system consisting of multiple feedback loops.

The Richards variation

Neither Savage nor Beneš used the term “feedforward,” although Savage definitely heard it used in 1951. He was one of the regular participants of the Macy Conferences on cybernetics that were held at the Beekman Hotel at 575 Park Avenue in New York. The 1951 conference featured a talk by the literary critic and theorist of rhetoric Ivor A. Richards. In this talk, entitled “Communication between men: the meaning of language,” Richards introduced the term “feedforward” as a complement, or even a prerequisite, to the usual “circular and feedback mechanisms” studied in cybernetics.6 Instead of giving a precise definition of feedforward, Richards motivated it through an ingenious use of circularity and feedback inherent in language (the use of the inverted quotes »…« was his):

Perhaps this thing on which I want to put the spotlight will be considered to be included in some ingenious way under the word »feedback.« But what I am going to stress stands in an obvious and superficial opposition to »feedback,« and it will, in certain frames of thought, be given nearly, if not quite so much, importance, and sometimes more importance than feedback itself in certain connections. It is certainly as circular. You have no doubt fed forward enough to see that what I am going to talk about from now on is feedforward. I am going to try to suggest its importance in describing how language works and, above all, in determining how languages may best be learned.

Feedforward, in Richards’ telling, has to do with arranging things in anticipation of the general shape in which one’s immediate future will unfold, so that appropriate feedback control mechanisms could be set in place, waiting to be actuated by the appropriate error signals if and when they come. This entails anticipating the effect, if not the exact realization, of disturbances before they happen. One of the examples Richards gave had to do with teaching children to read. He argued that good pedagogical practice would involve recognizing various heuristics and biases inherent in visual perception and preventing them from acting as disturbances to the process of learning in its initial stages. This would mean, for example, that the teacher should not introduce the letters “p”, “b”, “d”, and “q” at the same time since they are images of one another under rotations and reflections, and the visual system’s tendency to factor out such symmetries would cause unnecessary confusion:

The child couldn’t live life unless he saw a knife, say, as a knife, no matter which way up it was. It is bad technique to make a sudden transformation to script, in which it is all-important whether the u is upside down – or is it the n that is upside down? We penalize the bright child by setting a whole set of bogus traps for him in the script we begin to teach him. They don’t belong to the subject. They just betray him through his biological smartness.

The importance of invariance in pattern recognition has been pointed out in many places — e.g., in Wiener’s Cybernetics, in Pitts and McCulloch’s 1947 paper “How we know universals: the perception of auditory and visual forms,” and in Marvin Minsky’s 1961 paper “Steps toward artificial intelligence.” Biological visual pattern recognition systems have evolved to identify objects reliably despite uniform changes in size, position, and orientation. And yet, this obviously useful evolutionary adaptation acts as a disturbance when one is learning to read. Hence, Richards’ suggestion of staggering the introduction of such letters to counteract this disturbance effect can be seen as a feedforward control strategy that factors out invariances in low-level visual perception in order to induce higher-order invariances in letter and word recognition. (A curious side effect of this is the loss of robustness in such precisely crafted control systems: Most people would have a hard time trying to read the mirror image of a printed sentence.)

Richards recapitulates these ideas in a 1968 essay called “The secret of ‘feedforward,’” where he writes that feedforward

is the reciprocal, the necessary condition of what the cybernetics and automation people call “feedback.”

Whatever we may be doing, some sort of preparation for, some design arrangement for one sort of outcome rather than another is part of our activity. This may be conscious, as an expectancy—or unconscious, as a mere assumption. If we are walking downstairs, a readiness in the advanced leg (but indeed in our whole body) to meet something solid under its toe is needed if we are to continue. Usually, on the stairs, this feedforward is fulfilled. There is confirmatory feedback at the end of each step cycle—the foot finds the expected, the presupposed footing. Compare pitch-dark and broad daylight as to the degree of awareness we may have of our feedforward. If the feedback does not come, if it is falsifying and not verifying, we have to do something else and rather quickly. The point is that feedforward is a needed prescription or plan for a feedback, to which the actual feedback may or may not conform.

Evidently, feedforward is a product of former experience: a selective reflection of what has been relevant in similar activity in our past.

He goes on:

Feedforward is, as I see it, highly various. At one end of the scale, it can be a highly articulate examinable process, the sort of thing known as scientific hypothesis waiting to be okayed or destroyed by evidence. At the other, it may be hardly cognized or embodied at all, even in the vaguest schematic image. It can be no more than a readiness to be surprised or disturbed by one kind of event rather than by another: green-lighted or red-lighted by it. … [B]illions of hierarchically systematic cycles, through which we live and move and have our being, are guided in all they do for one another by concord or discord in their feedforward-feedback. And it is perhaps a reasonable suggestion that much in what we call ourselves and admit to be “us” includes these billions of concordant cycles.

This has a distinctly Hayekian ring to it—recall Hayek’s idea of the primacy of the abstract, namely that “the dispositions for a kind of action possessing certain properties comes first and the particular action is determined by the superimposition of many such dispositions.” Compare this with what Richards wrote in Poetries and Sciences (a 1970 re-issue with commentary of his 1926 book Science and Poetry):

We should picture the mind as a system of very delicately poised balances, a system which so long as we are in health is constantly growing. Every situation we come into disturbs some of these balances to some degree. The ways in which they swing back to a new equipoise are the impulses with which we respond to the situation. And the chief balances in the system are our chief interests.

Coda

We have now fed forward all the way to the end of the essay. In proper Soviet fashion, Shchipanov was “rehabilitated” in 1959, six years after his death at the age of 50 of a chronic respiratory illness. Researchers were now officially permitted to cite his work and to build on it. There were several interesting follow-up studies of his invariance principle at the Institute of Automation and Remote Control. For example, Maxim Braverman and Lev Rozonoer investigated structural stability of Shchipanov-type systems. A given property of a dynamical system is structurally stable if it is insensitive to small perturbations of system parameters. Rozonoer showed, in particular, that Shchipanov-type “invariant systems” are not structurally stable in the presence of small delays in the feedforward path from the disturbance to the control actuated by it. (Incidentally, Rozonoer was one of the authors of The Method of Potential Functions in the Theory of Machine Learning, a pioneering 1970 text that introduced kernel methods and advocated the use of stochastic gradient descent in machine learning. The other two authors were Emmanuil Braverman, Maxim Braverman’s father, and Mark Aizerman, one of the giants of Soviet control theory and a decorated World War II veteran who, as a young man, had enough courage and integrity to write a letter to Otto Schmidt, the Vice-President of the Academy of Sciences of the USSR, offering a rigorously reasoned yet impassioned defense of Shchipanov’s ideas.)

The importance of anticipation and the interplay between feedforward and feedback was also recognized early on by several Soviet psychologists and neurophysiologists, such as Nikolai Bernstein and Pyotr Anokhin. In the early 1950s their ideas were officially condemned as dangerous deviations from Pavlov’s thought (to be “rehabilitated” later, of course). Their work has been vindicated in modern neuroscience that seamlessly incorporates such control-theoretic concepts as the internal model principle, model predictive control, sliding mode control, and, indeed, Shchipanov-type disturbance-actuated control. In a way, Shchipanov in control and Bernstein and Anokhin in neurophysiology were arguing for the importance of small world models in a grand world, long before the term “world models” became a fashionable buzzword in AI. These researchers led their lives forwards, so that we can now use the benefit of feedback and appreciate their contributions from our vantage point.

Last night, Ben Recht challenged me to come up with a “grand synthesis” of what we have been hearing here so far. I don’t know whether this qualifies, but I think I can at least articulate some common themes and suggest a useful conceptual framework for this emerging field of “cultural AI” building on some ideas taken from cybernetics, structuralism, assemblage theory of Manuel DeLanda, and process philosophy of Alfred North Whitehead.

One of the goals of this meeting, namely to draw the outlines of an emerging field of “cultural AI,” runs up at once into the question of definitions and scope. What do we mean by culture, what do we mean by AI, and what do all the complementary viewpoints we have heard this week—literary theory, sociology, anthropology, computer science, machine learning, digital humanities, critical theory—say about the nexus of AI and culture? If we were to adopt the structuralist perspective to conceptualize AI as a cultural phenomenon,1 then it would make sense to project both culture and AI onto the three constitutive dimensions of structuralist view of systems, namely wholeness, transformation, and self-regulation.2 That is, we would view both “culture” and “AI” as closed, autonomous entities that are characterized by some set of invariants under a given collection of transformations. This decidedly cybernetic image is how, for example, structuralist linguistics has been viewing language or how structuralist anthropology has been viewing networks of social relations in human societies. What we would see from the outside is something like M.C. Escher’s “Drawing hands,” a pair of hands drawing each other into existence, co-creating each other. Second-order cybeneticists like Heinz von Foerster would invoke the image of the ouroboros and talk about “cognition computing its own cognitions,” “eigenbehaviors,” and so on:

As a professor of engineering who works on control and systems theory, I must admit that I like this metaphor a lot. It captures a great deal of how we think about feedback, regulation, control, things of this sort. However, I want to emphasize that, to an external observer, feedback control creates an illusion of closure and autonomy. It takes effort to create and maintain feedback systems, and we also need to be cognizant of historical and other factors that explain how these systems come and stay together and, just as importantly, how they eventually come apart. Both cybernetics and structuralism, by and large, elide these aspects of control systems (what James Beniger called “being” and “becoming”) and focus exclusively on what systems look like from the outside when they function as intended. If we adopt a systems view of culture and AI, we need to supplement the structuralist, closed-systems lens with the open-systems view nicely expressed by William James:3

Pluralistic empiricism knows that everything is in an environment, a surrounding world of other things, and that if you leave it to work there it will inevitably meet with friction and opposition from its neighbors. Its rivals and enemies will destroy it unless it can buy them off by compromising some part of its original pretensions.

William James often gets into trouble for these commercial metaphors (“buying off” here or “cash value” elsewhere), but they are valuable precisely because they highlight the role of friction, constraints, and trade-offs in complex systems. Put enough feedback control loops together, and they can create an illusion of an organism, a seamless whole composed of parts none of which can be conceived in isolation from others. Each component is, in fact, synonymous with, defined by, the totality of its relations to other components.4 Manuel DeLanda, to whose ideas I will come back in a moment, refers to these as relations of interiority.

This is one of the pitfalls of structuralism—it can tempt us into operating with reified generalities like “the society,” “the market,” or, closer to the theme of this meeting, “the culture” or “the artificial intelligence.”5 The key proposal I would like to put forward here, and thus to offer a synthesis of some of what we have heard earlier this week, is that we should adopt an alternative theoretical stance. If we want to understand complex systems under the rubric of cultural AI, then we should model them as assemblages, namely as wholes composed of relatively autonomous interacting parts characterized by relations of exteriority. These concepts, originating in the thought of Gilles Deleuze, have been developed into a comprehensive theoretical framework by Manuel DeLanda6 and, as it turns out, map pretty neatly onto how control and systems theorists reason about complex systems—in particular, how these systems are (or can be) constructed and how the patterns of interconnections between system components, both material and symbolic, give rise to the observed behavior of systems in interaction with their environments.

According to DeLanda, relations of exteriority

imply, first of all, that a component part of an assemblage may be detached from it and plugged into a different assemblage in which its interactions are different. In other words, the exteriority of relations implies a certain autonomy for the terms they relate … . Relations of exteriority also imply that the properties of the component parts can never explain the relations which constitute a whole, that is, ‘relations do not have as their causes the properties of the [component parts] between which they are established.’

On DeLanda’s account, autonomy refers to each part’s capacity to affect and to be affected by others. This is, obviously, context-dependent because the specific way in which the parts are interconnected and how they interact will select which of the capacities will be exercised and which ones will not be. Assemblage theory is the study of such wholes constituted by relations of exteriority and of the historical processes producing, stabilizing, and destabilizing them.

What exactly are the components that make up assemblages? Following DeLanda, we can speak of material and expressive components. Material components can be persons, equipment, data centers, physical infrastructure. Expressive components are datasets, code, model weights, rules, norms, laws, regulations, expectations of roles, organizational structures. This is not a fixed characteristic, but more of a context-dependent role, and each component can occupy a variable position on the axis between purely material and purely expressive (cash value, anyone?). The other dimension has to do with processes that either stabilize the assemblage or destabilize it; DeLanda, following Deleuze, calls these opposite tendencies territorialization and deterritorialization. These define the boundaries of an assemblage; they can make them sharper, bring the components together, or they can work in the opposite way.

Provisionally, then we can put forward the following points towards an “assemblage theory of AI systems:”

• AI systems must be understood through interaction between datasets and users

• datasets, users, interfaces etc. form assemblages

• interaction is mutual measurement: coding/decoding

• the ongoing process of measurement is a process of change: territorialization/deterritorialization

• who decides what to measure? what to do with those measurements?

The first two bullet points are self-explanatory, they simply establish the overall conceptual framing. The next point introduces the idea of measurement and of the related concepts of coding and decoding. I have discussed the measurement-centered view of machine learning elsewhere; here I want to frame it in the context of assemblage theory by appealing to a useful distinction Herbert Simon made between two types of descriptions of the world—namely, process vs. data:7

Pictures, blueprints, most diagrams, and chemical structural formulas are state descriptions. Recipes, differential equations, and equations for chemical reactions are process descriptions. The former characterize the world as sensed; they provide the criteria for identifying objects, often by modeling the objects themselves. The latter characterize the world as acted upon; they provide the means for producing or generating objects having the desired characteristics.

From this perspective, the basic epistemological claim underlying machine learning is that, to a large extent, it is possible to automate the extraction of process descriptions from data descriptions. In this sense, borrowing an apt formulation from Ben Recht, machine learning is engineered induction. Coding is the act of compressing data descriptions into process descriptions. However, in the spirit of Escher’s two hands drawing each other, there is a dialectic relating data and process, the flow from data to process is co-extensive with the flow from process to data. Decoding is the opposite act of generating data from process descriptions. This is, of course, the ethos of generative AI, but it is also the feedback loop that produces and reproduces culture. Novelty, creativity, spontaneous order can arise here because process descriptions encapsulated in AI models are by themselves inert, they need an external stimulus or prompt in order for generation to take place.

This is what we heard in Henry Farrell’s talk on the nexus of social, cultural, and bureaucratic technologies and from Cosma Shalizi on generative AI as mechanized tradition. Traditions are process descriptions of lore; when Cosma quotes Jacques Barzun about intelligence and intellect, he is also referring to the data-process dialectic. And this is where the question of values comes in (although, in a sense, it never really went anywhere—it was right there in William James’ quote about friction and opposition and buying off). As the components making up the cultural AI assemblages exercise their capacities to affect (or measure) and to be affected (or to be measured) by one another, we have to ask ourselves who decides what to measure and what to do with these measurements. These are the “hidden governance” aspects of AI that were highlighted by Abbie Jacobs in her talk, and they must be treated on the same footing as the questions of rationality, optimization, and other things that engineers care about.

And, indeed, humanistically minded engineers have been emphasizing these issues and raising related concerns all along.8 For example, this is how Sanjoy Mitter introduces the question of system effectiveness:9

System effectiveness is intimately tied to the issue of structure, the problem of measurement and the question of resources and values on the basis of which the system is evaluated for effectiveness … in a somewhat broad context where the systems can include both technological as well as social and economic systems. In order for systems to be effective, they have to be coherent … . The word coherence is being used here in the sense of Whitehead and is a concept which is broader than logical consistency. It requires viewing the system as a ‘whole’ which always has an environment and a value system (internal). Besides, the system residing in its environment is capable of observation by a multitude of external observers, each observer possessing its own value system.

This, once again, takes us back to William James’ pluralist empiricism, and it is very fitting to mention Whitehead here. His process ontology is in many ways similar to DeLanda’s assemblage theory because it also emphasizes exteriority of relations and the open systems view. Whitehead’s notion of coherence (which I have discussed elsewhere) is, as Steven Shaviro puts it10, “not logical, but ecological. It is exemplified by the way that a living organism requires an environment or milieu—which is itself composed, in large part, of other living organisms similarly requiring their own environments or milieus.” In addition to the dimensions ordinarily associated with instrumental rationality, it brings both ethics and aesthetics to bear on the problem of system design, instantiation, and maintenance. Moreover, emphasizing coherence and not just logical consistency prompts us to question various reified generalities that are constantly proffered by various actors as ultimately dispositive, such as the Silicon Valley framing of language as intelligence and of intelligence as a service. Shaviro puts it very nicely:11

We cannot live without abstractions; they alone make thought and action possible. We only get into trouble when we extend these abstractions beyond their limits . … This is what Whitehead calls ‘the fallacy of misplaced concreteness,’ and it’s one to which modern science and technology have been especially prone. But all our other abstractions—notably including the abstraction we call language—need to be approached in the same spirit of caution.

I would like to close by saying that I emphatically reject the original sin view of technology that pervades both the “AI ethics” and the “AI safety” camps. Ironically, it exposes their most dogmatic adherents as Heideggerian reactionaries who view technology as a revelation of an antihuman totalizing world order, which, depending on the camp you belong to, is either a colonialist profit maximizer or a superintelligent paperclip maximizer. Like Heidegger, they are obsessed with origins and are utterly uninterested in technology’s positive potential to realize what Whitehead called “creative advance into novelty.” As Shaviro argues, we would do better if we adopted Whitehead’s view instead:12

Whitehead’s reservations about science run entirely parallel to his reservations about language. (By rights, Heidegger ought to treat science and technology in the same way that he treats language: for language itself is a technology, and the essence of what is human involves technology in just the same way as it does language).

In summary, it seems to me that assemblage theory can offer a compelling conceptual framework for the emerging field of “cultural AI,” bringing together the complementary perspectives of technologically minded humanists and humanistically minded technologists. Thank you!

I won’t argue with Ben’s second point, as I largely agree with it. However, I do want to bring some nuance to the first point. Indeed, modern optimal control is a Cold War creation, and as such it proudly wears the mantle of instrumental rationality and ruthless cost-benefit accounting. Many of the key figures of optimal control were gainfully employed by the military-industrial complex — Richard Bellman, John M. Danskin, and Rufus Isaacs put in a lot of time at the RAND Corporation, Lev Semenovich Pontryagin was deeply embedded in the Soviet mathematical establishment and consulted for the military, and, of course, everyone knows about John von Neumann, whose early foundational work on game theory percolated into optimal control first through Isaacs’ work on differential games and, much later, through Tamer Bașar and Pierre Bernhard’s game-theoretic treatment of H∞-optimal control. And one common theme we can discern in all these works is that the model comes first.

From the vantage point of a historian, this makes sense. Modern optimal control grew out of the classical calculus of variations, which in turn grew out of classical mechanics. The excellent overview article by Héctor Sussmann and Jan Willems designates 1697 as the birth year of optimal control — when several solutions of the brachistochrone problem, posed about a year earlier by Johann Bernoulli as a challenge to the best mathematicians of the time, were published in Acta Eruditorum. This was the time of ascendant rationalism, which in the realm of natural philosophy put a lot of emphasis on mathematical models (given by systems of first-order differential equations relating positions, momenta, and forces) and on optimality (phrased in terms of least action principles and backed up by metaphysical commitments to pre-established harmony and the like). Subsequent developments by Euler, Lagrange, Hamilton, Jacobi, Weierstrass and others would later find new life in the mathematical theory of optimal control.

However, control theory had to first take an empiricist detour. While continental rationalists busied themselves with deciphering the book of nature written in the language of mathematics, industrial revolution was taking root in 18th-century England. When James Watt invented the centrifugal governor to stabilize the steam engine, he was not concerned with optimality, he wanted the shaft of the engine to rotate at a steady angular speed. Feedback control arose out of tinkering and hacking, models came later. Some of the first models and new mathematical concepts, such as stability, were put forward by James Clerk Maxwell. Maxwell’s 1868 treatise “On governors” worked out the criteria of stability for 2nd- and 3rd-order systems and posed the general case as an open question. It was solved independently by Routh in England (apocryphally, Maxwell’s academic rival during their Cambridge days) and by Hurwitz in Germany.

Until the 1960s, the mathematical toolbox of control engineering was imbued by robust empiricism. Systems were modeled in the frequency domain by transfer functions (rational functions of one complex variable), and, more often than not, exact models were not available. Instead, engineers relied on measurements and would fit various canonical models to them based on intuition and “clinical” experience. Once again, hacking and experimentation were the order of the day. The trusty three-term PID controller, invented in industry after a great deal of guesswork and hacking, owed more to Fordism and Taylorism than to model-based theorizing (see Stuart Bennett’s historical account). Various methods for tuning PID controllers based on empirical plant measurements were developed by industry insiders, such as the Ziegler-Nichols method which we don’t really teach to undergrads anymore. Tuning PID or lead-lag compensators based on frequency response measurements summarized in Bode plots is a subtle art, which in principle can be made entirely data-driven — just bust out the oscilloscope and go at it. Before control returned to models and rationalism in the 1950s and 1960s, it went through a data-driven empiricist phase spurred on by rapid industrialization and the rise of automation. We had to go through Bell Labs first before making it to RAND Corporation.

This empiricist view has much in common with modern RL. Policy optimization is closer in spirit to PID controller tuning even if there is an urge by both the theorists and the practitioners to phrase things in terms of approximate dynamic programming and Q-functions. Just like with PID controller tuning, we can only have assured performance locally; once we start nesting feedback loops inside other feedback loops, or add more stages to the dynamic programming recursion, things get out of hand rather quickly. And it’s even worse once uncertainty sets in. Quantitative Feedback Theory was developed by Isaac Horowitz as a valiant attempt to reinvent PID control for the age of uncertainty. QFT has its diehard adherents, but, again, nobody really teaches it to engineering students. However, at least here we have a clear recognition of the fact that optimality is a mirage. Instead, we set various performance specs and try to meet them as best as we can and hit the best trade-offs between them given available resources.

The LQR problem, as Ben points out, is one point of contact between old-school loopshaping craft and the modern optimal control framework. It is, however, an uneasy union of thoroughgoing empiricism and strict rationalism. The empiricist spirit is the heritage of the data-driven industrial control of the 1930s. The complementary rationalist methodology of 1950s-1960s optimal control is patterned on physics, which is not surprising given its roots in the calculus of variations. But physics cannot handle organized complexity well, and, anyway, what works isn’t optimal and what is optimal doesn’t work. But maybe we are not looking for optimality in the right place. Instead of trying to impose optimality on systems we are putting together, perhaps we could channel our creativity into optimizing our process — could we do better with fewer resources, for example?

Reading Morozov’s essay, I learned about Hans Otto Storm. A novelist and a radio engineer, Storm died in 1941, just days after the attack on Pearl Harbor, from an electrocution while installing a power transformer for the Army Signal Corps in San Francisco. He was friends with Thorstein Veblen, whose ideas on the role of “idle curiosity” in scientific discovery had influenced his thinking about the art and craft of engineering. Storm expounded some of his ideas in an essay titled “Eolithism and design,” which was published posthumously in 1953 in the Colorado Quarterly. In a way, this essay also gestures at something like poetic vs. bureaucratic technologies.

Storm wants to distinguish design, which he understands in the Weberian sense of rational action as matching available means to desired ends, from more organic, ecological modes of artifact-making, which he terms eolithic in reference to Stone Age eoliths, or “stones picked up and used by man, and even fashioned a little for his use.” The key difference between design and eolithism is this:

The important item of the definition from the point of view of method of craftsmanship, and the one which distinguishes the eolithic method fundamentally from that of design, is that the stones were picked up - picked up, that is to say, in a form already tolerably well adapted to the end in view and, more important, strongly suggestive of the end in view. We may imagine that person whom the anthropologists describe so formidably by the name of man strolling along in the stonefield, fed, contented, thinking preferably about nothing at all - for these are the conditions favorable to the art - when his eye lights by chance upon a stone just possibly suitable for a spearhead. That instant the project of the spear originates; the stone is picked up; the spear is, to use a modern term, in manufacture.

In other words, design is a rational activity; as Herbert Simon put it in The Sciences of the Artificial, “everyone designs who devises courses of action aimed at changing existing situations into preferred ones.” Here, the mention of preference is key, it is meant to indicate their temporal precedence to the process of design. A designer operates in what Jimmie Savage would call a small world, a setting where all possible states of affairs, contingencies, and consequences of various acts are codified and where a preference relation is given among the different consequences. It is a world where we can look before we leap, so an engineer can make use of simulation, prototyping, and numerical optimization to bring about the desired ends given the available means. By contrast, a large world is one where certain contingencies can only be encountered as a result of active exploration, rather than anticipated and accounted for in a theoretical model.

Storm, writing some thirty years before Simon, points out a similar distinction. Design operates in small worlds,

in those cases where the materials are uniform, and where there is considerable latitude for properly controlled experiment, so that the craftsman may have frequently impressed on his mind not only what happens according to the theory, but also what else may happen when the theory is ignored. Theory, experiment, and generalization become rather useless ornaments when applied to cases which do not in the nature of things repeat themselves, such as that of the biologist with twins, who had one twin baptized and saved the other as experimental control by which to estimate the effect.

By contrast, the eolithic mode is that of an encounter between (as Morozov nicely puts it) a Stone Age flâneur and an open world. There are no prefigured ends, values, or preferences; rather, they emerge spontaneously from this interaction, a particular affordance selected from among a multitude by the mind’s eye and a fortuitous coupling between an agent and an environment. There is either no theory guiding this interaction or, if there is one, it may be literally false. As Storm says,

I have known sailors who thought the tides came at the same hour every day, and mountaineers, outdoor people, who did not know that the fixed stars rose and set. Both of these specimens did from time to time order their movements by these wrongly apprehended natural phenomena, and both eventually muddled through - so that not only they but their theories with them survived into their ripe old age. … Their theories were not too wrong - after all, there were tides in the sea, and there were stars in the sky which did not move around very much. And the habit of being aware of these bulky natural verities brought in along with it an easy awareness of a multitude of other things individually quite below the threshold of classification, so that on the whole these people were workable, operating personalities who could get along in the world the way they found it, and whom one could depend on under the circumstances where one found them.

Storm’s concepts of eolithism and design could be mapped onto what James C. Scott called métis (local, practical knowledge acquired through experience) as distinguished from techne (rational knowledge put to work in the realm of uniform material and plans). This mapping is not entirely one-to-one because, unlike the eolithic craft, metis still presupposes goals that are stated beforehand. What is more interesting, though, is what eolithism and design have in common — the character of operational coherence.

In the context of scientific and engineering inquiry, this notion lies at the core of Hasok Chang’s latest book, Realism for Realistic People: A New Pragmatist Philosophy of Science. According to Chang,

operational coherence consists in aim-oriented coordination. A coherent activity is one that is well designed for the achievement of its aim, even though it cannot be expected to be successful in each and every instance. Operational coherence is based on pragmatic understanding; it consists in doing what makes sense to do in specific situations of purposive action

(emphasis in the original). Note that, even though Chang insists on talking about “design,” “aims,” and “purposes,” he does not stipulate their origin. Aims and purposes could be stated and codified ahead of time or they could be emerging spontaneously in the course of one’s interaction with the world. That makes both design and eolithism coherent. The stipulation of no guaranteed success is quite important here as well — think of Storm’s example of sailors or mountaineers achieving their aims despite holding theories that are literally false. Elsewhere in the book, Chang wants to emphasize that coherence is a much broader concept than logical consistency; it is context-dependent, open-ended, and provisional. Coherence can be gained and it can just as easily be lost.

Although Chang makes no mention of it, the notion of coherence is central to process ontologies that emphasize events rather than things, becoming rather than being. For example, Schopenhauer writes the following in the introduction to the first edition of The World as Will and Representation:

A system of thought must always have an architectonic connection or coherence, that is, a connection in which one part always supports the other, though the latter does not support the former, in which ultimately the foundation supports all the rest without being supported by it, and the apex is supported without supporting. On the other hand, a single thought, however comprehensive it may be, must preserve the most perfect unity. If it admits of being broken up into parts to facilitate its communication, the connection of these parts must yet be organic, i.e., it must be a connection in which every part supports the whole just as much as it is supported by it, a connection in which there is no first and no last, in which the whole thought gains distinctness through every part, and even the smallest part cannot be completely understood unless the whole has already been grasped.

Alfred North Whitehead’s Process and Reality defines it as follows:

’Coherence,’ as here employed, means that the fundamental ideas, in terms of which the scheme is developed, presuppose each other so that in isolation they are meaningless. This requirement does not mean that they are definable in terms of each other; it means that what is indefinable in one such notion cannot be abstracted from its relevance to the other notions. It is the ideal of speculative philosophy that its fundamental notions shall not seem capable of abstraction from each other. In other words, it is presupposed that no entity can be conceived in complete abstraction from the system of the universe, and that it is the business of speculative philosophy to exhibit this truth. This character is its coherence.

In the context of Storm’s distinction between design and eolithism, coherence can either be imposed and maintained in a top-down fashion or it can arise in a bottom-up way through a reciprocal interaction between a subject and an environment. Both bureaucratic and poetic technologies must be operationally coherent; we can apprehend technology as both the locus of instrumental rationality and as a crucial enabler of open-ended exploration driven by idle curiosity and play. We can either passively wallow in Heideggerian resignation to technology as enframing of the world as standing-reserve, or we can follow Whitehead and reconfigure our attitude toward engineering as “creative advance into novelty.” As he wrote in The Concept of Nature,

For natural philosophy everything perceived is in nature. We may not pick up and choose. For us the red glow of the sunset should be as much part of nature as are the molecules and electric waves by which men of science would explain the phenomenon.

This should be the motto of every poetic technologist.

In a way, Hayek’s ideas anticipated the so-called “system reply” to John Searle’s Chinese Room thought experiment: Even if we grant that the person confined to the Chinese Room lacks “genuine understanding” of Chinese, it would be much harder to argue this about the entire system comprising (1) the person, (2) the rulebook for mapping queries in Chinese to appropriate responses in Chinese, and (3) the environment that supplies those queries and evaluates the responses, insofar as it can maintain effective and coherent communication. Even if the system knowledge of Chinese (knowing-how, in the Rylean sense) could not be localized, it is there in a distributed form. Searle claimed that he could get around this by asking us to imagine that the person could memorize the rulebook, exit the room, and then just go on to simulate the system with its putative understanding of Chinese still without any trace of genuine understanding. (For example, he wouldn’t have a way of associating words in Chinese to objects in the world.) But this counter-argument falls apart: the Chinese Room with the person inside and the person who has memorized the rulebook are two different systems because their sensory interfaces are very different.

However, we should go a bit deeper into how Hayek thought about the fundamental role of rules and abstraction. While all of these ideas can be found in The Sensory Order, it is better to turn to Hayek’s paper entitled “The primacy of the abstract,” which he presented at the 1968 Alpbach Symposium organized by John Raymond Smythies and Arthur Koestler. The symposium, whose theme was Beyond Reductionism: New Perspective in the Life Sciences, brought together a diverse group of people — in addition to Smythies, Koestler, and Hayek, it featured talks by Paul Weiss, Ludwig von Bertalanffy, Jean Piaget, Jerome Bruner, C.H. Waddington, and even Viktor Frankl. The main thesis of Hayek’s paper is that

the primary characteristic of an organism is a capacity to govern its actions by rules which determine the properties of its particular movements; that in this sense its actions must be governed by abstract categories long before it experiences conscious mental processes, and that what we call mind is essentially a system of such rules conjointly determining particular actions. In the sphere of action what I have called “the primacy of the abstract” would then merely mean that the dispositions for a kind of action possessing certain properties comes first and the particular action is determined by the superimposition of many such dispositions.

This is a very sophisticated view, positing a layered cognitive architecture making use of abstraction and virtualization based on pattern classification. What Hayek calls abstractions are essentially labels for behavioral strategies (“dispositions”) that codify the organism’s tendencies for taking various actions that are determined partly by the organism and partly by the environment. These routines involve adaptive sensing, trial-and-error, internal simulation, and various feedback loops. Hayek gives the example of a lion about to attack its prey, where the particular movements of the lion will be determined by various external attributes (of the prey, the terrain, the ground, etc.) and internal variables (the lion’s proprioception, state of alertness, and physical fitness). So, in Hayek’s terminology, rules are abstractions — they abstract away the whole infrastructure supporting and facilitating a given kind of action in response to a given kind of stimulus. In other words, the mind projects itself onto the world, and the actual state of affairs is determined by the specifics of the coupling of the organism and the world. (In support of this view, Hayek cites Hermann Helmholtz, the Gestalt psychologists, J.J. Gibson, and Maurice Merleau-Ponty, among others.)

He goes on to say that

[w]hat this amounts to is that all the “knowledge” of the external world which such an organism possesses consists in the action patterns which the stimuli tend to evoke, or, with special reference to the human mind, that what we call knowledge is primarily a system of rules of action assisted and modified by rules indicating equivalences or differences or various combinations of stimuli. This, I believe, is the limited truth contained in behaviourism: that in the last resort all sensory experience, perceptions, images, concepts, etc., derive their particular qualitative properties from the rules of action which they put into operation, and that it is meaningless to speak of perceiving or thinking except as a function of an acting organism in which the differentiation of the stimuli manifests itself in the differences of the dispositions to act which they evoke.

Let us now get back to the Chinese Room. Hayek’s point about the primacy of the abstract obviously applies here: The person confined to the room does not receive an unstructured stream of sensory impressions. Being sequences of Chinese characters, they are highly structured. Moreover, the operator of the room can use all of his senses, memory, and other cognitive abilities. Thus, just as Hayek says, the abstract precedes the concrete in the sense that there is already a kernel of basic rules and sensory strategies standing ready to be deployed. The operator can recognize, classify, and act upon observed patterns in the external inputs, the instructions in the rulebook, and the external outputs prescribed by the rulebook. On the basis of this experience, he will hone his repertoire of different types of action, and (assuming sufficient variety in the queries coming from the outside) this repertoire will evolve and grow richer. With time, the inhabitant of the Chinese room will be able to make predictions about which sequences of characters are likely to follow certain other sequences, and will be able to anticipate, to plan, and thus to interact with the environment more effectively. This evidently constitutes a form of understanding, and can we even argue that this is not how we should think about understanding and knowledge in general?

Indeed, this is exactly what Hayek says:

the organism responds to—and thereby, as I like to call it, “classifies”—the various effects on it of events in the external world. This is the limited extent in which it can be said that these action patterns are built up by “experience”. It seems to me that the organism first develops new potentialities for actions and that only afterwards does experience select and confirm those which are useful as adaptations to typical characteristics of its environment. There will thus be gradually developed by natural selection a repertory of action types adapted to standard features of the environment. Organisms become capable of ever greater varieties of actions, and learn to select among them, as a result of some assisting the preservation of the individual or the species, while other possible actions come to be similarly inhibited or confined to some special constellations of external conditions.

Frank Rosenblatt, the inventor of the perceptron, acknowledged Hayek’s influence on his thinking.1 We also can recognize the Hayekian evolutionary metaphor in Gerald Edelman’s neural Darwinism. The closing paragraph of “The Primacy of the Abstract” will surely resonate with those of us who appreciate John Doyle’s thinking about layered architectures:

the processes I have been considering occur not just on two but on many superimposed layers, that therefore, for instance, I ought to have talked not only of changes in the dispositions to act, but also of changes in the dispositions to change dispositions, and so on. We need a conception of tiers of networks with the highest tier as complex as the lower ones. What I have called abstraction is after all nothing but such a mechanism which designates a large class of events from which particular events are then selected according as they belong also to various other “abstract” classes.

Do large language models possess knowledge and understanding in this Hayekian sense? They are obviously extremely complex “systems of … rules conjointly determining particular actions.” They are getting better at classifying and categorizing their inputs, and their context-conditional probabilities jointly determine the repertoire of outputs. Models like Claude Code produce programs, i.e., outputs that are themselves functions with some arguments left free for the environment to fill in. More sophisticated capabilities for perception and action are not far behind, although for that we need to finally go beyond souped-up chatbots. As for things like intelligence and consciousness, I’ll leave you with this quote from Hayek’s response to Viktor Frankl’s question about self-awareness and self-reflection in the discussion section:

If, as I believe it to be the case, the mind can be interpreted as a classifying machine, this would imply that the mind can never classify (and therefore never explain) another mind of the same degree of complexity.

everything is in an environment, a surrounding world of other things, and ... if you leave it to work there it will inevitably meet with friction and opposition from its neighbors.

Alfred North Whitehead’s monumental Process and Reality presents what is probably the most fully articulated process ontology that elaborates upon this open-systems view. In the new year, I will devote a few posts to exploring Whitehead’s thought in the context of technology. Here, though, I want to present a more focused take which, nevertheless, contains a kernel of process-oriented thinking.

I am a techno-pragmatist. I view technology as a means of problem-solving rather than of truth-seeking. This design viewpoint will naturally “meet with friction and opposition” at every layer, as it should be. Some of this friction and opposition has roots in material reality, some is grounded in societal and political domains. Nevertheless, instrumental rationality à la Max Weber is at the core of the technological worldview, and here we encounter a very important feature: In any sufficiently complex open system, implementing instrumental rationality at a particular layer of abstraction will inevitably expose (or even create) vulnerabilities in other interconnecting layers. Abstraction hides a great deal of complexity from view, and this is both its main virtue and its primary peril.

For me, the relevance of this to AI was evident even if I couldn’t articulate it precisely. However, things really clicked in my mind as I was carefully studying the work of Jean Carlson and John Doyle on highly optimized tolerance. Their first paper on this, published in 1999 in Physical Review E, was meant to provide an alternative to the physics-based “science of complexity” associated with places like The Santa Fe Institute. According to the Santa Fe worldview, features like self-organization, criticality, and universality (characterized by things like power law behaviors) were as applicable to the brain, the global economy, and other sociotechnical infrastructures as they were to frustrated spin glasses and other varieties of statistical physics models. Carlson and Doyle, however, argued very convincingly that the statistical physics viewpoint overlooks the multilayered, hierarchical nature of both engineered and evolved systems. Thus, they proposed the notion of Highly Optimized Tolerance (or HOT, for short) as an alternative explanation for the appearance of power laws in designed, rather than self-organizing, systems. Their main motivating example was the Internet:

The Internet is one example of a system which may superficially appear to be a candidate for the self-organizing theory of complexity, as power laws are ubiquitous in Internet statistics. It certainly appears as though new users, applications, workstations, PC’s, servers, routers, and whole subnetworks can be added and the entire system naturally self-organizes into a new, robust configuration. Furthermore, once on line, users act as individual agents, sending and receiving messages according to their needs. There is no centralized control, and individual computers both adapt their transmission rates to the current level of congestion, and recover from network failures, all without user intervention or even awareness. It is thus tempting to imagine that Internet traffic patterns can be viewed as an emergent phenomena from a collection of independent agents who adaptively self-organize into a complex state, balanced on the edge between order and chaos, with ubiquitous power laws as the classic hallmarks of criticality.

The core of the Internet, the Internet protocol IP, presents a carefully crafted illusion of a simple but possibly unreliable datagram delivery service to the layer above typically the transmission control protocol, or TCP by hiding an enormous amount of heterogeneity behind a simple, very well engineered abstraction. The TCP in turn creates a carefully crafted illusion to the applications and users of a reliable and homogeneous network. The internal details are highly structured and nongeneric, creating apparent simplicity, exactly the opposite from SOC and EOC. Furthermore, many power law statistics of the Internet are independent of density congestion level, which can vary enormously, suggesting that criticality may not be relevant.

The creation of the Internet is, indeed, one of the success stories of instrumental rationality. However, as Carlson and Doyle point out next, it comes at a price:

Interestingly and importantly, the increase in robustness, productivity, and throughput created by the enormous internal complexity of the Internet and other complex systems is accompanied by new hypersensitivities to perturbations the system was not designed to handle. Thus while the network is robust to even large variations in traffic, or loss of routers and lines, it has become extremely sensitive to bugs in network software, underscoring the importance of software reliability and justifying the attention given to it.

…

This ‘‘robust-yet-fragile’’ feature is characteristic of complex systems throughout engineering and biology.

Abstraction and virtualization are indispensable tools for enabling effective interaction. At the same time, they require systems to be open, to have the capacity to affect and to be affected by other systems. This is where we see emergence of unforeseen behaviors or new failure modes. Software that was used to design and implement some kind of a beneficial “user illusion,” such as various user interfaces, is an open attack surface for designing and implementing malicious user illusions (or even manipulating users and systems without any overt indication that this is happening). The possibility of this kind of “hijacking” is a universal trait of complex multilayered architectures, both engineered (think about computer viruses, cryptocurrency scams, DDoS attacks, etc.) and biological (think about actual viruses, parasites, cancers, autoimmune disorders, etc.). This is also a bug/feature in complex social systems like modern markets and democracies — highly beneficial abstractions like voting, finance, law, social networks are vulnerable to hijacking by virtue of their (relative) openness in the sense that they interact with other architectural layers in the overall system, and the constraints imposed on one layer will inherently deconstrain others.

In this sense, AI is, indeed, a normal technology. It aims to instrument certain kinds of “user illusions,” e.g., the illusion of communicating with intentional, anthropomorphic entities, and these systems are RLHF’d ad infinitum to force them to operate in a HOT state. However, this is exactly where we come face to face with the paradox of highly optimized tolerance1. Language is, itself, a multilayered architecture, where robustness at one layer can mask a great deal of of complexity at lower layers. Have you noticed the extra “of” in the preceding sentence? In case you haven’t, this is exactly what I am talking about: We are trading off the accuracy in the semantic layer against speed of processing at the syntactic layer, so small typos, repetitions and the like may not register at all if we pay attention to sentences and not to individual tokens like word parts or letters. We can go lower down the hierarchy of layers and talk about neuronal activity that takes place in our brains as we interact with the world. Much of it is un- or sub-conscious. That includes the tremendous web of associations and dynamical links on which conscious activity supervenes, and that’s where language offers both a powerful interface for abstraction and an attack surface vulnerable to hijacking. To my mind, this removes much of the mystery of Golden Gate Claude — and also shows what Sigmund Freud got largely right. The constraints of language deconstrain both the creation of new linguistic structures (and thus what can happen at layers above language) and the activity taking place in layers below language. This is the case both when humans interact with other humans and when they interact with AI systems. The only difference is that we somehow think that mechanistic interpretability is essentially distinct from the psychoanalytic hour just because we can mathematize the former but not the latter.

One of the last books I read this year was There Is No Antimemetics Division by qntm. The central conceit of the novel, that there is a secret organization devoted to protecting the world from antimemes (informational entities that can erase the traces of their having been experienced by human minds), could only have been born in our current cybernetic moment. We are surrounded by antimemes, such as the market, the TCP/IP protocol, social network engagement maximization, and now also AI assistants and companions. They increasingly demand our attention without drawing much attention to themselves, and it is important that we all remember: there really is no antimemetics division.

Happy New Year!

Fleuret’s take certainly applies to David Hilbert’s formalist view of mathematics as a game of symbolic manipulation according to fixed rules, without any attention paid to meaning. As we watch LLMs perform impressive feats of searching across vast corpora of mathematical facts, it is obvious that, indeed, these systems are poised to be much better players of the glass bead game compared to the best human mathematicians. As we already saw with AlphaZero and with AlphaFold, if a problem can be phrased as a game with fixed (if arbitrarily complicated) rules and if we have a way of evaluating various moves and of iteratively adjusting our strategies for picking a better move, then, indeed, it is only a matter of time before any such problem will, in Fleuret’s words, “fall.”

However, I think it is premature to say the same about metamathematics where, as Weyl says, “the game itself becomes the object of cognition.” This is the realm where we subject the games we play to a process of reflection, question the rules of these games, and invent new games. Intelligence, natural or artificial, is a joint property of the cognizing subject and the environment in which the subject is embedded, and this is where the tension between limitation and freedom comes into play. The key is to dissolve the limitations of closed formal systems by embedding them in larger open systems, where one has the freedom to introduce new axioms, new rules of inference, and new value systems. This act of reflective creation is what James P. Carse called an infinite game, a game without frontiers or fixed rules, a game which is not played with a fixed goal in mind, but with the motivation to keep playing. As such, it will increasingly involve both humans and AI systems in perpetual interaction.

Coming back to limitations versus freedom, when Weyl talks about it, he is referring to the tension between potentiality and actuality, becoming and being, data and process. That is,

the transition from the a posteriori description of the actually given to the a priori construction of the possible. The given is embedded in the ordered manifold of the possible, not on the basis of descriptive characteristics, but on the basis of certain mental or physical operations and reactions to be performed on it—as, for example, the process of counting.

The act of theoretical cognition is to transcend the limitations of the actually given by passing to “the field of possibilities that is open to infinity.” Here there are several options. One can follow L.E.J. Brouwer and other intuitionists and to view the field of possibilities as a potential infinity, something we can never fully access in its entirety, but which we can keep discovering iteratively using constructive procedures. Or one can go the route of Cantor and use the tools of set theory to visualize actual infinities. Weyl rejects Cantor’s platonic sensibilities in favor of moderate constructivism. Towards the end of The Open World he writes the following:

In the spiritual life of man two domains are clearly to be distinguished from one another: on one side the domain of creation (Gestaltung), of construction, to which the active artist, the scientist, the technician, the statesman devote themselves; on the other side the domain of reflection (Besinnung) which consummates itself in cognitions and which one may consider as the specific realm of the philosopher. The danger of constructive activity unguided by reflection is that it departs from meaning, goes astray, stagnates in mere routine; the danger of passive reflection is that it may lead to incomprehensible “talking about things” which paralyzes the creative power of man. … Hilbert’s mathematics as well as physics belongs in the domain of constructive action; metamathematics, however, with its cognition of consistency, belongs to reflection.

Moreover, he highlights the fundamental role of action in the progress of theoretical science:

[t]he task of science can surely not be performed through intuitive cognition alone, since the objective sphere with which it deals is by its very nature impervious to reason. But even in pure mathematics, or in pure logic, we cannot decide the validity of a formula by means of descriptive characteristics. We must resort to action: we start out from the axioms and apply the practical rules of conclusion in arbitrarily frequent repetition and combination. In this sense one can speak of an original darkness of reason: we do not have truth, we do not perceive it if we merely open our eyes wide, but truth must be attained by action.

When Weyl says that the objective sphere is “impervious to reason,” he means that you cannot understand the world by simply thinking about it or by computational simulation detached from experience. In other words, world models in Yann LeCun’s sense can only be arrived at through constructive action, predicting the next token is simply not enough. However, since world models are necessarily formal models, they are subject to the fundamental metamathematical limits of the kind described by Gödel and Turing — see, e.g., the recent work by Cubitt, Perez-Garcia, and Wolf on the undecidability of the spectral gap of certain types of quantum Hamiltonian models. Weyl can be forgiven for not having yet grasped the significance of the work of Gödel when he was giving the Terry Lectures in 1931. Our freedom to take constructive action runs up against the limitations on reflective reason; the limitations of reflective reason can in turn be overcome by further constructive action.

The world is infinitely complicated, yet the only way to investigate it is to ask questions. . . . The questions imply the answers and act as a “format” for converting simple structures into meaningful information. In this way, intention and meaning are imposed by the observer. To be a good observer requires being able to ask relevant questions of nature or, to put it differently, to have an interface that makes the world appear sufficiently simple to ensure survival. We need simple and effective geometries for interfaces, in contrast to the theories inevitably offered by physics.

This is a statement about the importance of measurement and, more immediately, about the importance of making the right measurements at the right level of complexity. Through the long and slow process of evolution, and then at a much faster pace after the emergence of culture, we have arrived at a wide variety of measurement strategies and interfaces:

• using our senses in feedback with locomotion and other outwardly projected behavior to query our environment and to build models of it in order to succeed in it (J.J. Gibson’s oft-quoted “we must perceive in order to move, but we must also move in order to perceive”);

• using language to probe our world, to organize our knowledge of it, and to develop shared intentionality through a system of evolving linguistic distinctions, concepts, speech acts, and webs of semantic relations connecting our senses, our descriptions of what the senses tell us, and our models of the world phrased in natural language;

• designing a wide variety of inquiring systems (or social/cultural technologies) for extracting, organizing, storing, and retrieving knowledge about the world that would not be easily accessible otherwise, distributed as it is in time and space and among multiple minds and different perspectives (see, e.g., Friedrich von Hayek’s view of spontaneous order in both markets and minds construed as information processors and pattern classifiers).

All of these systems and strategies, vastly different in scale and complexity, are observer interfaces in Koenderink’s sense. Each is equipped with its own “simple and effective geometries” that are, indeed, very different from the way physics apprehends the world. Extrapolating this framing to LLMs is very natural, as we already know from the writings of Henry Farrell, Alison Gopnik, Cosma Shalizi, and others. Thus, it is worthwhile to see how exactly LLMs fit the bill in light of Koenderink’s quote, whether they enhance our capabilities as observers, and whether they in turn can act as observers in feedback with us. This will be the theme of this post, and I will eventually come back from LLMs to markets, languages, and other distributed information interfaces.

The trouble with physics

Koenderink’s mention of physics is not incidental. John Archibald Wheeler’s “it from bit” is the motto of informational reconstruction of the physical world. We can read it in two ways. One is idealist: “in the beginning was the bit,” and everything else flows from information. This seems to have been Wheeler’s view. The other is more realist and points toward measurement as a mechanism of getting a purchase on the material world by collecting information about it, by interacting with it. As Koenderink says, we must be able to ask questions in order to investigate the world. Carl Friedrich von Weizsäcker, a renowned German physicist and philosopher, attempted to do precisely this through his theory of Ur-alternatives, binary measurements that yield only yes-or-no answers. With these building blocks, von Weizsäcker wanted to derive the structure of classical and quantum physics. Mathematicians also have a way of reconstructing various complicated objects (such as algebraic varieties, differentiable manifolds, or probability spaces) by starting with a sufficiently rich algebra of observables that can be used to peer at (measure!) the object of interest and then showing that there exists a one-to-one correspondence between the points making up this object and certain mathematical abstractions of mapping each feasible observable to a numeric outcome.

But this is not what Koenderink is talking about. At the end, when he contrasts “simple and effective geometries for interfaces” with “theories … offered by physics,” he is talking about what Wilfrid Sellars referred to as the manifest image of the world, the arena of ordinary everyday experience, in contrast to von Weizsäcker’s informational (re)construction of the scientific image using an elaborate game of twenty questions played against Nature. The interfaces Koenderink has in mind are our senses and the way our minds structure the experience derived from the senses. This is far removed from the pristine world of carefully controlled scientific experiments, where the links from perception to interpretable observations are extremely indirect and highly complex. Koenderink spent his life studying visual perception and that is what he mostly had in mind, but exactly the same ideas apply to all our other senses, including (as Berthoz would hasten to emphasize) interoception and proprioception. We couldn’t function in the world if we didn’t have a means for mapping regularities in our environment to structural information keyed to the pragmatics of living. Some of these strategies are slow, accurate, and expensive; others are fast, approximate, and frugal. Either way, it all comes down to measurement systems.

Machine learning as measurement

A recent paper by Fintan Mallory suggests to view LLMs as stochastic measuring devices. Fintan defines a measuring device as “an artefact that has been produced to alter its internal states in response to interactions with its environment in such a way that one can gain information about the environment from it,” and then he wants to argue that this is a novel and useful framing for LLMs. Along the way, he wants to draw a contrast between LLMs and traditional measuring devices (such as, say, thermometers). He writes the following about deep neural nets in general and about LLMs in particular:

Unlike traditional measuring devices, they have the memory resources to store models of the domain they measure and, in their generative capacity, are capable of running computational simulations of that domain.

I think, however, that the difference between LLMs and other measurement devices is not as significant as Mallory's paper suggests and that this framing applies more broadly to machine learning systems in general. However, viewing machine learning through the lens of measurement might not be very intuitive even to specialists, so we should be a bit more concrete about it.

Fintan’s definition of a measuring device agrees with other definitions of measurement systems one can find in the literature, such as in An Introduction to The Informational Theory of Measurement, a 1974 text by G. Kavalerov and S. Mandelstam, two Soviet researchers in the area of metrology, information processing, and control systems. Kavalerov and Mandelstam want to distinguish the carefully controlled world of high-precision physics experiments from the fast-paced world of everyday technology, where uncertainty is everywhere, time is of the essence, and a typical measurement system is a cascade of many functional units that convert either sensor data or outputs of other units to structured information. To them, all measuring devices are inherently stochastic. Moreover, the measurement systems Kavalerov and Mandelstam study also have memory because, in their theory, measurement is a dynamic process that generates and makes use of large datasets consisting of collected sensor measurements and their processed summaries.

In the context of machine learning, the process of collecting training data is, in fact, a process of measurement. In Kavalerov and Mandelstam’s classification, it involves primary perception or sensing of the quantities being measured, followed by selection of relevant attributes and features and by transformation of these into an appropriate data format. It is important to keep in mind, however, that there is a complicated cascade of transformations that takes us from measurements collected in the wild to a CSV file. Then the measured dataset is used to train a given machine learning system. The dataset can be viewed as the environment in which the training takes place, and the training process sets up an interaction between the environment and the machine learning system. In the course of training, the internal state of the machine learning system is altered in response to this interaction. The measurements acquired as training data are a finite sample from the domain of interest, and the adjustment of the model parameters during training can be seen as a process of transforming these measurements into information about the phenomenon being measured.

The internal organization of a trained machine learning model reflects the history of past measurements, i.e., the training data. When the model is used for inference or generation, it is also interacting with its environment, and the effect is also a change in its internal state that can be read off by the users of the model through an appropriate interface (e.g., text for LLMs). Again, taking a cue from Kavalerov and Mandelstam, it is entirely natural to interpret the overall transformation from inputs (e.g., prompts) to outputs as a cascade of measurement-like transformations starting with some primary perceptions and resulting in an information-bearing output signal.

Going back to Fintan Mallory’s paper, then, I would argue that the novelty of LLMs is not so much in their ability to store models of the domain being measured, but in what they are measuring, both during training and during generation or inference.

What are we measuring?

So, what do LLMs measure? As models of language, they measure informational regularities in language. Yet, keeping Koenderink’s quote in mind, we see that language itself is a stochastic measurement system coupled to the world through the members of linguistic communities. The evolution of language involves forming direct ties to the world by posing questions to it, followed by formation and stabilization of various concepts and distinctions with a rich network of internal relations between them. This is not a new point at all, going back at least to Zellig Harris’ informational theory of language. A given language, as a whole, is a dynamic system that operates in feedback with a community of speakers and comes to reflect their views, practices, and beliefs. Stable and frequent regularities in the speakers’ environment get coded as more compact linguistic utterances, sometimes sacrificing speed for accuracy, and there is also capacity for generating novel utterances, inventing new words, or completely changing the usage and meaning of existing words. This was also Shannon’s insight: language captures and stores the speakers’ implicit knowledge of the statistics of that language, and these reflect (if imperfectly) various regularities of the speakers’ shared environment. For our everyday needs as observers and agents, language is a good interface in Koenderink’s sense. With this understanding of language itself as a measurement system coupled to the world, we can now see what it is that LLMs measure: they measure the structures and patterns stored in (measured by!) language.

LLMs, by virtue of the mediated nature of their access to the world via language, can only measure certain kinds of information about the world, namely that encoded in the network of relations between concepts and other linguistic entities. They lack access to the direct sensory links between language and the world. Some researchers argue that language is not enough and that endowing learning systems with perception modules will be needed to resolve this problem.

Now, we come to the second aspect emphasized by Fintan Mallory, that of computational simulation. Simulation is used by scientists, engineers, and policy-makers as an investigative tool, so it is also a form of measurement whose goal is to obtain an answer to some question we wish to pose about the world. Indeed, we can easily come up with other examples of systems that both store models of their domain and can be used to simulate it. Markets are one obvious example and, in fact, this is exactly how Friedrich von Hayek conceptualized them. To Hayek, markets are world-coupled information processors endowed with memory and with capabilities for pattern classification and abstraction. This is the sense in which LLMs are similar to markets and other inquiring systems. And, just like we can view the training of LLMs as making measurements of measurements (the ones encoded in the snapshot of language at the time of training), the use of LLMs as generative systems is also tantamount to making measurements of measurements as the generative process unfolds. Here, though, the crucial difference is the presence of human users in the loop. Users pose questions in the form of prompts and use the LLM-generated responses to direct their activities. They have their own local knowledge which, in the best case scenario, they can combine advantageously with the massive trove of knowledge stored and structured in language models. And, just like with markets, blind trust in the system may prove detrimental; the end user of the simulation should have enough discernment to “trust but verify.” This is what being a good observer requires.

The first mention of the concept in a technical publication seems to be in a 1966 paper “Speculations Concerning the First Ultraintelligent Machine” by the mathematician Irving J. Good. Good is famous for the Good-Turing probability estimator that originated in the wartime efforts at Bletchley Park, as well as for having served as a technical consultant to Stanley Kubrick during the making of 2001: A Space Odyssey. The bulk of Good’s paper is fairly technical, devoted to mathematical exposition of ideas underlying probabilistic information retrieval, formation of associations based on experience building on ideas of Hebb, and the like. However, the opening sentence says:

The survival of man depends on the early construction of an ultraintelligent machine.

The beginning paragraph of Section 2 also cranks it up to eleven right away:

Let an ultraintelligent machine be defined as a machine that can far surpass all the intellectual activities of any man however clever. Since the design of machines is one of these intellectual activities, an ultraintelligent machine could design even better machines; there would then unquestionably be an “intelligence explosion,” and the intelligence of man would be left far behind … . Thus the first ultraintelligent machine is the last invention that man need ever make, provided that the machine is docile enough to tell us how to keep it under control. It is curious that this point is made so seldom outside of science fiction. It is sometimes worthwhile to take science fiction seriously.

The concept of “intelligence explosion” (which was articulated by Good in an earlier publication from 1962) is thus presented as a logical deduction from certain premises. In effect, Good presents a thought experiment, where we are to imagine a possible world which is just like ours in every respect except one: In this world, there exists “a machine that can far surpass all the intellectual activities of any man however clever.” In this world, then, an intelligence explosion is sure to take place simply as a consequence of the definition of ultraintelligence.

This thought experiment and its conclusion are the substantive core of all subsequent invocations of the idea. Its psychological appeal is undeniably powerful, but therein lies its potential weakness. To pinpoint this weakness, let us first inquire into the nature of this possible world and, in particular, whether Good’s definition of ultraintelligence does not do too much violence to all the usual ceteris paribus clauses one would have to invoke in order for the inference to the stated theory to be unproblematic. In other words, if we postulate this one small difference between Good’s world and ours, we better hope that it does not explode into a whole mess of differences that would make the background of the theory as strange and counterintuitive as the theory itself.

Small worlds vs. large worlds

The key distinction here is between small worlds and large worlds. In the context of statistical decision theory, this distinction was made by Leonard Jimmie Savage, one of the key thinkers associated with the subjective (or Bayesian) approach to statistics and also someone who had known Good rather well. In his 1951 book The Foundations of Statistics, Savage builds up the apparatus of Bayesian decision theory by first presenting an axiomatic framework based on two constructs: that of an act as a mapping from states of the world to consequences and that of a preference ordering among the various consequences. Starting from a list of axioms Savage defends as a reasonable foundation of decision theory from the personalist viewpoint (which he contrasts with the objective, or frequentist, viewpoint), he arrives at subjective probability and at the notion of expected utility maximization subject to the axioms of von Neumann and Morgenstern.

To all of this, Savage attaches a key caveat that it is only rational to apply all these constructs to “small worlds,” i.e., those in which we can always look before we leap. In other words, we know all the states of the world and can anticipate the consequences of all our acts. This is needed, for example, so that the notion of expected utility is well-defined. As Savage says,

though the “Look before you leap" principle is preposterous if carried to extremes, I would nonetheless argue that it is the proper subject of our further discussion because to cross one’s bridges when one comes to them means to attack relatively simple problems of decision by artificially confining attention to so small a world that the ‘‘Look before you leap" principle can be applied there. I am unable to formulate criteria for selecting these small worlds and indeed believe that their selection may be a matter of judgment and experience about which it impossible to enunciate complete and sharply defined general principles.

In other words, Savage argues that the concept of expected utility maximization only makes sense in a world where the decision-maker can take all conceivable information into account before choosing an act. All possible mistakes that could have been made will already have been corrected. There are “no alarms and no surprises” in small worlds. But, according to Ken Binmore, this qualification excludes the worlds of scientific inquiry, of microeconomics, and of high finance. As Binmore puts it in his 2009 book Rational Decisions,

in a large world, the possibility of an unpleasant surprise that reveals some consideration overlooked in Pandora’s1 original model can’t be discounted. As Savage puts it, in a large world, Pandora can only “cross certain bridges when they are reached.” Knee-jerk consistency is then no virtue. Someone who insists on acting consistently come what may is just someone who obstinately refuses to admit the possibility of error. In brief, Savage agrees with Ralph Waldo Emerson that foolish consistency is the hobgoblin of small minds. Only when our small minds are encased in a small world does he regard consistency as an unqualified virtue.

The world Good invites us to imagine is just such a large world because his definition tacitly rolls up scientific inference, macroeconomics, finance, and psychology into one tangled mess. For one, the definition of an ultraintelligent machine as one that surpasses humans in any intellectual activity is too open-ended. Is it possible to enumerate all conceivable human intellectual activities in advance? From history, we know that the introduction of new technologies, machines, social-economic arrangements, and political systems unlocks new intellectual activities and interests by amplifying human abilities. Asking us to imagine a machine that surpasses humans in any such activity tacitly assumes that we can forecast any such developments. So, the world that differs from ours “only” in one respect (ultraintelligent machines exist in that world but not in ours) is not a small world in Savage’s sense, it is a large world with huge indeterminacy in just about any realm we can think of: sociology, economics, technology, science, psychology. Before we can even begin to deduce intelligence explosion, we are confronted with an explosion of differences that makes Good’s world utterly incomparable to ours. It has too many bridges we can only cross when we get to them.

Establishing the phenomenon

Fundamentally, Good’s thought experiment is not a scientific one, it is psychological. Thought experiments of this type are not uncommon in analytic philosophy, and this is the domain where analytic philosophy can often become indistinguishable from science fiction. The 1980 book Real People: Personal Identity without Thought Experiments by the philosopher Kathleen Wilkes begins with a careful analysis of what it means to set up a successful thought experiment, i.e., the one in which “the jump from data to theory is relatively small.” As examples of such successful thought experiments in physics, she cites Simon Stevin’s proof of the law of equilibrium for a frictionless chain lying on an inclined plane or Einstein’s argument that classical electromagnetism is incompatible with the theory of relativity. What makes these thought experiments successful, in Wilkes’ description, is that they succeed in establishing the phenomenon in one’s imagination. On the other hand,

precisely because the relevant background is adequately fixed, so that the result of the imagined state of affairs is immediately clear. Were it not so, the inference could not be ‘unproblematic’. By contrast, when we have thought experiments in philosophy, there are as we shall see problems in making the inference—precisely because of the ambiguous uncertainty concerning the relevant background conditions, leaving it unclear whether we have indeed ‘established a phenomenon’. This means that our intuitions run awry, and the inferences are not only problematic, but the ‘jump’ from the phenomenon to the conclusion is made the larger because of the further need to imagine just what these backing conditions, under the imagined circumstances, would be. The ‘possible world’ is inadequately described.

In other words, successful thought experiments can be about small worlds only, and it is precisely the impossibility of adequately describing the background of the large world that prevents the offered thought experiment from successfully establishing the phenomenon. As an example of this, Wilkes takes aim at Derek Parfit’s famous thought experiments:

Consider next one of the familiar thought experiments to do with personal identity: that we might all split like amoebae. It is obviously and essentially relevant to the purposes of this thought experiment to know such things as: how often? Is it predictable? Or sometimes predictable and sometimes not, like dying? Can it be induced, or prevented? Just as obviously, the background society, against which we set the phenomenon, is now mysterious. Does it have such institutions as marriage? How would that work? Or universities? It would be difficult, to say the least, if universities doubled in size every few days, or weeks, or years. Are pregnant women debarred from splitting? The entire background here is incomprehensible. When we ask what we would say if this happened, who, now, are ‘we’?

Good’s, and everyone else’s, visions of superintelligence are equally incomprehensible for more or less the same reasons. They fail to successfully establish the phenomenon because the definitions they take as self-evident are anything but self-evident. One way in which this is evident is the unwarranted intermixing of mathematical or logical arguments with psychological notions, such as desire or drive.

Psychologism and its discontents

As I already noted above, Good’s definition of ultraintelligence is unworkable because it discusses human intellectual activities in a timeless setting of Savage’s smal world. It assumes that human psychology is fixed and that the range of human intellectual activities can be circumscribed in advance. From this, Good argues, ultraintelligent machines will be able to bootstrap their abilities in an intelligence explosion. This sort of “recursive self-improvement” is now one of the core concepts taken for granted in all the discussions on AI risk. However, a more explicitly psychological argument was put forward in 2008 by Stephen Omohundro in his paper “The Basic AI Drives.” Omohunrdo, a mathematician and a physicist by training, spent a few years as faculty in the department of computer science at my university and, among other things, was one of the authors (together with Stephen Wolfram) of a 1987 technical report envisioning a “tablet personal computer of the year 2000.”

Omohundro starts from the same premises as Good, but invokes the idea of dominated strategy in game theory to argue for the inevitability of recursive self-improvement by advanced AI systems. In a more technical paper from 2007, he argues that a superintelligent machine would converge to von Neumann-Morgenstern rationality by eliminating suboptimalities. In mathematics, this is a familiar argument, where one can arrive at a putative optimum by an iterative process that finds a local improvement from the current configuration. When no further improvement is possible, we arrive at a fixed point and then, if we wish to show global optimality, we have to argue that this fixed point is unique. This, for example, is the structure of the Dutch book theorems in subjective probability: Starting from the requirement of coherence of probability assessment (in the sense of de Finetti), we argue that any candidate assessment that fails to be coherent can be revised to a coherent one by means of conditionalization. To this mathematical argument, Omohundro attaches a psychological construct of drives any such superintelligent machine will inevitably possess: Efficiency, Self-Preservation, Acquisition, and Creativity. However, as argued eloquently by Václav Beneš (who, before going on to a distinguished scientific career as an applied mathematician at Bell Labs, wrote a dissertation on Quine at Princeton), mathematical theorems about the possibilties and limitations of complex systems (either formal systems of mathematical logic or adaptive systems like neural nets) are devoid of empirical content about human psychology precisely because they are mathematical statements. Aiming his critique at John Myhill’s psychological interpretation of Gödel’s and Church’s theorems, Beneš writes:

The psychological interpretation suggested by Mr. Myhill … is purely incidental in the sense that the same argument could be used to support a mechanical, cybernetic, biological, or even theological interpretation.

Omohundro’s concept of drives is vulnerable to the same critique: It either does not admit a psychological interpretation (in which case it can be formalized without appealing to vaguely Freudian terminology, but then it applies to any technological system) or it does (but then it is purely incidental to the mathematics, and there is no need to appeal to the trappings of mathematical rigor to discuss it).

Leaving aside the mathematical content of Omohundro’s argument2, the psychological appeal of these ideas is undeniable. But it is precisely the psychologism that undermines them because, once again, it forces us to give up a tidy small world where utility maximization makes sense in favor of a nebulous large world, where the relevant background is just as incomprehensible as the one in Parfit’s discussion about creatures that divide every autumn and fuse every spring.

The fables of subcreation

I am not arguing here against the need to envision possible catastrophic risks of advanced technologies and to use these analyses to influence policy. There is a need for such enlightened doomsaying. However, cloaking these discussions in mathematical arguments obscures their fundamental human-directedness. They are fables of what J.R.R. Tolkien termed “subcreation” in his essay “On Fairy-stories:”

Children are capable, of course, of literary belief, when the story-maker's art is good enough to produce it. That state of mind has been called “willing suspension of disbelief.” But this does not seem to me a good description of what happens. What really happens is that the story-maker proves a successful “sub-creator.” He makes a Secondary World which your mind can enter. Inside it, what he relates is “true”: it accords with the laws of that world. You therefore believe it, while you are, as it were, inside.

Such fables are simultaneously more powerful and more honest than thought experiments because, on the one hand, they are explicitly designed to provide a reflection of our psychological drives and constitution and, on the other, they are incredibly effective at establishing the phenomenon in Wilkesian sense. Rather than plowing through the X-risk literature, one can find much better discussion of the profound dangers of unfettered technocratic control in Tolkien’s Lord of the Rings or in C.S. Lewis’ That Hideous Strength. I would go so far as to say that the construction of fables and their hermeneutics is what makes continental, rather than analytic, philosophy the right framework for discussing the human-directed risks of technology, such as when Jean-Pierre Dupuy focuses on the antihumanist aspects of cybernetics.

John Langshaw Austin’s How to Do Things with Words, based on lectures he gave in 1955 and published as a book in 1962 (two years after his death from lung cancer at the age of 48), challenged the positivist idea that all meaningful linguistic utterances are descriptive and can be assigned a propositional truth value. During World War II, Austin was the leader of a massive intelligence group known as “The Martians,” which was responsible for aggregating, processing, and analyzing vast amounts of diverse intelligence data in preparation for D-Day. (Thomas Nagel’s excellent London Review of Books article on Austin has a detailed discussion of Austin’s wartime activities and how they have influenced his philosophical project.) As part of his “ordinary language philosophy,” Austin proposed a theory of speech acts centered around the notion of a performative utterance—that is, an utterance whose role is not to describe the world but to produce an action on or in it. For such performative utterances, the notion of truth does not make sense; what makes sense is whether they succeed or fail in a given situation. Accordingly, Austin classified such performative speech acts as felicitous or infelicitous. In a way, infelicitousness is the ordinary language philosophy counterpart of logical inconsistency of a formal system in positivist linguistic philosophy. Yet, it is a much richer concept than logical inconsistency. If you make a promise and then fail to make good on it, then the performative act of making that promise was infelicitous. If you make a joke and it doesn’t land, that’s infelicitous too. This is related to the concept of “open texture” of language, a term first used by Friedrich Waismann (a card-carrying member of the Vienna Circle) to describe the inherent vagueness or indeterminacy of empirical statements made in a natural language. For example, if I say that I am in a bad mood and then someone asks me to list all the reasons for it, I would not be able to come up with a complete, exhaustive list.

Austin’s philosophy of speech acts recognizes the prescriptive and regulative role of language. That is, it views language as a control technology. Language is a means for organizing and structuring action and interaction; as a control technology, it is also a complexity-reducing device.1 However, the open texture of language distinguishes it from control as programming, where one could (in principle) formally verify the correctness of a given program and, at the very least, expose any logical inconsistencies. Hilary Putnam’s Representation and Reality took the concept of open texture further, arguing that it presents a fundamental challenge to purely computationalist approaches to truth, intentionality, and semantics.

Now, while Austin and the members of his Oxford circle were influential in the realm of Anglo-American analytic philosophy at least for a while2, it was never their intention to endow the idea of “doing things with words” with a computational character. Natural language was messy, embedded, and embodied; computation was clean, abstract, and disembodied. Thinking of felicitous or infelicitous speech acts in quantifiable terms would have never occurred to them, you just knew it when you saw it. Paradoxes of self-reference and quandaries of undecidability were bad enough for computational theories of cognition. How could you possibly compute with words?

As it happens, computing with words was exactly what Lotfi Zadeh wanted to do from the moment he introduced the idea of a linguistic variable and the theory of fuzzy sets in the 1960s. Zadeh was born in Baku, Azerbaijan, in 1921, moved to Iran at age 10, graduated from University of Tehran with a degree in Electrical Engineering in 1942, and was a faculty first at Columbia University and eventually at Berkeley after getting his MS degree at MIT in 1946 and PhD at Columbia in 1949. Zadeh’s profile in IEEE Spectrum mentions that, “as a child, [he] was surrounded by governesses and tutors, while as a young adult, he had a personal servant.” He was a pioneer in systems and control3 who invested a great deal of effort into promoting the conceptual unity of electrical engineering and computer science. One of his motivations for developing fuzzy logic was the disconnect he perceived between the high effectiveness of mathematical system theory in dealing with “mechanistic systems” and its low effectiveness in dealing with “humanistic systems.” In a 1961 paper called “From circuit theory to system theory,” he wrote:

There is a fairly wide gap between what might be regarded as ‘animate’ system theorists and ‘inanimate’ system theorists at the present time, and it is not at all certain that this gap will be narrowed, much less closed, in the near future. There are some who feel that this gap reflects the fundamental inadequacy of conventional mathematics—the mathematics of precisely-defined points, functions, sets, probability measures, etc—for coping with the analysis of biological systems, and that to deal effectively with such systems, which are generally orders of magnitude more complex than man-made systems, we need a radically different kind of mathematics, the mathematics of fuzzy or cloudy quantities which are not describable in terms of probability distributions. Indeed, the need for such mathematics is becoming increasingly apparent even in the realm of inanimate systems, for in most practical cases the a priori data as well as the criteria by which the performance of a man-made system are judged are far from being precisely specified or having accurately-known probability distributions.

Although the machinery of fuzzy set theory and fuzzy logic was framed in mathematical terms and could be conceived as just another nonstandard logic, it had a polarizing effect that was amplified by the culture wars and the science wars of the time. Rudolf Kalman, another pioneer of systems and control and also someone who was not exactly known for a diplomatic disposition, wrote the following in 1972:

I would like to comment briefly on Professor Zadeh’s presentation. His proposals could be severely, ferociously, even brutally criticized from a technical point of view. This would be out of place here. But a blunt question remains: Is Professor Zadeh presenting important ideas or is he indulging in wishful thinking? No doubt Professor Zadeh’s enthusiasm for fuzziness has been reinforced by the prevailing climate in the U.S.—one of unprecedented permissiveness. ‘Fuzzification’ is a kind of scientific permissiveness; it tends to result in socially appealing slogans unaccompanied by the discipline of hard scientific work and patient observation.

The mathematician William Kahan was similarly unsparing:

‘Fuzzy theory is wrong, wrong, and pernicious.’ says William Kahan, a professor of computer sciences and mathematics at Cal whose Evans Hall office is a few doors from Zadeh’s. ‘I cannot think of any problem that could not be solved better by ordinary logic.’ What Zadeh is saying is the same sort of things ‘Technology got us into this mess and now it can’t get us out.’ Well, technology did not get us into this mess. Greed and weakness and ambivalence got us into this mess. What we need is more logical thinking, not less. The danger of fuzzy theory is that it will encourage the sort of imprecise thinking that has brought us so much trouble.’

It is not my intention here to debate the merits of fuzzy logic. The interesting part is the genealogy of ideas that eventually led Zadeh to proclaim in a 1996 paper that “fuzzy logic = computing with words” and then, in another paper published in 1999, to argue that, in contrast to computing with numbers that represent measurements, computing with words (or, more broadly, with propositions in natural language) is about manipulation of perceptions. Unlike idealized numerical measurements, perceptions are often vague, flexible, fallible, context-dependent—in other words, they perfectly exemplify the open texture of experience even when (or especially when) described in natural language. In that same paper, Zadeh says:

We cannot build robots which can move with the agility of animals or humans; we cannot automate driving in heavy traffic; we cannot translate from one language to another at the level of a human interpreter; we cannot create programs which can summarize nontrivial stories; our ability to model the behavior of economic systems leaves much to be desired; and we cannot build machines that can compete with children in the performance of a wide variety of physical and cognitive tasks.

What is the explanation for the disparity between the successes and failures? What can be done to advance the frontiers of science and technology beyond where they are today, especially in the realms of machine intelligence and automation of decision processes? In my view, the failures are conspicuous in those areas in which the objects of manipulation are, in the main, perceptions rather than measurements. Thus, what we need are ways of dealing with perceptions, in addition to the many tools which we have for dealing with measurements. In essence, it is this need that motivated the development of the methodology of computing with words (CW)—a methodology in which words play the role of labels of perceptions.

The list of failures, as Zadeh presented them in 1999, is a mixture of things we still cannot do well (that includes robots, large-scale economic modeling, or artificial machines that can perform on par with children in certain physical or cognitive tasks) and of things that today’s large language models can perform with ease (translation from one natural language to another or summarizing a story). On the one hand, it would certainly be apt to describe what LLMs are doing as a species of computing with words. On the other hand, the range of capabilities posited by Zadeh as the goal of computing with words is much wider than what LLMs can do. What can account for this?

I would like to offer an explanation which is partly historical and partly conceptual. While Zadeh’s fuzzy decision theory originally drew sharp criticism in the West4, it caught on in Japan and in the Soviet Union. In Japan, it was mostly adopted in consumer electronics and home appliance industry5; in the USSR, though, it fell on the fertile soil of a uniquely Soviet approach to cybernetics. Starting in the 1960s, there was a great deal of dialogue between Soviet cyberneticists (who, unlike their Western counterparts, readily embraced both the analog and the digital methods) and psychologists, who were interested in the art and science of human problem solving. This interest brought together the mathematician Dmitrii Pospelov and the psychologist Veniamin Pushkin, who were convinced that Herbert Simon and Alan Newell’s formalization of complex problem-solving as pattern recognition and local search (what Pospelov and Pushkin termed the “maze approach”) did not account for human creativity and insight the way their own “model approach” did. According to Pospelov and Pushkin, what Simon and Newell described was only the final stage of a more complex process involving the construction of an appropriate linguistic description of the problem that would allow the problem-solving agent to reason coherently from the starting conditions to the desired goals. In other words, before you can even begin making local moves in the maze, you have to first build the maze and figure out the rules of the game, and that’s where creativity and insight would come in. They presented these ideas in their 1972 book Reasoning and Automata, where they made the connection to the problem of control of large-scale complex systems.

The first thing to note was that the notion of large-scale complex system was ill-defined. Crediting Mikhail Bongard, another prominent Soviet cyberneticist, Pospelov and Pushkin argued that we should apply this label to those systems that are not amenable to strict formalization. In fact, such formalization may not even be desirable in the first place. In his 1986 monograph Situational Control: Theory and Practice, Pospelov listed the salient features of such nontraditional (or ill-defined) control systems:

1. Uniqueness.

2. Lack of a formalizable purpose.

3. Lack of optimality.

4. Dynamic or evolutionary nature.

5. Incompleteness of description.

6. Free will.

Taken together, all of these factors call for a new approach to control, so-called situational control. This terminology reflects the notion of situation as a summary of all relevant information pertaining to the current configuration,structure, and function of the control system. The process of controlling such a system amounts to selecting one control action from a finite repertoire of available actions that would cause a change from one situation to another. In a certain sense, the notion of situation is related to the notion of state: Both are meant to capture the current information available to the decision-maker for the purposes of prediction and control, but the key assumption underlying the construction of the state space is that of completeness, namely, that all relevant system attributes are amenable to observation, categorization, and abstraction. Unlike the state, the concept of situation reflects all the above peculiarities of working with large-scale complex systems and is inevitably influenced by subjective inputs, biases, and preferences of system operators and users.

This brought to the foreground the role of natural language as the only available means of describing and controlling such systems. They recognized that natural language could be the future of computer programming decades before Andrej Karpathy’s viral tweet about English being “the hottest new programming language.” From the very beginning, situational control theory recognized the open texture of language as both a challenge and an opportunity for designing radically new approaches to control. The approach they took was similar in spirit to Zadeh’s construction of fuzzy logics that starts with standard propositional logic and then “fuzzifies” it by relaxing the Boolean set membership criterion to a membership function taking values in the interval [0,1]. In the same vein, situational control theorists took as their starting point the standard notion of a formal system from logic. A formal system is composed of axioms, transformation rules, and a semantic interpretation. As such, it is a closed system—all semantically valid statements can be derived from the axioms by mechanical application of the transformation rules. Situational control theorists proposed to “open up” formal systems by making all their ingredients amenable to modification based on external inputs (e.g., by hooking them up to a learning system). This, they argued, transformed the elements making up the formal system into signs as they are studied in semiotics—that is, entities that can stand in relation to themselves, external objects, and external subjects via their respective syntax, semantics, and pragmatics imposed in a given context by convention. So, the term “semiotic control” and “semiotic system modeling” was also applied.

The function of natural language in semiotic control theory was to provide the control engineer with a means for expressing concepts, relations, and functional roles that are relevant for the control problem at hand. So, just like Zadeh’s fuzzy logic that started with the recognition of inherent vagueness of natural language for describing perception but ended up with yet another formal system, semiotic control theorists developed a formal Situational Control Language that bore very little resemblance to the open texture of natural language. They were remarkably open-minded and broad in their adoption of various formalisms—in addition to Zadeh’s fuzzy logic, Pospelov’s book discusses variants of temporal logic, causal logic, pseudo-physical logic, John Stuart Mill’s inductive logic, and Indian Navya-Nyāya logic. It has philosophical sections on concept formation, Piagetian psychology, and the fundamental incompleteness and absurdity of knowledge.6 Ideas from cybernetics and pattern recognition, such as the perceptron or clustering, appear as well. The historical overview at the end of the book traces the development of ideas that led to situational control theory and lists some practical successes in Soviet industry (shipping port logistics, control systems in aviation, oil refinery management, medical diagnostics, etc.). Nevertheless, just like fuzzy logic, situational control is now mainly a historical curiosity because, just like fuzzy logic, it did not exactly give up the idea of fully formalizable system design. For all their talk of ill-defined boundaries and vagueness of natural language, both fuzzy logic and situational control envisioned systems as symbolic and propositional all the way through. Thus, they ended up in the same blind alley as GOFAI, eventually outstripped by connectionist methods like deep neural nets and, now, LLMs and other Transformer-type architectures.

If we now return to Zadeh’s list of canonical tasks that rely on computing with words, we can see why some of the things he lists are now easily within the capabilities of state-of-the-art LLMs (such as machine translation or text summarization), while some are still beyond their reach. As I wrote in one of my posts about Daoism and AI, replacing fully propositional symbolic systems with massive connectionist architectures that rely on words (or tokens) as an information interface with their environment had momentous consequences. Some tasks can indeed be handled by systems that take tokens as inputs, generate tokens as outputs, but look nothing like “theorem provers” internally. Lotfi Zadeh’s insistence on referring to perceptions as amenable to “fuzzy” tokenization was directionally correct, but limiting tokens to words in a natural language was not. In the abstract of his 1999 paper, he wrote that

computing with words (CW) is inspired by the remarkable human capability to perform a wide variety of physical and mental tasks without any measurements and any computations. Familiar examples of such tasks are parking a car, driving in heavy traffic, playing golf, riding a bicycle, understanding speech, and summarizing a story.

However, most of the above examples are not instances of computing with words. Rather, they are examples of computing with natural signs, some of which may be words in a natural language, but many are not. Moreover, they do not admit an easily quantifiable metric of success like perplexity or cross-entropy or expected reward. The considerations of open texture and felicitous acts loom large, as they do in any context involving biological organisms or any other embedded and embodied system. Biological evolution has endowed us with incredible capacity for doing things with natural signs. Language is just a small part of it, and there is now a great deal of work using Transformer-like architectures on suitably tokenized nonlinguistic data (vision, motion, etc.). In fact, according to the “throwing hypothesis” put forward by the neuroscientist William Calvin, our ability to “compute” with language could have emerged by recruiting existing neuronal architectures that had evolved for better control of motion sequences, including timing, planning, and the use of forward predictive models with token-like information interfaces. If that’s the case, then “computing with words” is just a special case of “computing with signs,” the architectures for which were already present in hominid brains.

Apart from the book’s contribution to our understanding of Hellenistic philosophy of mind (Julia Annas’ book on the philosophy of mind of the Stoics and the Epicureans should be mentioned here as well), it is also valuable as a conceptual framework for thinking about artificial intelligence (or, more narrowly, about the interplay between perception, pattern recognition, learning, and knowledge). Viewed from this angle, Aristotle’s thought looks remarkably modern and can offer some new insights for both analysis and synthesis of AI systems.

Generative and architectural thinkers

It will be useful to frame the discussion that follows in terms of the dichotomy between generative vs. architectural style of reasoning, proposed in the context of AI by Philip Agre in Computation and Human Experience. According to Agre,

generative theories … involve mathematical operations capable of generating an infinite number of mental structures by the repeated application of a small number of basic rules

while

architectural reasoning attempts not merely to implement a given abstraction but to discover abstractions that both do the required work and admit of natural implementations.

Herbert Simon, Alan Newell, and Noam Chomsky are generative thinkers, whereas Marvin Minsky (when he wrote The Society of Mind) is an architectural thinker. The core distinction between generativists and architecturalists hinges on the role of abstraction:

The generative and architectural styles of research have led to different views about the relationship between abstraction and implementation in human cognition. Generative theories posit large, consistent abstractions that operate by their own formally specifiable laws. McCarthy and the other proponents of the reconstruction of human knowledge in formal logic, for example, take as their starting point the generative power of logical formalisms and the formal consistency presupposed by their semantics.

…

Architectural theories, by contrast, hold that the demands of physical implementation have profound consequences for the formulation of abstract theories of cognition. Minsky in particular emphasizes physical locality so strongly that cognition becomes a large, fragmentary collection of mutually inconsistent abstractions, each bearing a different relationship to its physical implementation. He formulates his theory not as a single unified mechanism, or even as a single list of axioms of design, but as a constellation of mini-theories, each examining some feature of human intelligence - whether language, reasoning, decision-making, memory, emotion, imagination, or learning - on the level of engineering intuition.

The same tension can be found in the twentieth-century philosophy of science, where logical empiricists (a.k.a. logical positivists) are firmly in the generativist camp, whereas the proponents of the so-called New Mechanist Philosophy are architecturalists. Viewed through historical lens, this distinction makes perfect sense: The positivists of the Vienna circle, whose influence on later logical empiricists like Hempel was profound, took physics as their model science; by contrast, the New Mechanists (Bechtel, Craver, Darden) drew their inspiration primarily from the life sciences, specifically molecular biology and neuroscience. Newtonian physics, as interpreted by logical empiricists, is a paradigmatic generative framework: Everything in the Universe can be explained through consistent application of the laws of physics to the given initial condition. By contrast, biology and neuroscience are more concerned with how living organisms and living brains produce their activity. William Wimsatt is another example of architectural thinking in philosophy of science. I will return to the New Mechanists later, after introducing Modrak’s synthesis of Aristotle’s perception-based framework. For now, let’s tentatively agree to place Aristotle in the architectural camp.

Aristotle’s foundational principles

Modrak begins by stating, in modern terms, what she believes are five foundational principles underlying Aristotle’s theory of perception. These principles, comprising three descriptive principles and two prescriptive ones, “inform Aristotle’s treatment of the perceptual faculty and account for its unity and coherence.” Here they are, in Modrak’s words:

A. Descriptive Principles

1. Psychophysical Principle. Many states, if not all, that are ordinarily assigned to the soul are psychophysical states, namely, psychical states with physical realizations.

2. Actuality Principle. A cognitive faculty is potentially what its object is actually.

3. Sensory Representation Principle. If a cognitive activity has a sense object as its focal object, the psychic faculty involved is a perceptual faculty.

B. Prescriptive Principles

1. Analytic Principle. A psychological explanation should begin with an account of the constituent parts of the phenomenon under consideration and then make this account the basis for extending the explanation to cover more complex phenomena of the same sort.

2. Normative Psychophysical Principle. Psychological explanation at its most complete will take the psychophysical character of psychological states into account.

The key notion associated with Aristotle is hylomorphism, or the organizational unity of matter (hyle) and form (morphe). The above foundational principles instantiate this doctrine in the context of perception, referring to Aristotle’s idea that the soul (psyche) gives form to (or enforms) the body, as well as to the interplay between formal and material causes. Aristotle’s project is ambitious: Starting with the five senses, he wants to develop a cohesive account of perception, including apperception, or the ability to synthesize complex judgments based on multiple senses; imagination or phantasia, an umbrella term under which he includes sensory illusions, memory, dreaming, voluntary action, and even some forms of discursive reasoning; and, finally, the rational faculty, or the noetikon, which he attributes exclusively to humans. This emphasis on not just what the perceptual activity is, but also on how it is produced and what role different senses and faculties play in its production, is what makes Aristotle an architectural thinker, rather than a generativist like his mentor Plato.

On Modrak’s account, the Psychophysical Principle tells us that psychophysical states have both a functional description as mental states and a physical realization as states of the central nervous system. However, Aristotle’s functional description is closer to that of the biologists’ rather than that of the cognitive scientists’ since his psychophysical account does not decouple the functional view from the physical realization view. This is consistent with the hylomorphism doctrine and is very different from the multiple realizability ideas in functionalist accounts of cognition. The Actuality Principle and the Sensory Representation Principle together constitute a form of realism about sensory representations, but Aristotle views these representations as icons in the semiotic sense, i.e., as signs that stand for objects through relation of similarity.

The two prescriptive principles are explicitly architectural in nature. The Analytic Principle asks for explanations in terms of mechanisms that are constructed from basic constituent parts to form more complicated arrangements (e.g, when Aristotle begins with the analysis of the five basic senses and then proceeds to build up an explanatory framework for more complex perceptual phenomena). However, the psychophysical nature of the mental states has to be emphasized throughout, as demanded by the Normative Psychophysical Principle, so the resulting account does not amount to reducing mental states to nothing but neurophysiology (as eliminativist accounts along the lines of Paul and Patricia Churchland would do).

Multimodal perception and koine aisthesis

Aristotle’s account of the senses centers on the idea of logos, or rational mean. Sensory representations are given relative to opposing pairs (e.g., quiet vs. loud, dark vs. bright, bitter vs. sweet). The processing of composite percepts, when multiple sensory modalities provide information about a given object, is the function of what Aristotle called koine aisthesis, or the common sense. In accordance with the Analytic Principle, Aristotle aims for economy of explanation and avoids introducing another faculty where a perceptual faculty would do the work. Moreover, in line with his empiricist epistemology (e.g., he argues that knowledge is impossible without perception), learning plays a role in forming apperceptual judgments or in using the input of one sense to infer the input of another. For example, a stable association of “white” and “sweet” could arise in certain contexts — if I see a white object presented on a dessert plate and if I had seen and eaten similar-looking desserts in the past, then I may reliably infer that this object will be sweet to the taste without having tasted it.

In some way, this view of multiple sensory modalities acting in concert and reinforcing each other, with the aid of experience and learning, is reminiscent of Gerald Edelman’s ideas on neuronal group selection — assemblies of neurons that strongly respond to certain sensory modalities will be differentially reinforced as the organism acquires more experience; moreover, simultaneous presentation of multimodal sensory data about a given object will lead to reinforced connections between the neural assemblies corresponding to these different modalities (what Edelman refers to as re-entrant connections). Similar discussion of coherence among the senses and the unity of perception can also be found in the writings of Alain Berthoz, who argues for the importance of anticipation and prediction in addition to integrating the current sensory data into a whole, as well as for the (still not well-understood) role of simultaneous activity in many different regions of the brain and of temporal synchronization. Emery Brown’s description of general anesthesia in terms of synchronization and desynchronization of neural oscillations among different subsystems in the brain seems relevant here as well.

Imagination, or phantasia

Aristotle wants to explain how perceptual activity is produced and how it contributes to the functioning of the organism in its environment. As he writes in De Anima,

the soul of animals has been defined by two faculties, the faculty of discrimination (kritikon), which is the function of thought and perception, and the faculty that originates movement with respect to place.

Being an empiricist, Aristotle acknowledges that not every instance when these faculties are exercised involves perception of an external object — for example, when we anticipate or plan or dream. Guided by the Analytic Principle, Aristotle still associates perceptual objects with these types of activities, the phantasmata that are generated by the imaginative faculty or phantasia. As I pointed out earlier, Aristotle makes phantasia do a lot of work. To assimilate imagination into Aristotle’s general framework, Modrak formulates the following important idea:

Aristotle includes in the full formal description of a sensory experience the conditions under which the experience occurs. Perception occurs under standard conditions. Phantasia occurs under nonstandard conditions, that is, under conditions that are not conducive to veridical perception.

This covers sensory illusions, which arise when coherence among multiple senses breaks down; dreams; mental simulation and planning before taking an action (e.g., deciding where and how to move to avoid a predator); and certain forms of discursive reasoning that are nonpropositional in nature. All of these, according to Aristotle, involve manipulation of phantasmata, and it is not hard to see echoes of this in modern neurophysiological theories (e.g.., the role of efference copy in planning and coordinating movement) or in Daniel Dennett’s discussion of Skinnerian, Popperian, and Gregorian creatures which differ in their capabilities to perform internal simulations, to form hypotheses and conjectures, and to recruit tools in their external environment for the purpose of planning and acting. Memory is also an instance of phantasia, where we recall that a particular sensory impression occurred in the past.1 In line with the architectural style of reasoning, we can conceptualize both perception under standard conditions and the exercise of phantasia as production, manipulation, and exchange of icons (signs standing for objects through relation of similarity) and indices (signs standing for objects by being their causal effects) among different modules and architectural layers in the brain or in the AI system, as done, for example, by Manuel DeLanda.

Thought, or the noetic faculty

In Aristotelian epistemology, both the external objects of perception and their psychophysical correlates are particulars. Each of them is tied to a concrete instantiation either as part of objective external reality (Aristotle was a realist about objects of perception under standard conditions) or as a concrete psychophysical state. The internal sensory representations Aristotle has in mind are nonsymbolic, and both humans and nonhuman animals have the ability to manipulate them as they interact with their environment. However, humans, according to Aristotle, have the unique ability for rational thought, which involves manipulation of universals by the noetic faculty. Modrak explicitly says that “the mode of representation employed by the noetic faculty is properly described as symbolic.” However, these symbolic representations cannot exist by themselves, but require nonsymbolic (specifically, phantasmatic) vehicles. For example, when a geometer contemplates a problem involving triangles, the underlying thought process makes use of images of triangles as well as a considerable degree of abstraction:

The geometer ignores particularizing features such as size in order to treat the drawing as an arbitrarily selected instance of its class. Similarly, an image when used to facilitate reasoning about a type is treated as a representation of a token of the type under consideration, and the thinker ignores the features that are peculiar to it. The symbolic employment of a phantasma is already a distinct mode of representation. Moreover, there is some reason to believe that Aristotle envisages a kind of internal language as the vehicle for noetic representation. According to the De Interpretatione, ‘Spoken words are symbols of affections in the soul and written words are symbols of spoken words’’. The attribution of a quasi-linguistic character to noetic representation would also explain the persistent association of noetic capacities with ratiocination.

Aristotle’s noetikon is a manipulator of sentences in “mentalese,” which crucially relies on the perceptual faculty; this is exactly what Dennett had in mind when he spoke of theorem provers wrapped in an overcoat of transducers and effectors in The Intentional Stance. The universals figure prominently in Plato’s thought as well; however, what distinguishes Aristotle from Plato is that, for him, the universals (or syntactic objects, as we would say now) are formed in two ways: By induction from the particulars, which requires perception, or by valid logical inference, which requires ratiocination. Aristotle uses a curious military metaphor to describe the formation of universals out of a corpus of particular perceptions in The Posterior Analytics:

[O]ut of perception memory, as we say, comes to be and out of memory occurring frequently with respect to the same thing experience comes to be. For memories which are many by number make up one experience. And out of experience or out of the universal resting as a whole in the soul, the one beside the many, which is one and the same in all of these, comes to be the first principle of art and science— of art, if it concerns coming to be, of knowledge if it concerns being.

Neither are these present in us as definite dispositions nor do they come about from other dispositions which are more capable of knowing but from perception. Just as in a battle, when a rout occurs, if one takes a stand, and another takes a stand, then another, until the original formation is achieved. And the soul is such as to be capable of being affected in this way.

The first paragraph is almost Humean in spirit, arguing for the pragmatic inference of necessary connection from constant conjunction. The second paragraph, with the invocation of a rout in a battle, almost gestures at the Hebbian mechanism for learning. The overall picture that emerges is reminiscent of Aaron Sloman’s Virtual Machine Functionalism, where the notion of “virtual” refers not to virtual reality, but to virtualization, or the formation of manipulable objects and mechanisms on higher layers of a complex architectures while respecting the constraints imposed by lower layers (constraints that deconstrain). In particular, we can view the formation of symbolic representations (or universals) as virtualization built on iconic-indexical substrates, which in turn can be instantiated as an integrated activity of a number of perceptual mechanisms. Yet another piece of evidence in favor of viewing Aristotle as an architectural thinker!

New Mechanist Philosophy and the importance of architecture

I will close by mentioning a close link between Modrak’s modern take on Aristotelian hylomorphism and the New Mechanist Philosophy. A good source for this is Daniel De Haan’s “Hylomorphism and the New Mechanist Philosophy in Biology, Neuroscience, and Psychology.” The proponents of New Mechanism are interested in showing how a given phenomenon is produced by its causes. New Mechanistic explanations are not reductive in the usual sense of “X is nothing but Y;” instead, they posit the mosaic view of science (or what Sunny Auyang in her book on complex system theories called “federal unity of science” and contrasted it with the “imperial view” of the reductionists). As Carl Craver writes in his 2007 book Explaining the Brain: Mechanisms and the Mosaic Unity of Neuroscience,

the mosaic view treats the unity of science as the collaborative accumulation of constraints at multiple levels. Whereas reduction focuses on relations of identity, supervenience, and ontological reductive links, the mechanistic mosaic view emphasizes the importance of explanatory relevance as the bridge between levels. Finally, whereas reduction models emphasize the importance of explanatory reduction to fundamental levels, the mosaic view can be pluralistic about levels, recognizing the genuine importance of higher-level causes and explanations. The mosaic unity of science is constructed during the process of collaboration by different fields in the search for multilevel mechanisms.

This is the essence of the architectural style of reasoning about complex systems, although here I agree with John Doyle about the importance of not confusing levels with layers. Levels pertain to a hierarchy arranged by scale, whereas layers describe functional organization. William Wimsatt uses the term “perspectives” to refer to layers in this sense. A given layer can be described on multiple levels (e.g., we could look at a particular region in the brain and look at its organization as an assembly of neurons, then look at the scale of individual neurons; etc.). Architectural explanations move across levels and layers. In this sense, Aristotle’s integrated view of perception, action, thought, and knowledge is a superb example of an architectural theory, and may yet show its usefulness in the context of AI as some researchers are starting to contemplate moving away from training on language and towards the new era of experience.

more than that of his colleagues in the natural sciences, the position of an ‘expert’ in the study of human behavior resembles that of a sorcerer who can make the crops come up or the rain fall by uttering an incantation. And because the facts with which he deals are seldom verifiable, his customers are able to demand to be told what they like to hear, and will punish the uncooperative soothsayer who insists on saying what they would rather not know — as the princes used to punish the court physicians for failing to cure them.

Without getting into the merits of Andreski’s polemic, his general diagnosis was directonally correct. The phenomena he had identified, captured by pithy chapter titles like “Manipulation through description,” “The smoke screen of jargon,” “Hiding behind methodology,” “Quantification as camouflage,” and “Ideology underneath terminology,” are all very real and all present to this day. However, the powerful image of what was supposed to be empirical inquiry turning into sorcery applies today, more or less in the way Andreski had phrased it, to the field of artificial intelligence, but with a few twists and ironies.

The subjects of Andreski’s criticism were all highbrow academics, their larger cultural influence as public intellectuals powered by the cachet of scholarly tomes and serious publications in prestigious journals. The flow of ideas was from academia to the society at large. With AI, however, it is academic researchers who have succumbed to magical thinking and ritualistic incantations originating largely in industry and fed by the credulous coverage in popular press. Every phenomenon identified by Andreski is on display here, including the neverending stream of academic publications where researchers interact with massive models like ChatGPT or Claude and deploy loaded terminology to promote the view that the only way to predict or to control the functioning of such systems is by an intricate system of charms, spells, and incantations. When Herbert Simon wrote in The Sciences of the Artificial about approaching the scientific inquiry about computers as a subfield of natural history, where we would “study them as we would rabbits or chipmunks and discover how they behave under different patterns of environmental stimulation,” he was probably imagining something closer to ethology in the spirit of Konrad Lorenz and Nikolaas Tinbergen, not papers about sparks of AGI or the evil vector. Alas, the technocratic impulse is wedded to crude animism.

The appeal to AI as some abstract intelligence you can tap into for any purpose, without considering the appropriateness of it as a technology in each given setting from a pragmatist point of view, is driven largely by ideology. This should give us some pause. As Andreski wrote in the chapter aptly titled “Techno-totemism and creeping crypto-totalitarianism,”

to come back to pseudo-cybernetics: its veiled promiscuously conservative ideological message has endeared it to the bosses throughout the world (no matter whether capitalist, communist, clericalist, militarist, racialist or what not) and enabled its devotees to obtain control over funds which, of course, brought them applause from the academic multitudes. Profiting from the awe which any mathematical-sounding terms inspire among the non-numerate practitioners of the social sciences, as well as from the mathematically competent scientists’ naivety about social and political problems, the pushers of pseudo-cybernetics have been able to achieve fame as experts on politics without ever having said anything relevant.

Sounds familiar?

Three weeks ago I received the sad and shocking news that my PhD advisor, Horace P. Yuen, passed away at the age of 79. Everyone always says on such occasions how much they had been influenced by their advisor. I won’t be an exception to this. Horace’s influence on me has indeed been truly profound and extended from the usual academic matters to philosophical reflection and a general attitude toward life and human affairs.

My first encounter with Horace was in 1998, before I even started entertaining any thoughts of pursuing a graduate degree. I was an Electrical Engineering junior at Northwestern University. At that time I was mainly interested in optoelectronics and fiber optics and was eyeing an engineering career in industry, at a place like Lucent or Corning. There was the usual chatter among our cohort about which classes to take and with whom, and, when it came to the second required course in electromagnetics, the overall sentiment was something like, “don’t take it with Yuen, he’s an old school guy with a PhD from MIT and he expects you to know all this math.” Some part of me must have reacted like, “oh yeah? I think I could totally do it,” so I registered for his class. That was the turning point.

He would stroll into the lecture room, carrying a manila folder with two or three handwritten sheets of notes. He barely looked at them when he lectured. He wrote on the board in perfect longhand and would often add personal anecdotes about science and scientists or philosophical digressions about the difference between physics and engineering. One of his frequent sayings was, “mathematicians care about rigor, engineers care about precision.” What others had been saying about his demands on our mathematical knowledge was not entirely accurate—while he certainly emphasized mathematical sophistication, to him it was not about having lots of tricks at your disposal, but more about your ability to drill down to the conceptual core of things, even when thinking about something as basic as the definition of angle. Compared to your typical class where the emphasis was about grinding out problem sets, this was a refreshing experience.

My interest in photonics led me to quantum electronics and quantum optics, and Horace was the one to talk to about these things. He was legendary for his 1976 work on squeezed states of light, and also because he drove an opalescent maroon Jaguar with the license plate that said “CHRONON” and was seriously into competitive ballroom dancing. In my senior year, I signed up for his graduate course on information theory and was a regular in his office, just coming in whenever I had some half-baked thought or a question about some reading I was doing at the time. His door was always open, and he would never turn anyone away. That was a double-edged sword though, because he was also very direct and did not mince words. At some point, when I came in with another one of my half-baked ideas, he looked at me and said, “Max, I don’t think you are capable of abstract thought.” Did that discourage me? Not at all. He must have sensed that because the next week he handed me a copy of Michael Artin’s Algebra and said, “I got this for you. I learned abstract algebra from Artin when I was an undergrad at MIT, this is the way to learn it.”

Horace’s information theory course was challenging but incredibly rewarding not just because of the original way he presented the material (his introduction of Shannon entropy as “asymptotic combinatorial complexity” will forever be etched in my memory), but also because of all the stories he would tell. He was first a student and then a researcher at MIT when Shannon was on the faculty there, and his insights into the history and the conceptual development of information theory were remarkably deep. He gave us a lot of additional reading, including Shannon’s original papers, Jim Massey’s “Information theory, the Copernican system of communication,” and David Slepian’s “On bandwidth.” Most importantly, he would continually emphasize the importance of not losing sight of practical engineering considerations when working with mathematical models.

When I eventually became his PhD student, I had complete freedom to work on whatever I wanted. Horace was always there to suggest relevant references. He had an incredible memory and a quick way of seeing straight to the heart of the matter. At one point, when I became interested in reliable computation with unreliable circuit elements after having read von Neumann’s famous work on this problem, I asked Horace whether it made any sense to talk about something like “computation capacity” in analogy to Shannon’s channel capacity. He answered right away, “sure, take a look at Winograd and Cowan’s Reliable Computation in the Presence of Noise.” He had zero tolerance for hype and self-promotion. Every time I would come into his office and say something like, “have you seen this recent paper by so-and-so? they did such-and-such,” he would often reply, “well, they claim that they did it, is it really true?” This skeptical attitude is one of the things I learned from him. And, of course, he remained true to his nature. When one of his frequent collaborators was visiting from Italy, the three of us were in Horace’s office, and, in the middle of a heated discussion about some derivation on the board, Horace exclaimed to him: “Come on, how is it that you don’t understand this? Even Max understands it!” And yet, despite all this, he has been unwavering in his support and encouragement.

We remained in touch after I defended my PhD in 2002, and even later when I switched fields from quantum information to machine learning and control. He would take me out to dinner either at one of his favorite Cantonese restaurants in the Chicago area or at one of the upscale establishments in Evanston, and we’d have long conversations about whatever was on our minds. He was always open to learning new things, and he brought his skeptical attitude, his incredibly broad knowledge of a vast array of subjects, and his remarkable memory to everything. I don’t remember exactly how the subject of philosophy came up, but at one point I was telling him about my reading of MacIntyre’s After Virtue. Little did I know that, among many other things, he was deeply read in both continental and analytic philosophy, going back to the time when he and his ex-wife Sunny Auyang would attend philosophy seminars at MIT and study Heidegger and Husserl together. In that discussion we had about MacIntyre, he brought up Nozick’s Anarchy, State, and Utopia and Rawls’ Theory of Justice and had interesting critical takes on both and on their relation to MacIntyre’s views.

He retired in 2021, shortly after the COVID pandemic, and we kept having long discussions over email. He wanted to understand recent developments in machine learning and AI and how they relate to ideas in philosophy of mind. As usual, he was incredibly quick in the way he understood things and got straight to the core; his replies to me were full of references to Rudolf Carnap, Daniel Dennett, and Jaegwon Kim. He was unconvinced by Kim’s physicalism, but found his books to be the best defense of the physicalist view. Just as in the old days, he believed that electrical engineers, and not computer scientists or physicists, were best equipped to make sense of AI; he was unimpressed by Geoff Hinton’s alarmism but found Yann LeCun’s views congenial to his own perspective. In many of our exchanges, he stressed the importance of approaching things from what he called his “Buddhist/Daoist/scientific perspective.” He argued that a comprehensive view of the world and of our place in it requires all three: Buddhism in order to understand our inner mental experience as directly given, Daoism in order to grasp how our inner experience affects and is affected by the web of causal relations in the world, and science in order to maintain coherence between the Buddhist/Daoist manifest image and the modern scientific image.

In some our last exchanges, we were getting into philosophical issues of Buddhism and Daoism, and I wanted to get Horace’s perspective as I was making my way through Chad Hansen’s book. We went deeply into the “Happiness of fish” passage from The Zhuangzi, and my amateur interest was obviously no match for Horace’s decades-long study of both classical and modern Chinese philosophy. He had many, many things to say, with references to Schopenhauer (whom he considered one of the deepest interpreters of Buddhism) and to the philosophy of Xiong Shili, of whom I had never heard before. I wish I did not delay responding to his latest email, which turned out to be his last to me. Here is what he wrote in one of our exchanges:

The 'realism' description of humans, together with the added human complexities on top, seems primarily a result of evolution. This is a scientific worldview that I have to reconcile with Buddhism and Daoism and whatever basic thoughts I may have. Maestripieri quotes Darwin, "He who understands baboon would do more towards metaphysics than Locke". The challenge to my perspective as a whole is clear.

It is empty to claim, as Buddhists will in response, that human reality is 'not real'. Why would they even bother to respond then? They are concerned with something, including the 'unreal' realities, with all their life instructions and prohibitions. Although I can argue further for them, they are not authentic nihilists. Daoists, especially Zhuangzi, would dismiss all these from a transcendental disposition which is seemingly not nihilistic, but logically appears to be just an 'active' flip side of nirvana.

Farewell, Horace. I hope you are now happy and carefree, like those minnows in the River Hao.

One of the signatures of uncontrolled complexity is instability. When control engineers talk about instability, they refer to systems that can exhibit violent divergence from intended behavior and, once they enter this dangerous zone, are incredibly resistant to corrective control actions. Properly designed controls keep systems in their stable regions, but the sheer scale of possible dangers calls for respect and humility on the part of the engineer. Unstable systems are everywhere—in markets, energy, public health, biotechnology, financial industry, transportation, political decision-making, and more. The behavior of individuals and institutions involved in the operation and use of these systems is one of the sources of instability, but it can also contribute to reliable functioning when interconnected with properly designed control policies. These include not only explicit rules and regulations, but also norms, conventions, and culture. Technology can (and should) play a role as well. However, it is precisely because these control mechanisms keep instabilities in check, there is a risk of complacency on the part of individual actors, which can (and, as we are witnessing in real time, does) lead to crises. Here are some salient examples.

Air traffic safety

After the horrific plane crash in DC, there was a renewed conversation about air traffic safety in general. So-called “near misses” seem to occur more often than previously thought, yet somehow, because most of these events were not catastrophic, there was no widespread realization that various layers of control that keep air travel safe, from technological solutions to human operators, are being stretched thin. Increasing numbers of close calls should serve as error signals that alert the users and the operators to potential for catastrophic failure. Instead, in practice they seem to make people complacent. They assume that, because nothing terrible has happened so far, nothing terrible can happen in the future.

Vaccination

Vaccination is one of our most successful control technologies, explicitly designed for systems with potentially highly destructive unstable modes (epidemics and pandemics). Its efficacy derives both from feedback control that keeps the populations of pathogens in check and from network effects on the level of human populations that result in herd immunity. It certainly feels like highly infectious diseases, such as measles, mumps, chicken pox, or polio, are basically a thing of the past. The resulting reduction of complexity overlays on the general lack of awareness of the feedback loops needed to keep these diseases in check. The interplay of these two effects leads to complacency on the one hand and to vaccine skepticism on the other. The growing numbers of people choosing not to vaccinate their children even against measles and chicken pox threaten to upend the carefully maintained stable equilibrium state precisely because this stable equilibrium state has such low externally perceived complexity.

Public administration and rationality

As we speak, Elon Musk’s DOGE operatives are bulldozing straight past the Chesterton’s fence that surrounds the maze of technocratic solutions and kludges responsible for the day-to-day functioning (such as it is) of the US government. When Jürgen Habermas wrote about the legitimation crisis in 1973, he was talking about the public’s diminished confidence in the ability of the modern administrative state to resolve or mitigate conflicts of values or worldviews that inevitably arise as an emergent property of market liberalism. Habermas viewed this as a crisis of rationality, which he understood specifically in communicative terms and for which the vibrant public sphere is a prerequisite. What we are witnessing is also a crisis of rationality, albeit of a different nature. As Bent Flyvbjerg writes in “Rationality and power,”

the fact that the power of rationality emerges mostly in the absence of confrontation and naked power makes rationality appear as a relatively fragile phenomenon; the power of rationality is weak. If we want the power of reasoned argument to increase in the local, national, or international community, then rationality must be secured. Achieving this increase involves long term strategies and tactics which would constrict the space for the exercise of naked power and Realpolitik in social and political affairs. Rationality, knowledge, and truth are closely associated. “The problem of truth,” says Foucault, is “the most general of political problems.” The task of speaking the truth is “endless,” according to Foucault, who adds that “no power can avoid the obligation to respect this task in all its complexity, unless it imposes silence and servitude.” Herein lies the power of rationality.

That first sentence, “the power of rationality emerges mostly in the absence of confrontation and naked power makes rationality appear as a relatively fragile phenomenon,” is a crisp statement of the phenomenon noted by Åström in the context of the public sphere and its relation to power.1 Rationality, understood in Weberian terms as matching of available means to desired ends, is a feedback control mechanism, but, to borrow an apt term from Jean Carlson and John Doyle, it is robust yet fragile. It is robust insofar as a bunch of programs written in different dialects of COBOL, some from many decades ago, interface with each other relatively smoothly for the US treasury payment systems to keep chugging along; yet it is fragile when it faces the maw of DOGE’s wood chipper.

The Åström critique

Actors in socioeconomic systems are capable of reflecting on being controlled and acting as a result of this reflection. This can manifest itself in different ways. For example, the Lucas critique in macroeconomics states that the decision rules used by individual economic agents can (and will) change in response to changes in macroeconomic policies, thus potentially invalidating the models that had been informing policy design. Lucas was urging macroeconomists to pay attention to microfoundations, i.e., to the behaviors and interactions of individual agents comprising the economy. Michael Arbib, Oliver Selfridge, and Edwina Rissland put it nicely in “A dialogue on ill-defined control”:

the task of the economic decision-maker is not to control a physical system that knows nothing, but to control a system that is in turn modelling the controllers! And, of course, the economists are part of the economy that is to be controlled. The world, which is the ill-defined system that we are trying to control, is a world full of people. And they are each trying to control certain aspects of a world full of people.

A colorful (and currently relevant) example of the Lucas critique in action is about whether the government should keep investing in expensive protective measures for various facilities that had never been compromised in the past. However, if this investment is eliminated, then the facts on the ground will change, and a security violation may occur thus invalidating the predictive model. The Åström critique, if I may call it that, points to a related but different phenomenon. Applied to sociotechnical contexts, it describes the state of affairs where the agents in a system are so inured to the complexity-reducing function of control that they become completely oblivious both to the dangers of uncontrolled complexity and to the ever-present need for control mechanisms to keep this complexity at bay. Like the Lucas critique, the Åström critique is also a story about microfoundations, but here the individuals or institutions alter their behavior not because of a perceived change in policy, but because of a perceived absence of complexity. Because the control mechanisms have been working as intended, the participants in the system may begin acting as if the controls were not there at all. If this keeps going, we may one day forget what these controls were in the first place and how to implement them. This is our Weberian moment, rationality must be secured.

Those who know others are perceptive
those who know themselves are wise
those who conquer others are forceful
those who conquer themselves are strong
those who know contentment are wealthy
those who strive hard are resolved
those who don’t lose their place endure
those who aren’t affected by death live long

— Dao De Jing, chapter 33 (translation by Red Pine)

“Perception is external knowledge. Wisdom is internal knowledge. Force is external control. Strength is internal control. Perception and force mislead us. Wisdom and strength are true. They are the doors to the Tao.”

— Li Hsi-Chai’s commentary on Dao De Jing 33

As I wrote in Parts I and II, the central thesis of Chad Hansen’s A Daoist Theory of Chinese Thought is that classical Chinese philosophy viewed language as a control technology, a tool for coordinating and regulating behavior. Putting on our control engineer’s hat, we can therefore pose two questions:

1. What are the capabilities and the limitations of language as a control technology?

2. If we are to make the best use of this control technology, what should be the internal structure or architecture of a good regulator?

The first question is about what we can and cannot do with language; the second question is about realizability. According to Hansen, the Chinese answer to the realizability question is different from the Indo-European one. The latter is intimately linked with symbolic representations that are manipulated according to the rules of propositional logic. It’s an old idea going back at least to Leibniz and Hobbes. For example, in The Leviathan Hobbes writes that

reason is nothing but reckoning - that is, adding and subtracting - of the consequences of general names agreed upon for the marking and signifying of our thoughts.

Experience in the form of sense data impinges on an information processor that applies a series of truth-preserving transformations. As Daniel Dennett put it memorably in The Intentional Stance, a reasoning subject is just a theorem prover wearing an overcoat of transducers and effectors. This is the GOFAI paradigm of Minsky, McCarthy, Simon, Newell, Pylyshyn, Fodor, etc. By contrast, the Chinese answer is more like connectionism or like the Hayekian sensory order. The processing is parallel, distributed, and nonrepresentational (at least in the sense that the entities being manipulated are high-dimensional vectors rather than logical variables, and the transformations acting on them cannot be readily mapped to some sort of a truth-preserving semantics). It’s pattern recognition and pattern prediction all the way down and all the way up, and we adopt the pragmatic attitude — everything is fine as long as our de (virtuosity) acts to align the discourse dao (laws, customs, norms, regulations) with the performance dao (our behavior in the world, including speech acts).

In this context, we can think of LLMs as models of linguistic control systems that seem to vindicate the Chinese (or connectionist) view of language. Their internal architectures are manifestly nonpropositional. The French philosopher Jean-Pierre Dupuy, in his book On the Origins of Cognitive Science, suggested that one can view the construction of mechanistic models of the mind through the lens of Giambattista Vico’s constructivist thesis of the equivalence (or interconvertibility) of the true (verum) and the made (factum). As finite beings, we are best at rationally grasping those phenomena which we can model and simulate. As long as we don’t stumble into the fallacy of identifying the brain with the latest neural net architecture, the existence of LLMs counts as evidence against the necessity of symbolic mental states (World 2 entities of Popper and Eccles) and for the sufficiency of the connectionist sensory order. We can now move on to the question of capabilities and limitations of language as a control technology. We start by carefully examining the notion of tokens.

Rectification of names as the token grounding problem

The concept of tokens, “the universal interface” of LLMs, maps neatly onto the classical Chinese concept of ming (names). Ideally, names or tokens should not be analyzable into smaller units. That is, they should be grounded in the capacity for making appropriate elementary bian (distinctions) which, in turn, engender more complex distinctions and actions. Stevan Harnad referred to this as the symbol grounding problem; in classical Chinese thought, this was the problem of the rectification of names. Harnad’s proposal was to interpret symbols as conventional names attached to combinations of two basic types of nonsymbolic representations: iconic and categorical. I won’t belabor the details of Harnad’s framework but appeal instead to a somewhat different proposal made by Manuel DeLanda in his book Materialist Phenomenology.

DeLanda’s objective in that book is to develop a philosophy of perception that does not presuppose language, but which nevertheless recognizes the importance of control using some sort of a sign system or a code. Just like James Beniger, DeLanda situates the first appearance of controls of this type in the first living systems:

when sensory organs began to evolve, the world had to already possess something they could exploit: the water, the air, as well as the ambient light that bathes the planet’s surface had to be populated by signs. But what kind of signs could these be? Certainly not symbols, since these can stand for something else only through an arbitrary social convention. But there are two other kinds of signs that do not depend on human communities to exist and could, therefore, be candidates for the natural signs we require: indices and icons. Roughly, an index is a sign that stands for an object by being a causal effect of it, while an icon stands for an object through a relation of similarity.

Examples of indices include smoke indicating fire, animal tracks indicating the presence of prey or predators, a change in the color of flame indicating the presence of a particular reagent, or the change in the conformation of an allosteric enzyme indicating the presence or absence of a substrate that can bind to it. Examples of icons are topographic maps encoding things like different elevations in a given terrain or the pattern of grooves on a vinyl record standing for the sound recording. One can even imagine hybrid iconic-indexical representations. DeLanda’s very nice example of one is using an app like Google Maps on your smartphone: the layout of the map on the screen is an icon that bears a relation of similarity to the physical layout of your physical surroundings, while the dot corresponding to your current location on the map is an index since it is causally linked to your physical location. DeLanda argues that the evolution of biological complexity made liberal use of such iconic-indexical semantics, rather than symbolic representations. Components in biological systems are characterized by their capacity to produce icons and indices and to consume icons and indices produced by other components when they are linked together. The most elementary such components are mindless cognitive agents, yet they serve as building blocks for organisms and for minds.

DeLanda’s emphasis on icons and indices as natural signs provides a viable solution to the grounding problem without foreclosing the possibility of much richer generative semantics using symbols and natural language. Moreover, systems that rely on icons and indices for control and coordination can be implemented using connectionist principles. For example, the structured activation patterns in multilayer neural nets, such as the ones discovered in vivo by Hubel and Wiesel or modeled in silico by Olshausen and Field, can be interpreted as icons and indices standing in appropriate relation to either the external stimuli impinging on the sensory surfaces of the neural net or the internal stimuli generated within the net. They become explicit, or intentional, signs as soon as they are recruited by other assemblies in the system that can make use of them. The key to biological complexity, and ultimately agency and intentionality, is in the rich architecture of systems that can be built from such simpler assemblies.

The limits and the possibilities: Laozi and Zhuangzi

The significance of this perspective for us is twofold. First, by recognizing the arbitrary, conventional nature of language as a symbol system, we can question the constancy of the dao of language and linguistic systems, either human or artificial. As Hansen points out, the concern with constancy was of primary importance in classical Chinese thought. Some, like Mozi and the neo-Mohist School of Names, sought to ground the constancy, reliability, and appropriateness of linguistic dao in quantifiable objective standards. Others, like Laozi, adopted the radical skeptical attitude recognizing the fundamental limitations of language and arguing for the inherent impossibility of a constant dao rooted in convention.

The famous opening lines of Laozi’s Dao De Jing are usually rendered as follows:

The way that becomes a way
is not the Immortal Way
the name that becomes a name
is not the Immortal Name

According to Hansen, however, this standard translation introduces a monist, mystical interpretation in terms of a single ineffable Dao, which is not justified. For one, elsewhere in the text of Dao De Jing there are references to multiple daos (the great dao, the dao of heaven, the dao of water, the dao of the dark moon, etc.). Moreover, if we place Dao De Jing in its historical and philosophical context, we cannot simply assert that Laozi’s usage of dao is radically different from how this notion was used by Confucius or by Mozi. All of these texts use the concept of dao in roughly the same way, as either a prescribed or a realized pattern of behavior or action. From this perspective, the opening lines of Dao De Jing are not about the ineffability of some “cosmic Dao,” but about the inherent limitation of any dao that can be expressed in language. Language is a social control technology for humans and by humans. As such, the distinctions induced by any discourse dao and implemented in any performance dao are based on arbitrary convention and can therefore change arbitrarily. As Hansen writes,

both Confucius and Mozi try to select some prescriptive discourse to be made the constant discourse guide for society. Both understand that the guidance must include training in the use of names— rectifying names — to make the discourse dao generate the intended performance dao. This constancy goal, Laozi announces, is hopeless. We have no way to fix how we may project our use of names in new circumstances. We cannot know whether we have chosen to mark slightly different distinctions or the same. Our social training (learning the code or following models) has no clear implication about how to project in new circumstances. So we cannot guarantee constant guidance from any dao that is generated by following a discourse consisting of names.

Hence, Laozi is concerned with the reliability of language as a control technology and with its inherent dangers. Hansen puts it thus:

For Laozi, what lacks constancy is not the experienced world of particular physical objects, but the system of name use. No unchangeable systems of discourse exist. This is so not because things change, but because names (and their distinctions) do. We are, so far at least, not dealing with a Chinese Heraclitus reflecting on how rapidly things change.

…

Names mark distinctions, not classes of objects. Their knowledge is of ways to do things, including making distinctions and using names. That knowledge must change since each situation is unique. Daoists interest themselves only in this observation. Independently of any actual flux in the world itself, our systems of guidance attach to the world in constantly changing ways depending on conventional, and therefore changeable, practices. That is the philosophical problem about constant guidance that captures the attention of Chinese theorists.

Laozi’s vision is, in this regard at least, radically anti-technocratic. Look at Chapter 80 of Dao De Jing, for example:

Imagine a small state with a small population
let there be labor-saving tools
that aren’t used
let people consider death
and not move far
let there be boats and carts
but no reason to ride them
let there be armor and weapons
but no reason to employ them
let people return to the use of knots
and be satisfied with their food
and pleased with their clothing
and content with their homes
and happy with their customs
let there be another state so near
people hear its dogs and chickens
but live out their lives

This is not so much a rejection of technology as a cautionary message that any technology can condition and control us as much as we think we can condition and control it. The worry about such loss of autonomy was expressed eloquently by Ivan Illich in his essay “Silence is a commons:”

machines which ape people are tending to encroach on every aspect of people's lives, and … such machines force people to behave like machines. The new electronic devices do indeed have the power to force people to “communicate" with them and with each other on the terms of the machine. Whatever structurally does not fit the logic of machines is effectively filtered from a culture dominated by their use.

The machine-like behaviour of people chained to electronics constitutes a degradation of their well-being and of their dignity which, for most people in the long run, becomes intolerable. Observations of the sickening effect of programmed environments show that people in them become indolent, impotent, narcissistic and apolitical. The political process breaks down, because people cease to be able to govern themselves; they demand to be managed.

Daniel Dennett’s fear of “counterfeit people” is another expression of this view, and the same idea is expounded in more detail in David Runciman’s recent book The Handover: How We Gave Control of Our Lives to Corporations, States and AIs. Laozi was worried about much the same things in China of 6th century BC, and even the recent piece by Marion Fourcade and Henry Farrell about how LLMs could upend the rituals underlying human organizations could as well have been written by Laozi about the rites and regulations codified by Confucian bureaucrats.

However, we can temper the pessimism of Laozi somewhat if we recognize and embrace the plurality and the open-endedness of the varieties of dao embodied either in language or in other types of sign systems. Language is inherently ambiguous, but we have no way of escaping it (aren’t we using language right now?). The generativity inherent in complex system architectures making use of icons, indices, and signs is simultaneously a source of existential unease (in the Heideggerian sense) and of wonder. While Laozi was all about the unease, Zhuangzi was all about the wonder. He urged us to recognize that, while there is no “god’s eye view” of reality, there are different perspectives that can be linked together, at least locally, because our actions and discourse (coming, as they are, from within our particular perspective) by means of qing, or “reality feedback.” Hansen puts a Kantian spin on this:

Kant seemed to think of his parallel conception of the thing in itself as a manifold. It was a multiple source of feedback responses or sensations. Daoists are theorizing in a tradition that does not stress constructing concepts out of sense experience. They deal with using language to make distinctions. A Daoist form of transcendental idealism would naturally tend to characterize the thing in itself counterpart as a one. Where Kant imagined knowledge as unifying a manifold, Daoists imagine it as distinguishing a whole into its parts. Zhuangzi, like Kant, allows that we cannot know anything about the absolute object of our conceptual systems. We cannot know, in particular, that it is either one or many. Our metalanguage of a higher or once-removed perspective shows us only a perspective on the plurality of perspectives. Mysticism and skepticism emerge together. What might be one is what we cannot in principle know, given this concept of knowledge.

Of course, we can only interpret this reality feedback through the system of values inherent in our own perspective, which we can nevertheless compare with those of others. This feedback is a form of shi-fei (this-not this) reinforcement learning. Zhuangzi talks about spontaneous practice and mastery of various skills in this context, and he celebrates this in every domain of life, no matter how exalted or how humble (see, for example, the famous passage about the butcher Ding). This is remarkably reminiscent of the dinstinction between closed-loop control as conscious ratiocination and open-loop control as learned automaticity — learning a new skill, such as playing a musical instrument or even learning to walk, relies at first on deliberate conscious feedback and self-supervision and then gradually transforms into a smoothly practiced routine. This also echoes Heidegger’s distinction between simply being-in-the-world and adopting the theoretical attitude, which we have discussed earlier. Once again, Hansen draws a distinction between symbolic, serial information processing and connectionist, parallel processing:

Zhuangzi's conception of action at a high level of skill stresses the spontaneity of the response to a total situation. That response is nearly immediate because the feedback process becomes second nature. In initial training we learned distinctions consciously, deliberately, and with frequent correction. When we have picked it up, we shift the distinction making out of consciousness. It is as if our consciousness were the central processing unit and the distinction is made by an unconscious parallel processor. The parallel processor sends the shi-fei result to out consciousness. We come upon things as such-and-such. Our response to it is as if we had responded unmediated to the world in which the the distinction was a given.

…

The achievement of skill mastery becomes the mature Daoist notion of spontaneity. Intuitive, immediate, nonselfconscious, intimately aware sensitivity to context in performance marks this mastery. In cybernetic terms, our actions involve constant feedback mechanisms which operate at such speed and accuracy, that they resemble massive parallel processing. It is not that I should turn my mind off. The point is that a parallel processor now handles my walking, which once took my full concentration. It frees the central processing unit for other activities: concentration on reading the map or carrying on the philosophical argument. Central processing consciousness also kicks in whenever we are learning or coming to a hard place. Sometimes I have to pay attention to my walking and cannot continue the philosophical conversation. In normal skilled action, the mind is processing a vast number of clues at once. It guides our action without routing the information through our con- scious central processing unit. Our mind is both shut off (the central processing unit) and yet fully turned on (the parallel-processing feedback guidance of action).

This computer analogy of the Daoist view of intuition or spontaneous action gives us a way to illuminate the contrast in the theory of mind in the two traditions. The Western concept of mind and of the computer is of the information processor. The CPU takes in information. The processor exists relative to a cognitive stuff, information. The unit stores information in memory cells, operates on the informa- tion, and reports the result, information output. This reflects the Western focus on conscious thought, deliberation, and choice as cognitive.

The Chinese view of heart-mind focuses on guiding behavior. The mind receives reality-feedback inputs (qing) and processes them all at once (paralleling processing). The output is not a computational result stored in some memory cell or reported out as information. The output is an action. When we have learned to do anything like second nature, we constantly adjust our performance to myriad clues in the environ- ment. We do not mediate the fine adjustment of motor skills by conscious choice. We act, it seems, directly in response to the external environment without CPU involvement.

The dao of generative architectures: constraints that deconstrain

The overall view that emerges from all of this is of a complex layered architecture, where the operation of higher layers makes use of virtualization and abstraction through the interface of icons, indices, and symbols. This idea, discussed in an insightful paper by John Doyle and Marie Csete, is the key to all systems relying on organized complexity and on sign systems as interconnection interfaces. Language is one such architecture, but the same principles underlie the workings of biological organisms, social and legal systems, and technological infrastructures like the Internet. The range of possibilities that could be explored by engineered systems that communicate via the interface of tokens is endless, and so is the range of possible failure modes, including catastrophic ones. This obviously holds for massive connectionist systems that produce and consume various types of signs (or tokens, in LLM-speak). Borrowing an idea from biology, Doyle and Csete speak of all such complex architectures as constraints that deconstrain. The basic idea is that, even though there are constraints on the repertoire of basic building blocks (say, a given system of icons, indices, or symbols), these constraints can be transcended using compositionality, abstraction, and virtualization in higher layers. This is the dao of all complex system architectures.

Zhuangzi and technology

My own perspective is Zhuangzi’s optimism tempered by Laozi’s recognition of the limitations of technocratic governance. Hansen puts it nicely as follows:

Zhuangzi's supposed attitude to modern, rationalized daos such as science, rational morality, or the rule of law would be that they all presuppose something. All rational daos are systems of shiing and feiing based on the desirability of calculation and the realist's regulative ideal, the assumption of a single correct answer. But it does not follow that he would reject them. We can take Zhuangzi's fanciful discussions of the people who can survive in fire, be warm in freezing temperatures, fly in the clouds, stride on the moon, and wander beyond the four seas as speculative predictions. There could be ways of assigning shi and fei which would lead to such accomplishments. Science is an example of such a dao.

In making that point in ancient China, however, Zhuangzi was urging exploration of new daos. He did not have a clear conception of the scientific dao of hypothetical-deductive reasoning. Meeting science now, he might applaud this marvelously beneficial dao. He may also note that we measure its success using its own internal standards of cheng (completion). Who knows, after all, what other ways of getting skill over nature might still be possible? Why rest satisfied with science as a dao? Something, as much more powerful than science as science is more powerful than peasant divination, may still be discoverable.

Should that possibility justify skepticism of science? Zhuangzi's answer, I suggest, would be no. For us, science is the usual, conventional, shareable system of settling shi-fei. There is certainly no reason to abandon it, in fact, one can get as proficient in its evaluative standards as in any other craft or skill. Thus there is no reason to regard Zhuangzi as either antiscientific or antirational.

Happy New Year!

1. Is the discourse dao a reliable guide for performance dao?

2. How can performance dao be aligned with discourse dao?

According to what Hansen calls the “computer analogy,” we use language to program others and are in turn programmed by them. This is how language (a World 3 artifact, in Popper’s interactionist philosophy) makes contact with, and has causal efficacy in, World 1 of material entities. Here’s how Hansen frames it:

Society uses language to guide our behavior. Elders teach us to conform to conventional ways of making distinctions among thing kinds in choosing and rejecting courses of action. We do not learn language in isolation from other ritual practices. Language guides behavior because learning the community's language induces us to adopt a socially shared way of reacting differentially to the world. Chinese thinkers explain this function of language in terms of the scope structure that dominated their attention.

There are two main currents of thought regarding this: the pragmatic realist one (underlying both the baseline Confucianism, with its emphasis on proper rites and the well-ordered social hierarchy, and various reactions to it, such as Mozi’s utilitarianism, Mencius’ innatism, or the neo-Mohist school of names) and the skeptical one (daoism proper, associated with Laozi and Zhuangzi). I will leave the daoist skeptics aside for Part III, here I will focus on the realists.

The Western-Chinese axis as the symbolist-connectionist axis

Hansen’s computer metaphor was inspired by the cognitivist turn in philosophy of mind, associated with people like Daniel Dennett or Paul Churchland. The cognitive science revolution has rekindled the debate between the adherents of the symbolic paradigm of Good Old-Fashioned AI and the followers of the connectionist view originally promoted by Frank Rosenblatt and later instantiated in the theory and practice of neural net learning. The symbolic view emphasized sequential information processing and logical reasoning, whereas the connectionist view emphasized parallel distributed processing, without any explicit symbolic intermediate representations. Hansen comes down on the connectionist side as he rejects the view of programs as arguments-on-paper and instead goes for the input-output formulation.

From this vantage point, the GOFAI symbolic approach is tied to logical reasoning and problem solving, as laid out in the works of Herbert Simon and Allen Newell: The agent’s environment poses problems, which are reflected as external experience. Experience, in the form of sense data, is converted into internal symbolic representations which are then manipulated according to computational rules of inference. David Marr expresses this sentiment forcefully in the final chapter of his Vision:2

[Retinal image] is a continuous two-dimensional array with few points of manifest interest. Yet by the time we talk about people or cars or fields or trees, we are clearly being very symbolic, and I think again that most would find suggestions of symbols in Hubel and Wiesel's (1962) recordings. Our view is that vision goes symbolic almost immediately, right at the level of zero-crossings,and the beauty of this is that the transition from the analogue arraylike representation to the discrete, oriented, sloped zero-crossing segments is probably accomplished without loss of information.

This is the internalist-symbolic take on cognition, tied to Fodor’s mentalese. By contrast, from the connectionist viewpoint, the external experience is an input that triggers the execution of a program implemented in a parallel distributed fashion by a deep neural net. This is how Hansen phrases it:

Western thought, in effect, treats experience as the programming step: experience generates the inner language of ideas with which we calculate. Chinese thought treats the programming as the social process of reading in guiding discourse. Experience then merely triggers execution of the program. The computer has program control tied to the external state of affairs. The senses provide discrimination—branching input to trigger different parts of the socialized program.

Hansen does not mention Hayek, but this is the Hayekian sensory order in a nutshell, a proto-connectionist framework emphasizing the role of the nervous system (or, in daoist terms, xin, the heart-mind) as an instrument of sophisticated classification. The classifications are made in a parallel, distributed fashion, involving both pattern recognition and pattern prediction. As we will see next, the classical Chinese emphasis on names (ming), distinctions (bian), and this/not-this (shi-fei) system of positive and negative reinforcement fits these ideas quite well.

Rectification of names, rectification of tokens?

Hansen frames his analysis of Chinese social-pragmatic philosophy of language in terms of the basic notion of ming, names. The dao of the guiding discourse consists of these names, which closely correspond to the concept of tokens in LLMs. Viewed through this lens, rectification of names, zhengming, refers to the process of aligning the tokens in the discourse dao with the correct tokens in the performance dao—in other words, it’s what we know as RLHF. Hansen puts it thus:

Executing a dao-program requires that we correctly register the external conditions and sensitively adjust our responses. At the base of that requirement is this: we should know the boundary conditions for applying each name used in the guiding discourse. This explains the role of rectifying names and its importance in the Confucian model. Confucius does not use definitions. His concern would be with correct behavioral response, not with cognitive content or meaning. Having control of the word amounts to triggering the right procedure in response to external conditions. Besides, giving a definition would merely duplicate the interface problem. More words or internal programming can help only if the programmer has already properly adjusted our de (virtuosity) to the external conditions for application of those words. A definition can only help if we correctly apply the words in the definition, so it cannot be the general solution to the problem of adjusting behavior. The basic solution is the equivalent of debugging. Run the program in real time and have the teacher (programmer) correct errors.

Linguistic input, arranged as a string of tokens, induces distinctions (Hayekian classifications or discriminations, which can be arbitrarily complex), which ideally should result in acts and behaviors deemed shi as opposed to acts and behaviors deemed fei. The often-expressed view is that LLMs serve as an empirical, constructive proof that intelligent behavior is all about “predicting the next token.” But, if we apply the more sophisticated view based on aligning the discourse dao with the performance dao, it’s not really about predicting the next token, but rather about performing appropriate actions in appropriately categorized situations. You need a good tokenizer (the generator of ming) and a good policy for interacting with your social milieu via the interface of tokens.

The Mohist critique (reward is not quite enough)

Mozi’s critical response to the Confucian tradition was centered on the issue of reliability and constancy. If the goal of rectifying names is to promote the making of appropriate distinctions which would lead to appropriate responses and actions, how do we know that the discourse dao sets the right standard? Mozi’s philosophy is utilitarian, although his utilitarianism is neither hedonistic nor subjective. His evaluation criteria are framed not in terms of happiness or pleasure or satisfaction of desire, but in terms of quantifiable material standards. If we stick with our reinforcement learning analogy, then Mozi’s concern is that reward (the shi-fei signal) is not enough, it needs to be tied to a constant standard. To Mozi, with his background as a craftsman and an engineer, accuracy is just such a constant standard. An action taken in response to a language-driven distinction is reliable if it is reliably accurate in some sense. Low perplexity in LLMs, both at training time and during inference and execution, is one such measure of accuracy. This was elaborated by the philosophers of the neo-Mohist school of names, who constructed a pragmatic semantics of knowledge by framing knowledge as skill (Gilbert Ryle’s “knowing-how”) rather than the Socratic notion of justified true belief. Moreover, this skill must be matched by a reliable disposition to take successful action, such that the operative distinctions are reliably projectable to previously unseen but similar situations. As I will detail in Part III, Laozi and Zhuangzi have been (rightly) skeptical about constancy and reliability of language informed by fluid and ambiguous social conventions.

Summing up (so far)

The main point made by Hansen is that classical Chinese philosophy of language provides a convincing demonstration that we develop a rich theory of interaction of language and action without relying on propositional reasoning and serial information processing. In other words, we can bridge Popper’s Worlds 1 and 3 without going through World 2 of symbol-like mental states. The classical Chinese view of language (and, as it turns out, modern connectionist paradigm in AI as well) interprets strings of names (tokens) as the external interface guiding program execution, where program is understood not propositionally as a proof or a syllogism, but as a parallel and distributed mechanism for pattern recognition, pattern prediction, and action. The way I see it, framing the capabilities of LLMs in terms of reasoning, internal representations, and mental states only complicates matters. Instead, inspired by Hansen’s take on daoist “anti-language” skeptics like Laozi and Zhuangzi and by Manuel DeLanda’s materialist phenomenology, I propose a semiotic reframing of LLMs using tokens derived from natural signs (i.e., icons and indices) as opposed to conventional signs (i.e., symbols). This will be the subject of Part III.

(to be continued)

Now, language is a World 3 entity par excellence. The Popperian partial autonomy of language lies in its generativity in the Chomskyan sense.1 Language, as a cultural technology residing in World 3, is a means of doing things with words in World 1, as per J.L. Austin. By learning to acquire and use language, human minds have first-hand experience of grasping these World 3 entities via mental correlates of words, sentences, and more elaborate linguistic constructions. Language is descriptive and argumentative, both of these qualities find their reflection and representation in World 2 of mental entities, and only via the latter can language have World 1 causal efficacy. From this, it is only a short step to mentalese, Jerry Fodor’s language of thought. For Popper, though, this is a way to reject physicalism by arguing that, without the mental conduit of World 2, no meaningful interaction between World 1 and World 3 is possible.

The current debate about the internal workings of Large Language Models is, I think, best viewed through this Popperian lens. We mine World 3 for training data to feed into these massive systems and expect them to interact with, and causally affect, World 1. The Popperian interactionist stance would be that the accumulating empirical successes of LLMs provide evidence for the reality of their mental states, their ability to represent their world and to reason about it. However, one could also argue that the successes of LLMs are actually evidence against the universality of interactionism—it may be possible to bypass World 2 of mental states and still use World 3 objects to control World 1. To put it differently, this debate really comes down to the question of the role of language—is it a means for representing and describing reality or is it a tool for coordinating and regulating action?

Popper seems to come down on the former side. He takes it for granted that language is descriptive and argumentative, so the World 2 links between Worlds 1 and World 3 are propositional, deductive, Aristotelian. He shares this view with the logical positivists of the Vienna circle. On the other hand, someone like Wittgenstein2 or Heidegger would come down on the latter side. Most of our behavior and interaction with World 1, being-in-the-world, is not propositional, it is embodied in routines, to use Philip Agre’s terminology. Only when something interrupts the flow and demands our attention and reflection, do we adopt what Heidegger called the theoretical attitude. Sunny Auyang describes it in her book Mind in Everyday Life and Cognitive Science:

The primary meaning of our being, [Heidegger] argued, resides not in theoretical and scientific thinking but in our everyday handing of things and being with other people. Imagine yourself driving a car, negotiating the sharp turns, enjoying the scenery, listening to music on the radio, musing about the friend you are going to visit. The car handles so well you are almost oblivious of it; it is so integrated into your activity it has become a part of you, so to speak. All things—your friend, the car, the music, the road, the scenery—are interrelated and constitute your world with its primary meaning, a living meaning. Everything is significant and contributes to your purposive activity, but the significance is tacit and not explicitly articulated. The car is something you maneuver to keep on the road that leads to your friend. The car, the road, and your friend are handy. They relate to each other, and together they constitute the world of primary significance.

If the car breaks down, however, it ceases to be handy equipment and becomes a mere thing present to you. Then you switch to the theoretical attitude and try to figure out what is wrong with the thing and what to do with it. In doing so, your world acquires a secondary significance. According to Heidegger, the theoretical attitude is a common mode of being human, albeit a secondary and derived mode. When you switch to the theoretical attitude, you abstract from many cares and concerns that you are normally involved in, focus your attention on a few things, and make them stand out by themselves as things in space and time. You trade the richness of wider experience for the clear and refined vision of a few objects.

Now, this being-in-the-world described by Heidegger is still bound to language, though the bonds are not those of propositional notion of truth. While reasoning is obviously involved, it is not a syllogism-on-paper type reasoning that Popper seems to have in mind as the prime example of World 2 as interface between Worlds 1 and 3. It is a pragmatic reason, focused on empirical success. With this, comes a different view of the role of language—as a control technology, rather than a means of filing and representing facts. This, according to Chad Hansen’s fascinating book A Daoist Theory of Chinese Thought,3 is how classical Chinese thinkers conceptualized language:

All the ancient [Chinese] thinkers viewed languages as a way to coordinate and regulate behavior. No one in this tradition developed a theory that the central function of language was representing or picturing facts or reality.

According to Hansen, a good way to grasp the difference between the Indo-European view of language, centered on propositional truth, and the Chinese daoist view of language, centered on effective and reliable behavior in the world, is through what he calls, interestingly, the computer analogy:4

Substitute the notion of a computational program for the familiar picture of the mental-arguments concept of reasoning. Use the computer model to explain how a physical being can process language and how language guides action in a real world context. We need not explain a computer's operating by attributing to it mental or semantic content, inner consciousness, or experience.

A computer operates with a program. We input the program—load it into the computer. That process changes the computer's dispositions in complex ways. It will now behave differently to different inputs. We may use the language of stimulus-response or even of intuition, but the relation of input and output need not be simple minded. It can come from a very complicated program. The computer's dispositional state has a physical realization (an electronic functional state) that may produce very subtle behavioral responses after a complicated calculation.

The explanation of the computer's behavior does not require our habitual contrast of belief and desire. The program itself causes the behavior (of course, using energy supplied as electricity and oil, requiring a dust-free environment, and resting—cooling off). …

If we accept that the Chinese philosophy of mind did not rest on a mind-body dichotomy, then we surely could use something like the computer analogy to explain intelligent human behavior. The computer analogy illustrates how a physical thing could reason, act, be moral, and function in the world as humans do. A computer can print out the result of a complicated calculation when we input the data. It does a calculation though it does not have beliefs. We do not suppose that the computer reflects on the meaning of the premises and sees the conclusion of its proof in a kind of rational insight. Informally, it helps us see why we do not need to assume that Chinese philosophers took the traditional Western theory of the relation of mind, language, and the world for granted.

He further writes:

We get a model that explains the central position of dao (way) in Chinese philosophy. A dao is analogous to a program. Confucius viewed education as inputting the inherited dao (guiding discourse) of the sages. We study and practice a dao. We learn to speak and act properly by studying the Book of Poetry or the Book of Rites. This view accounts for Confucius' distinctively non-Western attitude that instilling tradition is a realization and fulfillment of human nature instead of a limiting constraint. The de (virtuosity), as the traditional formula had it, is the dao (way) within a person. It is the physical realization of the program that generates the behaviors. When we have good de (virtuosity) our behavior will follow the dao (way). The program runs as intended in us. Good de (virtuosity) therefore, is like a combination of virtue (when compiling a moral dao (way)) and like power (because executing instructional programs enables us to do things). Our virtuosity is the translation of an instruction set into a physical, dispositional potential.

Daoist thinkers make this view of things especially clear. The Laozi introduces the idea that we create desires by learning guiding discourse—gaining knowledge of what to do (know-to). The programming model explains many issues of classical Chinese thought. Notice first that we are, in a sense, programming each other. Our outputs include language that is input to others. The importance of maintaining cultural traditions, the family, the father model of the ruler and the educational role of political society all have clear motivations once we adopt this model. …

Instantly this model gives us a new conception of the roles of language and mind. Pragmatic (action centered) rather than semantic analyses now make more sense. Language guides and controls behavior. It does this by restructuring our behavior guiding mechanism, the xin (heart-mind) The common translation of xin as heart- mind reflects the blending of belief and desire (thought and feeling, ideas and emotions) into a single complex dispositional potential. We need not attribute separate structures of reasoning and feeling to computers to explain their behavior.

Hansen’s use of “program,” and his overall take on language, is very similar to James Beniger’s ideas. However, his notion of computation (and of program) seems to be a lot more liberal than Beniger because he explicitly rejects the identification of programs with proofs and instead goes for something closer to Marvin Minsky’s society of mind framework. In the next post, I will discuss the relevance of this way of thinking to LLMs (in particular, the emphasis on rectification of names, on making of actionable and projectible distinctions, and on using language as a guide to reliable behavior in the world).

(to be continued)