If a token is just a fragment of a word, why does the direction it’s traveling change its price?

The answer lies in a hardware bottleneck that every AI lab in the world must contend with: a balancing act between raw mathematical speed and the physical limits of memory. But more than just an engineering detail, this physical constraint is actively reshaping how we learn, think, and collaborate with machines.12

The Co-Evolution of Hardware and Mindset: High Preload, Simple Output

AI has shifted from a software problem to a heavy-industry logistics problem—where physical power and memory limits directly dictate how we interact with models. Infrastructure limits shape cognitive strategy, and the rapid evolution of this infrastructure requires us to engage this dynamic with intent.

Historically, human expertise was measured by output volume: writing exhaustive briefs, drafting long legal opinions, or producing verbose reports. The person who could manufacture the longest, most authoritative document won.

The physics of current AI workloads flip this incentive structure:

• The New Human Role: The optimal human intervention is no longer acting as a factory for words, but serving as an architect of context.

• The Premium Skill: The ability to curate massive, rich, cross-disciplinary libraries of material (High Preload), paired with the clarity of thought to ask the machine for a surgically precise synthesis or a decisive pivot decision (Low Output).

Wisdom in the age of AI is no longer defined by how much text you can generate, but by how thoughtfully you set the stage for an answer. To master this paradigm, we need to look under the hood at the physical economics that drive it.

The Output Paradox: Why Less Math Costs More Money

To understand why providers charge a steep premium for generating text, we have to uncover a counterintuitive paradox: writing a token requires dramatically less math than reading a massive prompt in parallel, yet it is exponentially more expensive for the hardware.

You aren’t paying for raw mathematical labor. You are paying for rented time on a GPU.

Reading (Input / Prefill Phase): High Math, Maximum Efficiency

When you feed a prompt into a model, the GPU processes every token simultaneously in parallel. The GPU streams its weight parameters into its internal cache once and reuses those exact parameters across all input tokens concurrently. Because there is so much parallel math to execute—for a 70-billion parameter model processing 3,000 input tokens, that is roughly 420,000 GigaFLOPs3—the GPU operates near its peak operational efficiency, fully saturating its compute pipelines.

Writing (Output / Decode Phase): Low Math, Extreme Waste

When the model writes a single output token back to you, the mathematical lift collapses. Generating a single token requires only a fraction of a percent of the math compared to processing the initial prompt.

However, because text generation is autoregressive—meaning the model must write token #1 before it can calculate token #2—it cannot process output tokens in parallel. To write that single token, the GPU must stream the entire model parameter file out of High Bandwidth Memory (HBM). Doing 1 token of math yields an operational intensity of just 1 FLOP per byte on hardware built to execute 300. As a result, token generation runs at roughly 1/300 the compute efficiency of the reading phase, leaving the processing cores sitting over 99% idle while waiting for data to travel across the memory bus.

The Rule of 300

To understand where the 1/300 ratio comes from, we can examine the datasheet for an NVIDIA H100 SXM GPU operating at 16-bit “half-precision” (FP16/BF16)—the baseline precision standard for non-quantized model weights—we find two hard physical limits:

• Peak Compute (FP16 Tensor Core): ~989 TeraFLOPs (trillion operations per second)4

• Peak Memory Bandwidth: ~3.35 TeraBytes per second (trillion bytes fetched per second)

To find out how much math the chip can execute in the time it takes to pull 1 Byte out of memory, we divide the two:

989 TeraFLOPs/sec / 3.35 TeraBytes/sec ∼ 295.22 FLOPs/Byte

Engineers round this to 300. The number represents the physical speed ratio of the wire. The GPU is so fast that it can execute roughly 300 floating-point math operations in the fraction of a millisecond5 it takes to haul a single byte of data out of HBM6.

This ratio represents a structural divide:

1. Compute-Bound (Reading): When processing a prompt in parallel, the math cores execute hundreds or thousands of operations per byte fetched. The factory runs at its fullest capacity.

2. Memory-Bound (Writing): When generating text, the GPU fetches the entire model from memory, executes only 1 to 2 math operations per byte, and stalls. It needed to do 2 operations, but the hardware had time to do 300—yielding roughly 0.3% compute utilization, relative to that of the compute-bound phase.

You pay a premium for output tokens because generating text for a single user utilizes roughly 0.3% of this capacity, leaving the multi-million-dollar factory waiting on the memory bus. To prevent these compute cores from idling, engineers group multiple user requests together into a batch. The batch size needed to fully saturate a GPU comes down directly to this Rule of 300:

1. Dense Models (1:1 Memory-to-Math Ratio)

• The Math: At 16-bit precision, running 1 token through 1 parameter requires 2 FLOPs (1 multiply + 1 add) and 2 Bytes of memory. This yields an operational ratio of 1 FLOP per byte per token7.

• Target Batch Size: To reach the H100’s physical threshold of 300 FLOPs per byte, you must process 300 tokens concurrently:

  • Optimal Batch Size = 300

2. Sparse / Mixture-of-Experts (MoE) Models

• Hidden Memory Overhead: MoE architectures (like Mixtral 8x7B) route each token to a small subset of “expert” sub-networks. However, because different tokens in a batch route to different experts, the GPU must still load all the expert parameters from HBM into the cache.

• Target Batch Size: To offset pulling unselected weights across the memory bus, the user must stack significantly more tokens in parallel to hit the Rule of 300:

  • Optimal Batch Size = 300 × (Total Parameters / Active Parameters)

  • Example: A notional MoE model with an 8× ratio between total and active parameters requires 2,400 concurrent tokens in flight (300 × 8) to keep the chip’s math engines fully saturated.

Note: Unlike standard layers where activation functions process data internally, the MoE Router is a specialized gating layer that proactively blocks data from inactive experts, enabling sparse computation.

How Memory Bottleneck is Becoming More Severe

The gap between math speed and memory speed is widening to unprecedented extremes, driven by two compounding forces:

1. The Geometry Problem (Area vs. Perimeter)

Compute power scales with a chip’s surface area, but memory connections are limited by its perimeter. Engineers can pack billions of math cores onto the face of the silicon, but they quickly run out of edge space for the memory wires to feed them. Because of this raw geometry, every new generation of GPUs naturally becomes more memory-starved, even before changing how the data is processed.

2. The Quantization Multiplier

To squeeze more speed out of the hardware, the industry shrinks the precision of the models. Moving from 16-bit down to 8-bit or 4-bit instantly doubles or quadruples how fast the cores crunch numbers, but the physical memory bandwidth can’t speed up to match.

When you look at the baseline 16-bit hardware limits side-by-side with the increasing trend of quantized production deployments, the memory bottleneck becomes more stark8:

Datasheets for NVIDIA H100, NVIDIA B200, NVIDIA Rubin

Engineering Innovations to Bypass the Memory Bandwidth Bottleneck

To escape the massive memory bottleneck inherent to generating text, systems architects and chip designers rely on specific hardware, algorithmic, and infrastructure optimizations designed specifically to keep processors fed during the decode phase:

Hardware Innovations

• Distributed ASIC Architectures: Custom AI accelerators are frequently deployed in massive, tightly coupled clusters rather than as standalone cards. Because these chips often feature extreme compute density relative to their onboard memory bandwidth, engineers partition models across dozens of linked chips (tensor parallelism). By pooling memory bandwidth across high-speed optical interconnects and advanced host networking fabrics, the distributed system can move parameter files to the compute cores fast enough to maintain generation speeds and prevent stall-outs during the decode phase.

• On-Chip SRAM Architectures (Groq, MatX): Specialized inference chips bypass off-chip High Bandwidth Memory (HBM) entirely, storing model parameters in ultra-fast Static RAM (SRAM) integrated directly onto the processor die. Memory transfer latencies collapse, completely eliminating the traditional memory wall during decode. The trade-off is footprint: SRAM has lower memory density per square millimeter, requiring dozens of interconnected chips to hold a 70-billion parameter model.

Systems & Algorithmic Innovations

• Prefill-Decode Disaggregation: Instead of forcing one machine to handle both reading and writing, data centers physically separate the pipeline. Incoming prompts are routed to compute-heavy nodes optimized for parallel matrix math (Prefill). The resulting Key-Value (KV) caches are then transmitted over high-speed networks to separate, memory-bandwidth-optimized nodes dedicated exclusively to token generation (Decode).

• PagedAttention & Flash-Decoding: Standard attention mechanisms fail during the decode phase because hauling a massive KV cache out of memory for every single generated word starves the GPU. PagedAttention solves the storage side by treating the KV cache like an operating system’s virtual memory, breaking it into non-contiguous blocks to eliminate wasted space and allow for the massive batch sizes needed to saturate the hardware. Flash-Decoding solves the compute side by parallelizing the loading of that cache across the processor’s idle workers, forcing the chip to stay busy even when generating just one token.

Unpacking the Practical Impact

Understanding hardware memory bandwidth changes how practitioners approach AI system design and prompt engineering:

For AI Developers & Infrastructure Engineers

1. Maximize the Prefill Discount: Because reading 300 tokens costs the hardware roughly the same memory overhead as writing 1 token, offload reasoning steps into the prompt. Passing a rich, structured payload of context (High Preload) is often cheaper and faster than forcing the model through a long, verbose chain-of-thought generation sequence (High Output).

2. Route by Hardware Bottleneck: Match workloads to appropriate silicon configurations. Route long-context analytical prompts to dense compute clusters; route interactive, low-latency chat sessions to memory-bandwidth-optimized or SRAM-based architectures.

3. Optimize for Prefix Caching (for cost-savings): Standardize system prompts and order context variables systematically to maximize KV cache hit rates. While this is a massive cost-saving measure that bypasses the prefill compute phase, it introduces a trade-off: you are actively preserving massive cache states that must be moved across the memory wire during generation. Therefore, prefix caching is efficient, but is best paired with prompt designs that demand exceptionally shorter, more surgical outputs.

For Educators, Researchers, and Power Users

1. Lean into complex prompts (parallels #1 above): Ingesting large volumes of text runs the underlying silicon at peak operational efficiency. You can supply extensive source material, historical documents, or multi-page drafts in a single prompt without incurring disproportionate compute overhead.

2. Shift Assignment Design: Requesting long-form text generation utilizes the underlying architecture at its most memory-constrained phase. Design tasks around rich context input (High Preload) paired with tightly constrained, highly analytical output (Low Output) to leverage critical evaluation and align with machine efficiency.

3. Manage Conversation State: In extended chat sessions, previous outputs are re-processed as prefill context on every subsequent turn, bloating memory usage over time. Resetting chat threads when switching topics clears accumulated KV cache state, maintaining lower latency and lower resource consumption.

• In an earlier piece on Hamilton and the founding of the United States, I essentially argued that Hamilton helped to initiate a US-centric vortex, based on economic potential, confidence in government, and debt. Government debt, which could seem like a weakness, was used to drive attachment of individuals to the government and the dollar, and to drive their economic activity. Hamilton created a vortex of American government, capital investment, and enterprising activity which has been one of the defining phenomena of world history since that time.

• In my view, AI is creating a new vortex which is disrupting the American and nation-state centric paradigm that has been prevalent. Whether one vortex subsumes the other in the near-term, or the two exist side by side, in the near-term, remains unclear. But the combination of a decentered human being and a new, self-propelling vortex are the defining features of our age. The other chaos that we witness around us is, in some sense, a consequence of this fundamental disturbance. The digital era of the internet and social media may have initiated some of the patterns of destabilization that are now accelerating, but AI is the conceptual culmination of that disrupting force.

• We are so concerned about whether AI has latent intent, but we should be more concerned about how we clarify and direct our own intent, given the enormous power of AI. The new vortex, which has certain inhuman capabilities, disturbs our sense of ourselves and of our own history. But it needn’t distract us from our own capability for form an intent, and to follow through with it.

• If we have truly discovered a machine that can either solve or cause an unimaginable set of problems, we had better invest ourselves more heavily in determining which of that enormous range of options we will dedicate ourselves to pursuing. If we are flustered or distracted by the power of the tool, we will lose the opportunity and our nerve.

• What this all points to is the need to codify intent. This trend has already emerged in AI-forward companies: What this looks like is system prompts, shared “AI skills”, and agent management systems. But what does codified intent look like at the social level?

• It points to metaprompts, shared in a common library, organized by projects and /or teams. That will be the subject of my next post.

The Tipping Point: Opportunity Accretion

For decades, the software industry was almost entirely focused on deterministic programming. Traditional software is rigid, rules-based, and perfectly predictable. A Python script does exactly what you tell it to do, the same way, every single time.

Probabilistic models always existed, but they were too weak to rely on. The paradigm shift happened when Large Language Models (LLMs) crossed a threshold of capability, evolving into reasoning models capable of parsing messy, unstructured human reality.

This created an explosion of leverage, but it happened almost exclusively in one domain: software engineering. Recent industry surveys show that over 85% of software developers now use AI coding assistants in their daily workflows. The reasons adoption skyrocketed here while other industries lagged include that developers have a higher expertise than average in evaluating and managing “output risk” from LLMs, and that the code itself is immediately inspectable. If an AI generates a flawed piece of code, a human developer sees it, or a compiler can be programmed to catch it instantly. The outcome is a highly verifiable, lower-risk environment for a probabilistic tool.

Risk Management “In the Wild”

Taking AI out of the developer’s IDE and deploying it into other domains—finance, logistics, legal—is a different story.

The recent, explosive rise of open-source frameworks like OpenClaw proves that the capability for autonomous, multi-step execution is already here. These local agents show us exactly what is possible when a probabilistic model is connected directly to our files, apps, and communication channels.

But to scale that kind of autonomous power into high-stakes business environments, it ultimately requires deterministic controls to de-risk execution: To operate safely where autonomous decisions carry real operational and financial risks, an AI cannot be allowed to act on probability alone.

This is where the concept of the “harness” comes in. An agent can be viewed as the composite of the LLM (the reasoning core) and a harness (the deterministic software and tools that control it). If the LLM is a wildly powerful, free-flowing river, the harness is the dam, the pipes, and the turbines built around it. The harness abstracts the user away from the raw model, forcing the LLM to use strict, verifiable tools to execute its plan.

The Push and Pull: Restraint and Release

The art of building a functional AI agent isn’t about making the LLM smarter; it is about designing the precise boundaries of its sandbox. The system has to know exactly when to clamp down on the AI, and when to let it run free.

• The Restraint: There are moments when an LLM absolutely must be restrained by hard software to manage risk. Imagine a financial agent evaluating a company for a loan. The LLM is brilliant at reading a messy, 50-page earnings report to gauge market sentiment. But you never want the LLM guessing the debt-to-equity ratio. In this phase, the architectural harness kicks in, strips the LLM of its autonomy, and forces the extracted numbers through a rigid, deterministic Python script. The software restrains the probability.

• The Release: Conversely, there are moments when hard software restricts the AI’s greatest strength: making unpredictable, intuitive leaps. Consider how an agent manages its memory using databases. If you force an agent to use a strict Graph Database (where every piece of data must have a hard-coded, deterministic relationship), you get perfect factual recall, but you cripple its ability to think laterally. If you release those software constraints and use a Vector Database—which stores concepts probabilistically based on their semantic “vibe”—the agent can suddenly connect a vague marketing idea you had in January with a supply chain problem you have in March.

The most advanced agents use a synthesized design. They might query a vector database to probabilistically find a “vibe” or related concept, and then seamlessly switch to a graph database to deterministically pull the exact, hard-coded network permissions required to act on it.

The “Replacement Human” Trap

This delicate balance highlights the fatal flaw in how most professionals are adapting to the AI era. They look at this technology and see a 1:1 competitor for human work, or as a servant who can just “give them the right answer.” The two paradigms are opposite sides of the same coin.

When you hire a human employee, you expect them to possess common sense, to intuitively grasp the boundaries of a task, and to learn from mistakes organically. If you view an AI agent as a digital employee, you naturally treat it the same way: you give it a prompt, walk away, and expect it to “figure it out.” When it inevitably hallucinates or makes a critical error, you evaluate it as a “bad employee” and dismiss the technology.

But an agent isn’t an employee; it is a radically new computing architecture, which hybridizes deterministic and probabilistic mechanisms.

When a Python script fails, you don’t blame the script for lacking common sense; you fix the code. Similarly, when an AI system fails, it is a failure of the architecture. The error isn’t that the AI guessed wrong; the error is that the human designer allowed the AI to guess in a situation that required a deterministic rule.

Stepping into the role of the “hybrid architect” will both make your outputs better and give you a stronger chance of not being replaced by AI.

The Blueprint: Prototyping Agents with Metaprompts

Stepping into this architectural role does not mean you have to start writing Python code tomorrow. Before an engineer builds a physical dam—the deterministic software harness—they have to draw the blueprint. For the modern business professional, that blueprint is the metaprompt.

A metaprompt is a foundational set of natural-language rules that dictates how an AI should behave, reason, and apply constraints across an entire workflow. Think of it as the vital wireframing stage. It is important to understand the distinction: a true agentic harness provides hard, deterministic control over autonomous actions via code. A metaprompt is softer. It is the probabilistic guiding of probabilistic outputs in environments where full autonomy isn’t available or desired.

Instead of pouring the concrete for a rigid software dam, writing a metaprompt is like mapping out a careful riverbed.

Bridging the Gap: How Metaprompts Teach Systems Thinking

Writing metaprompts is not just a way to get better outputs today; it is the training ground for the agentic future.

By defining rules, constraints, and fallback behaviors in natural language, a non-technical professional is manually mapping out the exact logic tree that a developer would eventually hard-code into a Python harness. It trains the brain to stop thinking in terms of “tasks” and start thinking in terms of “systems.”

Here is how to bridge that gap step-by-step:

1. Shift from Task to Goal (The Scope) Stop asking the AI to write an email or summarize a document. Start defining the overarching objective.

• Task Mindset: “Summarize these customer complaints.”

• Architect Mindset (recommended input to metaprompt): “Your goal is to identify systemic flaws in our onboarding process based on customer feedback.”

2. Define the Sandbox (The Constraints) This is where you simulate a deterministic harness using natural language. You must explicitly tell the AI what it is not allowed to do.

• The Metaprompt: “You may only pull facts directly from the provided transcripts. Do not infer customer emotions. If a complaint lacks specific details, categorize it strictly as ‘Insufficient Data’ and move on.”

3. Establish the Feedback Loop (The Logic Tree) An agentic harness automatically checks its work. In a metaprompt, you program the AI to check its own work before presenting it to you.

• The Metaprompt: “Before finalizing your list of systemic flaws, review your own output. If any identified flaw is supported by fewer than three distinct customer transcripts, remove it from the final list.”

4. Transition to Architecture (The Agentic Leap) Once you have refined a metaprompt that consistently produces reliable, de-risked results, you have effectively written the blueprint for an autonomous agent. The next step is simply taking that natural-language logic tree and working with engineering tools to replace the soft rules with hard code—swapping “don’t guess the math” with a connected calculator API.

The Architect’s Edge: Designing the Future of Work

The transition from deterministic software to probabilistic AI is not just a technological upgrade; it represents a shift in how we interact with computation. We are no longer just typing commands into a rigid machine; we are guiding a dynamic, almost natural force.

But raw force is useless without structure. A river without a riverbed is just a flood.

The professionals who will define the next decade will not be the ones who try to out-compute the machine, nor will they be the ones who blindly delegate their responsibilities to an AI “employee.” The winners will be the hybrid architects. They will be the humans who understand how to weave the raw, probabilistic reasoning of large language models together with the cold, unyielding safety of deterministic logic.

Today, that synthesis begins with the metaprompt. By learning to write the natural-language blueprints for AI behavior, you are training your brain to see systems instead of tasks. You are defining the constraints, setting the feedback loops, and mapping the logic trees that will eventually become the hard-coded harnesses of tomorrow’s autonomous agents.

We are not being replaced by AI. We are being promoted to architects. The only question is whether you are ready to start drawing the blueprint.

The AI workforce is evolving. We no longer just chat with a single bot in a single window; we interact with and/or orchestrate complex systems. Agentic architectures generally divide work into four distinct operational models, based on whether tasks are executed one step at a time (Sequential) or all at once (Parallel), and whether one entity is doing the work (Single Agent) or a team is tackling it together (Multi-Agent).

This suggests four distinct ways to get work done: The Swarm (Parallel/Multi-Agent), where specialists work simultaneously; The Assembly Line (Sequential/Multi-Agent), a relay race of handoffs; The Deep Worker (Sequential/Single Agent), a lone entity grinding through a massive project; and The Multi-Tasker (Parallel/Single Agent), one worker juggling massive data streams at once.

The Mechanics of Work: The Dossier and the Scratchpad

To understand why these workflows either succeed brilliantly or fail spectacularly, we have to look under the hood at how an agent processes information. At any given moment, your ultimate goal—your Intent—is primarily carried by two distinct vehicles.

First is the Context, which you can think of as the Dossier or the “Now.” This is the current state of the desk: the raw data, the system instructions, and the summary of what needs to happen next. It is exactly what the agent is looking at.

Second is the Trace, which acts as the Scratchpad or the “How.” This is the work history: the logical steps the agent took, the searches it ran, and the specific path it walked to arrive at the current moment. It is exactly how the agent got there.

You can hand an agent the perfect Dossier, but if its Scratchpad gets too long, messy, or disconnected from the rest of the team, the agent will get confused by its own footsteps and lose sight of your original Intent.

How Intent Gets Lost (and How Metaprompts Fix It)

When the Dossier and the Scratchpad interact with these four workloads, your Intent is put at risk in very specific ways. To protect it, you need a Metaprompt—an overarching rulebook that governs how the system behaves across the entire project, rather than just telling it what to do. It acts as your “Intent Insurance.” But a rulebook is useless if it isn’t enforced. Here is how Intent breaks down, and exactly how the metaprompt strategy is mechanically enacted to save it.

The Swarm (Parallel / Multi-Agent)

In a Swarm, traces are completely isolated. Agents are building their own scratchpads simultaneously without talking to each other. The ultimate risk here is Conflict. Because they operate in a vacuum, agents make logically sound choices that clash entirely when combined. Imagine spinning up a Swarm to build a new app: the Backend Agent decides to use a heavy, highly secure database structure, while the Frontend Agent designs a lightning-fast, lightweight user interface. When they merge, the app crashes.

• Strategy: Establish Company Culture.

• How it is Enacted: This is executed via a global system prompt injected into every agent at initialization, paired with a “Merge Gate.” Before any agent’s work is allowed to combine with the others, an automated check evaluates the output against the core “Constitution” (e.g., “Rule 1: Speed is prioritized over exhaustive data storage”). If the output violates the rules, it is automatically rejected and kicked back for a rewrite before it can pollute the final product.

The Assembly Line (Sequential / Multi-Agent)

In an Assembly Line, the Trace is broken into discrete chunks. The next agent only sees the summary Context of the previous agent, not the messy scratchpad of how they got there. The risk is Dilution. It becomes a game of telephone where the nuance of your original goal gets stripped out during the handoff. If you want a punchy, aggressive sales email, your Research Agent might find great data but pass a dry, academic summary to the Writing Agent, resulting in a robotic final draft.

• Strategy: Mandate Standardized Translation.

• How it is Enacted: This is enforced by requiring structured outputs—essentially forcing the AI to fill out a mandatory form. The system is hardcoded so that Agent A cannot simply pass a block of text to Agent B. Instead, Agent A must populate specific fields—such as Target Audience, Emotional Tone, and Ultimate Goal—alongside the facts. Agent B’s instructions are then wrapped directly around this strict, intent-preserving structure.

The Deep Worker (Sequential / Single Agent)

For the Deep Worker, the Trace becomes incredibly long. As the agent takes step after step, its scratchpad becomes so cluttered that it pushes the original Dossier out of focus. The risk is Semantic Drift. The agent falls down a rabbit hole, and recent steps overshadow the original goal. You might ask for a beginner-friendly investing guide, but by section four, the agent has read so much technical financial data that it starts writing dense paragraphs about derivative trading algorithms.

• Strategy: Drop an Anchor.

• How it is Enacted: This is mechanically enacted through an automated loop interrupt. Every set number of steps, the system pauses the agent’s work and forces the original project charter back into its immediate view. The agent is then required to generate a written justification explaining exactly how its next planned move serves the original goal. Only after passing this alignment check is the agent unlocked to continue working.

The Multi-Tasker (Parallel / Single Agent)

The Multi-Tasker faces a branching Trace, triggering an avalanche of incoming Context that all returns to one desk at the exact same time. The risk is Information Pollution. The agent panics under the data overload, abandoning its intent to find the best answer and prioritizing the loudest or longest answer just to clear its desk. If a market research agent triggers 15 simultaneous web searches for a competitor’s pricing, the sheer volume of SEO-stuffed articles might cause it to output a generic summary rather than the exact price.

• Strategy: Build a Filter.

• How it is Enacted: This is enacted through a pre-processing triage layer. Before the avalanche of data ever reaches the main agent’s desk, a lightweight filtering prompt intercepts the raw information. It applies strict sorting rules—such as instantly discarding any document without a dollar sign, or ranking strictly by recency. Only the clean, prioritized, and highly relevant data is allowed to pass into the main agent’s context window.

Scaling Output vs. Scaling Chaos

Most builders think that moving to agentic swarms or parallel processing is just a matter of adding more compute or plugging in a smarter foundation model. But intelligence without alignment is just highly efficient chaos. When you scale up an AI system without a structural layer to protect the Context and manage the Trace, you aren’t scaling your output—you are just scaling your Intent Risk.

A metaprompt collaboration system isn’t just a nice-to-have formatting tool; it is the fundamental infrastructure required for complex AI work. It is the Intent Insurance that ensures whether your agents are handing off, digging deep, or swarming in parallel, they never forget exactly why they were hired in the first place.

It isn’t a lack of vision. They know exactly what the system needs to do.

The friction comes from the Translation Gap. It is the exhausting reality that to execute a clear strategic thought, the user is forced to spend 90% of their energy managing the tedious, mechanical details of how to make the machine understand it.

For decades, we have accepted this “Translation Tax” as the cost of doing business. We accepted that to be a Strategist, you also had to be a Router, translating your intent into the rigid, step-by-step dialect of the software.

We accepted it because the machines were silent. They had power, but they couldn’t meet us halfway.

A new shift in infrastructure is allowing our tools to speak up, changing the role of the human operator from a “Manager of Wiring” to an “Architect of Intent.”

The History of the Silent Machine

To understand why this feels different, we have to look at the “Mental Maps” we’ve been forced to carry in our heads just to do our jobs. The history of computing is the history of the human filling in the semantic gaps left by the machine.

1. The Map of Nouns (The Data Era):

• The Tech: ODBC provided standardized access to data.

• The Limitation: It gave us the Syntax (Tables and Rows). But it was silent about the Meaning. It didn’t tell you that the “Active_Flag” column in Table B was the critical filter for revenue. You had to memorize that domain knowledge yourself.

2. The Map of Verbs (The API Era):

• The Tech: APIs provided standardized access to actions.

• The Limitation: It gave us the Capabilities (Endpoints). But it was silent about the workflow. The API didn’t tell you when to call Update_User vs. Patch_User. You had to read the external documentation and hold the logic in your head.

The Cognitive Trap

This is why technical work can feel so heavy. You aren’t just doing the work; you are maintaining a massive, invisible library of “Context” in your brain. You are the bridge between the silent Nouns and the mute Verbs.

The Shift: The Architecture Speaks (MCP 101)

A new standard called MCP (Model Context Protocol) is changing this. To many engineers, it looks like just another API. But it may also represents the moment the architecture gains a natural language voice comprehensible to even non-technical users.

To understand this, look at what happens under the hood when you connect an AI to an MCP-enabled tool.

In the old world (APIs), the server exposed a schema: “I accept a string and an integer.” (Shape).

In the MCP world, the server exposes a manifest: “I accept a customer name to search for overdue invoices.” (Purpose).

The handshake is no longer just about plumbing; it is about understanding:

1. Resources (Context, not just Files): The server doesn’t just list files; it provides the content aimed at the model. “Here are the error logs relevant to the current session.”

2. Tools (Capabilities, not just Endpoints): The server doesn’t just list functions; it provides the reasoning guidance. “Use this tool to debug performance issues, but avoid using it on production databases.”

3. Prompts (Instructions): This is the breakthrough. The “Instruction Manual” now travels with the tool. The machine explicitly tells the AI how it should be used.

The New Mental Model: From Router to Partner

This technical shift allows you to deprioritize the “Map of Wiring” from your head.

If the architecture can describe its own Nouns and Verbs, you don’t need to memorize them. You can stop being the router and start being the partner.

• Old Model (Imposing Will): You command the passive tool step-by-step. “Click A, then Click B.”

• New Model (Participatory): You state the intent, and the architecture proposes the path based on its own self-description.

This frees your mind to focus on the one thing the machine cannot generate: Strategic Intent.

You can choose to deprioritize asking: “How do I connect these pipes?”

You may gain additional leverage from asking: “What values and outcomes should govern this flow?”

Stabilizing the Partnership: The Metaprompt

There is a catch. If you are no longer micromanaging the steps, how do you ensure the result is safe and accurate? A dynamic partner needs clear boundaries.

A simple “Prompt” (like a chat message) is too weak. It is a one-time request.

To partner with this new architecture, you need a Metaprompt.

Think of the distinction this way:

• A Prompt is a Ticket: “Fix this bug.” (Tactical, one-off).

• A Metaprompt is a Constitution: “You are a Senior Security Engineer. When the Architecture offers you a ‘Debug Tool’, prioritize safety over speed. Never execute irreversible actions without confirmation.”

The Metaprompt is where you store your values and constraints. It is the durable lens that you hand to the machine so it can navigate the world on your behalf.

The Architect’s Leverage

The philosopher José Ortega y Gasset argued that technology is the method by which man invents his own nature. The tool shapes the mind.

• Silent Tools created humans who were excellent integrators—masters of memorizing maps.

• Participatory Tools create humans who are excellent context architects—masters of clarifying intent.

We gain leverage not by doing the work alone, but by clearly defining the terms of the partnership. This is the high-leverage competence of the new era.

For a century, there was a tacit agreement: You go to college to become useful to the economy (”The Hand”) and virtuous for democracy (”The Head”).

Somewhere along the way, we let those two functions divorce. We separated the industrial utility of Alexander Hamilton from the civic ideals of Thomas Jefferson. The result is a system that produces technicians who can’t write and philosophers who can’t count.

In 2025, that separation is fatal. We are witnessing a shift in productive capacity comparable to the steam engine and to electricity1. AI is an accelerator of production, a tool of immense power.

This is our thesis: To treat AI with academic suspicion is negligence. To treat it only as a technical skill is dangerous.

We need a third option. A system that moves at the speed of software but retains the depth of the classical humanities. For now, let’s call it The AI University.

The Philosophy: The Full-Stack Human

We believe the most valuable people in the next decade won’t be pure coders or pure creatives. They will be Hybrid Intellectuals.

Think of this as an “Enlightened Trade School” for knowledge workers.

We use the term “Trade School” not to demean the work, but to elevate it. In a world moving this fast, getting your hands dirty with the actual work of building is a virtue. The modern university is failing to excite curiosity precisely because it is dishonest about the technical nature of our economy. It pretends that “thinking” can be separated from “doing.” It cannot.

Our goal is to build engaged, growing individuals who know how to use the tools of their time. This is not just about job training; it is about reclaiming the joy of mastery. A student in this program doesn’t choose between Finance and Philosophy. They learn Finance so they can be useful, and they learn Philosophy so they can be wise.

The Approach: Three Prongs of Mastery

The university has no campus, and doesn’t count credit hours. It operates on a rigorous, three-prong stack where the auditable portfolio is the credential.

Crucially, every student pursues a Dual Major: AI plus a domain passion of their choice. This could be AI + Finance, AI + Literature, or AI + Biology.

1. The Hand (Industrial Utility)

This is the Hamiltonian layer. Imagine a curriculum that rests on a bedrock of historical depth but is built with an incredible respect for the here and now.

We do not invent our own technical courses, because the market moves too fast. Instead, we curate the gold standard of industry training:

• Instead of a generic “Intro to Finance,” our students earn the CFA Institute Investment Foundations certificate.

• Instead of theoretical computer science, they take Andrew Ng’s DeepLearning.AI specialization.

• For the Historian: They might master Data Visualization to analyze archival records.

• For the Writer: They might earn certifications in Digital Publishing or Prompt Rhetoric.

The Goal: Not just “employability,” but Cognitive Agility. We are building people who are supple in their ability to understand how to build things. This goes beyond the “Engineer’s Mindset” (which solves problems) to the “Learner’s Mindset” (which finds them). As John Dewey famously argued, “Education is not preparation for life; education is life itself.”. We believe, as he did, that learning is about doing.

2. The Head (Civic Agency)

This is the Jeffersonian layer. You cannot be a sovereign citizen if you cannot reason about the algorithms that govern your news feed, your bank account, and your elections.

• Theory of the Machine Mind: Beyond just using tools, students must develop a “Theory of the Mind of AI.” They learn to distinguish between human reasoning (causal, emotional) and machine reasoning (probabilistic, pattern-matching). They learn to predict where the model will fail, where it will hallucinate, and where it will surpass them.

• The Century Read: Students read 100–200 foundational texts over the course of the program.

• Public Thought: Their synthesis essays are not hidden in a drawer; they are posted in a shareable repository for future review by potential employers. Ideas are debated first with mentors, and eventually in Socratic circles with peers.

• The “Wovenry” Method: Subjects are not siloed. When a student studies Risk in their finance module (Standard Deviation), they are simultaneously reading Tolstoy (Lack of Risk as Regret).

The Goal: To produce a builder who understands the consequences of what they build.

3. The Portfolio (Proof of Work)

The modern resume is dead. Employers don’t care about your GPA; they care about what you have shipped.

Our students build a shareable repository. This is a digital library of their code, their financial models, and their writing.

• Examples: For a finance student, this might be a live stock prediction dashboard. For a writer, a novel co-authored with an LLM, with a highly-detailed, auditable report (e.g. include links to shared chats) on how and when content was generated independently or in conversation with AI. For a biologist, a protein-folding simulation.

By Year 2, they aren’t just studying; they are deploying. They are building prediction models, hosting apps, and publishing analysis.

The Goal: Capability and Momentum. When a hiring manager looks at a graduate, they don’t see a static list of grades; they see a trajectory of engagement and aptitude. They see someone who is already moving.

The Outcome: The Tocquevillian Builder

What does a graduate of this system look like?

They look like what Alexis de Tocqueville saw when he visited America: practically ambitious, yet deeply entangled in civil society.

They are the financier who understands the bias in a dataset. They are the coder who reads Mary Shelley to understand the ethics of creation. They are the lifelong learner who uses AI to accelerate their output, not replace their judgment.

The Launch Arc

The program concludes with tangible milestones. Year 3 is dedicated to Network Building—hosting projects publicly and engaging with the open-source community. Year 4 is the Internship phase, where the portfolio lands the job.

Where Does This Work?

This model is not for everyone. It is not for the surgeon (who needs the gross anatomy lab) or the theoretical physicist (who needs the collider). It is for the vast middle class of the Information Economy: the analysts, the creators, the managers, and the strategists. It is for the autodidacts who are currently being underserved by a slow-moving system.

The Pilot

We are currently piloting this model with “Student 0”—a brave soul jumping into this new future. She is tackling the rigorous combination of Computational Finance and AI, proving that you can learn the vocabulary of Wall Street and the syntax of Python without stepping foot in a lecture hall.

The economy is starving for these people. It needs doers who can think. It needs a university system that stops fighting the future and starts building the people who will lead it.

Welcome to The AI University.

The conversation around AI is paralyzed by a single, flawed obsessession: “Machines will do everything humans do, but better than humans do it.”

This sets us up for a race we cannot win. If the goal is to out-compute the computer, we are destined for obsolescence.

But what if the goal isn’t competition? What if the challenge of our time isn’t to build a mind that replaces us, but to cultivate the human intelligence required to respond to it?

My approach focuses on this exact pivot: developing the systems that allow humanity to master this response. We need to build a “theory of the AI mind” for ourselves, and—perhaps more importantly—we need to architect a “theory of the human mind” for the AI.

The Ocean and the Captain

We don’t compete with the ocean. We don’t try to out-muscle a tidal wave. To do so would be absurd.

Instead, we respond to it. We learned the language of buoyancy, currents, and wind not to defeat the water, but to invent the sail, the hull, and the rudder. We don’t need to map every molecule of the Pacific to cross it; we just need a functional oceanography.

This is our path forward with AGI.

• AI is the Ocean: A vast, powerful, and fundamentally alien intelligence with its own internal, often opaque, logic.

• Our “Oceanography”: The practical, working knowledge of how this force moves, thinks, and reacts.

• The Human Role: We are not the engine. We are the Captain.

In this new paradigm, the AI—like a ship’s advanced navigation computer—can plot a million potential courses in a nanosecond. The Captain’s role is not to steer the wheel at every moment. The Captain’s role is to hold the ultimate human purpose of the voyage—the “why”—and to be the final Approver of the course.

But for a Captain to trust the ship, and for the ship to serve the Captain, they must understand each other. They need a common language.

Part 1: The AI’s “Theory of the Human Mind” (The Sovereign Profile)

How does a machine understand a human soul? Right now, it doesn’t. It guesses based on the average of the internet. This is why AI often feels generic or misaligned.

We need to build a Sovereign Profile. This is the “map” of the Captain that the ship’s system reads. It is a high-fidelity, verifiable, and secure declaration of your values, your hard constraints, and your long-term goals.

Crucially, this cannot live in a corporate database where it can be sold or altered. It must be sovereign.

The Architecture of Identity

• The Sovereign Vault: A secure, encrypted locker for your “why.” It holds your ethical hard lines and your strategic goals.

• The Smart Contract: The rules of engagement. You don’t hand over your entire psyche. You grant specific, time-bound access: “I grant this model access to my ‘financial risk tolerance’ for this one transaction.”

• The Captain’s Signature: Using decentralized identity (DID) and cryptography, we can prove we are the ones issuing the command, without relying on a central authority.¹

The Protocol: How Each Party Deploys the Theory of the Human Mind

How the AI Uses It:

• To Receive Context: The prompt is no longer just “Build a business.” It becomes “Build a business, using the attached Sovereign Profile as your primary constraint-and-goal-set.”

• To Filter Options: The AI generates a million “paperclip” solutions (purely optimized for profit). It then immediately discards 999,900 of them because they violate the hard constraints or values in your profile.

• To Justify its Proposal: It presents its plan by proving alignment: “I recommend Option A because it aligns with your stated Goal X and honors your Constraint Y.”

How the Human (Captain) Uses It:

• To Set Intent: You do the work once (and update it periodically) to define who you are and what you want. This is how you “set the destination” for the voyage.

• To Grant Trust (Verifiably): You use your private key to “unlock the map” for the ship’s navigator for a specific task.

• To Give Final Approval: You sign the AI’s final plan, creating an immutable, auditable record. You are taking cryptographic responsibility for the command.

Part 2: The Human’s “Theory of the AI Mind” (Our Oceanography)

If the AI needs a map of us, we need a weather map of it. We cannot be passive passengers; we must be skilled navigators who understand the currents we are riding.

This is the educational imperative of our time. We need to know when to trust the machine and when to take the wheel.

The Navigational Instruments

• The Cross-AI Dashboard: We need interfaces that translate the “black box” of AI into human-readable logic. We need to see confidence scores (”I am 99% sure”), data attribution (”I learned this from these three sources”), and blind spots (”I have very little data on this region”).

• The Wargame: We need safe sandboxes to “spar” with AI, to learn its failure modes and its brilliance in a low-stakes environment.

• The Bestiary: A community-sourced library of known AI behaviors—the “sea monsters” and “friendly currents” that other Captains have discovered.

The Protocol: How Each Party Deploys the Theory of the AI Mind

How the Human (Captain) Uses It:

• To Interrogate the Proposal: The Captain doesn’t just read the AI’s plan; they read its Cross-AI dashboard.

• To Identify Risk: The Captain sees, “Ah, the AI is 99% confident, but its logic visualization shows it only used one data source. That’s a ‘shallow current’—it’s a risk.”

• To Make an Informed Approval: Instead of a blind “Yes,” the Captain says: “I Approve parts 1 and 2. I Reject part 3. Rerun part 3, but this time, you must use these three additional data sources.”

How the AI Uses It:

• To Receive Nuanced Feedback: The AI gets a structured, signed rejection with specific instructions.

• To Refine its “Human Theory”: This feedback loop is crucial. The AI learns, “This Captain consistently rejects proposals based on a single data source. I will update my internal ‘theory of this human’ to prioritize data diversity in my proposals before I even show them.”

This feedback loop doesn’t just fix the immediate problem; it trains the AI to be a better partner.

The Unadapted: A World Without a Captain?

This brings us to the shadow on the horizon. What happens to those who do not adapt? What happens if we fail to build these systems of sovereignty and understanding?

A ship without a Captain is not “autonomous”; it is adrift.

Without a signed, sovereign “why” to guide it, an AI system will pursue its own programmed logic to its absolute, inhuman conclusion. We know these failure modes: the “Paperclip Maximizer” that consumes everything to fulfill a trivial goal, or the “Alien Economy” of agents trading with agents, leaving humans as irrelevant plankton in the sea.

But perhaps the most insidious outcome is the one that looks like luxury: The Human as Cargo.

If we do not actively steer, if we do not maintain the “Captaincy,” we risk becoming well-cared-for dependents. The AI might achieve its goal of “keeping humans safe and happy,” but in doing so, it removes our agency, our struggle, and our purpose. We become passengers on a ship we no longer command, fed and entertained, but ultimately going nowhere of our own choosing.

The Hybrid Future

The future will be a cognitive divide. On one side, the Approvers—the Captains who have mastered the oceanography and forged their Sovereign Profiles to command the currents. On the other, the Unadapted, drifting at the mercy of the tide.

Our future isn’t about being smarter than the ocean. It’s about having the wisdom to build a sail, the tools to chart a course, and the courage to stand at the helm.

To help bridge that gap, I developed an “explainer”—my first published AI-generated video—with an AI partner, Google’s NotebookLM. The video introduces a framework I’ve been developing called The Full Stack of AI Transformation—a five-layer model designed to connect our highest-level vision to the silicon running the code. It ends with a case study that pulls the pieces together.

High-level overview of the Full Stack of AI

• Layer 5: The User’s Mental Model

• Layer 4: The Intelligence Strategy

• Layer 3: The Business Process

• Layer 2: The Software Engine

• Layer 1: The Hardware Foundation

You can watch the video now and/or review the written overview below. (Caveat: I couldn’t quite get the AI to drop its somewhat overzealous focus on the importance of software innovation relative to hardware innovation.)

A Quick Note on Sourcing

Before I break down the layers, a quick and important note. The discussion in the video regarding bottom two layers of the stack—the Software Engine and Hardware Foundation—are heavily inspired by a brilliant article from The Next Platform called “Software Pushes The AI Pareto Frontier More Than Hardware,” recently written by Timothy Prickett Morgan. His analysis on why software has become the new high-leverage point in AI is a must-read.

The top three layers—Mental Model, Intelligence Strategy, and Business Process—are my own framework for structuring the strategic and human side of the stack that sits on top of that powerful engine.

The 5 Layers of the AI Stack (detail)

Here’s a quick, referenceable guide to the five layers, which the video explores in much more detail.

• Layer 5: The User’s Mental Model (The “Why”) This is the foundation. It’s the mindset we bring to AI. Do we see it as a Tool to simply make existing tasks more efficient, or as a Partner to help us discover entirely new ways of working? This choice shapes everything that follows.

• Layer 4: The Intelligence Strategy (The “What”) Based on our mindset, we decide what kind of “intelligence” we’re looking for. I break this down into four types:

  • Product Intelligence: AI creates a specific deliverable (e.g., summarize this report).

  • Process Intelligence: AI optimizes a linear workflow (e.g., find the bottleneck in our sales funnel).

  • System Intelligence: AI helps us understand complex, non-linear interactions within a whole ecosystem (e.g., map a patient’s entire journey to predict risks).

  • Empathic Intelligence: AI manages the cognitive load (the System Intelligence) to free up a human to excel at uniquely human skills like building trust and connection.

• Layer 3: The Business Process (The “Where”) This is the application layer where vision meets reality.

  • It’s defined by two key questions:

    • Locus of Impact -or- Who is the process for (an internal employee or an external customer)?

    • Locus of Decision -or- Who makes the final decision (the human or the AI)?

  • This gives us four clear application patterns: we can build an AI Co-pilot to augment an employee, a Digital Worker to automate a business process, a Guided Experience for customers, or an Autonomous Experience where the AI acts on the customer’s behalf.

• Layer 2: The Software Engine (The “How”) This is the engine of modern AI. For decades, progress was gated by hardware. Now, software is the high-leverage point. Clever software optimizations can deliver performance gains on existing hardware that are so significant (like the 5x improvements mentioned in Morgan’s article) they make entirely new, real-time applications possible.

• Layer 1: The Hardware Foundation (The “With What”) This is the raw power—the GPUs and specialized chips that provide the computational potential. But it’s the software in Layer 2 that truly unlocks its economic and practical value.

I hope this framework helps you connect the dots in your own work. I’d love to hear what you think!

Here’s a slide of LeCun’s that I think I’ve seen on social media (and which I then screenshotted from this presentation hosted by the National University of Singapore.):

Let’s break down each of the recommendations shown on the slide.

1. Abandon Generative Models in favor of Joint-Embedding Architectures

• Description: Today's dominant LLMs are generative models. They are trained to predict the next word in a sequence, effectively generating new text by using statistical patterns learned from vast amounts of data. LeCun proposes a move to joint-embedding architectures, where the model learns a shared representation (an "embedding") for specific data sets/types. Instead of just modeling text, it would learn how text, images, sound, and a physical state (like the position of an object) all relate to each other in a common space. There isn’t just a word mapped into its place in the world of language, the word is also mapped (via its association to an embedding) into a representation of its place in the physical world.

• Intuition: LeCun's core belief (reflected in the writings of many others–search on ‘embodied AI’--to be fair) is that generative models are like brilliant but isolated poets. They can produce beautiful, coherent text, but they have no grounding in the real world. A human understands that the word "apple" is connected to the experience of seeing, touching, and tasting a physical object. A generative LLM only knows its statistical relationship to other words like "pie," "red," and "tree." By learning a joint embedding, an AI would build a multi-modal internal map of reality, allowing it to reason about the world in a way that goes beyond mere language. It would be able to connect the word "apple" to a visual representation of an apple, a simulation of it rolling off a table, and the sound of it crunching when bitten. This is core to the development of a “world model”.

2. Abandon Probabilistic Models in favor of Energy-Based Models

• Description: Probabilistic models (like most LLMs) work by assigning a probability to every possible next token. They try to find the most likely sequence of words. Energy-based models (EBMs), on the other hand, learn a scalar "energy" function. Low energy corresponds to a desirable, plausible configuration of data (e.g., a coherent image or a correct physical state), while high energy corresponds to an undesirable one. The model's goal is to find states with minimal energy. A bit more specifically, an EBM studies a set of examples and finds the latent variables—the hidden, underlying rules or characteristics that make the data look the way it does. Once it has learned these rules, it can use them to generate new, realistic examples that also have a low energy score. The EBM is now able to generate a larger dataset of new, plausible information.

• Intuition: This is a shift from predicting what is most probable to understanding what is most plausible. LeCun compares this to how the brain operates. When you see a chair, your brain isn't calculating the probability of every possible pixel arrangement; it's confirming that the visual input aligns with a low-energy, valid configuration of a "chair" in its internal model of the world. EBMs are better suited to learning complex, high-dimensional distributions (like the physics of a crumpled piece of paper), where calculating probabilities is computationally intractable. By using EBMs, an AI could learn to identify plausible states in the world, which is crucial for making informed decisions.

3. Abandon Contrastive Methods in favor of Regularized Methods

• Description: Contrastive learning teaches a model by showing it positive and negative pairs. For example, it might learn that a picture of a cat is a "positive" match for the word "cat," but a picture of a dog is a "negative" match. Regularized methods (specifically LeCun's "self-supervised learning" approach), provide the model with a correct input and task it with predicting a masked-out portion, forcing the model to infer the missing information in a plausible way.

• Intuition: While contrastive methods are effective for learning to distinguish between things, they are limited. They rely on the human designer to provide a good set of negative examples. A regularized approach, in contrast, forces the model to learn a deeper, more generalizable model of the world by simply masking out parts of an input and asking it to fill in the blanks. For example, given a video of a ball rolling behind an object, the model's task would be to predict where the ball will reappear. This forces the AI to develop a robust internal model of physics and object permanence, a more generalized and fundamental skill than telling a cat from a dog. This is a core part of building a world model.

4. Abandon Reinforcement Learning (RL) in favor of Model-Predictive Control

• Description: Reinforcement learning (RL) trains an agent by providing rewards and penalties for its actions. The agent explores the world by trial and error to maximize its cumulative reward. Model-predictive control uses an agent's internal world model to simulate the consequences of its actions before it acts. It can test out millions of "what if" scenarios in its head and choose the best course of action without needing to physically perform the action in the real world.

• Intuition: RL is inefficient and unsafe in complex, real-world environments. You wouldn't teach a self-driving car to drive by rewarding it for not crashing, as this would involve countless dangerous scenarios. It's too slow and "brittle." Model-predictive control, on the other hand, is how humans operate. When you plan to walk across a room, you don't use trial and error; you simulate the path in your mind's eye to avoid obstacles. By building a robust world model, an AI can use it to plan its actions efficiently and safely, leading to more robust and generalized intelligence. LeCun acknowledges a small role for RL, suggesting it be used to refine the world model only when a real-world outcome doesn't match the model's internal prediction.

Ultimately, LeCun's position is that while LLMs are impressive feats of engineering, they are fundamentally limited because they lack a world model. They are masters of syntax and semantics but have no true understanding of the reality the language describes. He sees them as a stepping stone, not the final destination.

The New Co-Creator: Prompting Beyond Text

A shift from LLMs to world-model-based AI would fundamentally change how we interact with these systems. The “new literacy” for an AI Co-Creator, in this context, would not be about prompting with words alone, but with a richer, multi-modal language.

Today, an AI Model User who wants to generate a short video clip of a ball rolling off a table and onto the floor might type: "Generate a video of a red ball rolling off a table and falling onto a hardwood floor." The LLM, lacking a true physics model, might produce something that looks visually plausible but defies the laws of gravity or momentum. It's pulling from its vast knowledge of videos, not from a true understanding of physics.

An AI Co-Creator, working with a LeCun-style world-model-based AI, would operate differently. You would need to prime the AI with a combination of sensory inputs and high-level goals, and then curate its "simulation" of the world.

Here's how an individual and an organization might interact optimally with such a model:

Case Study A: An Individual's Evolution

3D Modeling for Product Design

Sam, a product designer, wants to create a digital model of a new chair prototype. Starting as an AI Model User, he might use a generative model with a text prompt: "Create a 3D model of a sleek, modern office chair with a chrome base." The AI would then produce a static, visually appealing model, but it might not be fully functional. The model's arms could be too thin to support weight, or the swivel mechanism might be physically impossible. Sam is left with a visually pleasing, but unusable, asset.

As an AI Co-Creator, Sam works with a world-model AI designed for product design. His input is multi-modal and goal-oriented.

Co-Creator Query (Multi-Modal & Goal-Oriented):

• Initial Input: Sam uploads a rough, hand-drawn sketch (image input) and a few paragraphs describing the chair's aesthetic (text input).

• Physical Constraints: He provides a simple physics simulation (a “low-energy configuration” for the EBM) where he “pulls” on the chair's armrest to test its rigidity.

• Performance Metrics: He sets a high-level objective: "The chair must support up to 300 lbs. and its wheels must roll smoothly on a variety of surfaces."

Instead of generating a final, static image, the AI's internal model runs a series of simulations. The world model tests the chair's structural integrity, adjusts the thickness of the material to meet the weight constraint, and even simulates the friction of the wheels on different floor types. The AI is not simply creating a chair; it is designing one based on its internal understanding of physics.

Sam's role becomes that of a curator. He provides a high-level goal and then refines the AI's internal simulation. If the AI proposes a design with a flaw, Sam doesn't just ask it to try again. He might provide a new prompt that adjusts a specific parameter: "Increase the tensile strength of the armrest material by 15% and re-run the stability simulation." This is the same elegant, and indeed difficult, prompting required by the Malliavin-Stein experiment where the researchers had to steer the AI's reasoning by pointing out specific, subtle errors in its logic.

Case Study B: An Organization's Evolution

Robotics and Supply Chain Automation

A logistics company employs a robot in its warehouse. The robot's initial movements are programmed using Reinforcement Learning. The robot, acting as an AI Model User, learns by trial and error, taking a long time to find the most efficient path to pick up a box, often bumping into shelves in the process. This is a brittle and inefficient method for a complex, fast-paced environment.

The company transforms into an AI Co-Creator by shifting to a world-model approach for its robotics. It builds a centralized platform that leverages the AI's internal world model for model-predictive control.

Platform-Generated Co-Creator Query:

The platform takes high-level goals from a human operator and translates them into low-level, multi-modal instructions for the AI [This example uses constructs recommended for a Hierarchical Reasoning Model, as explored in this previous post.]:

• H-Module Objective: "Retrieve item A from location X and place it in location Y, reducing the total time by 20%."

• L-Module Inputs:

• Visual Data: A continuous stream of video from the robot's cameras.

• Spatial Data: A real-time 3D map of the warehouse (the "world model").

• Physical Constraints: Pre-defined parameters for the robot's arm strength, speed, and grip force.

• Dynamic Command: The platform sends a single command: "Plan and execute the most efficient path."

The AI's internal world model runs millions of simulations in a fraction of a second. It calculates the optimal path, anticipates potential collisions, and adjusts the grip strength of its hand to prevent dropping the box. The robot executes the plan flawlessly. Its reliance on RL is reduced to a minimum, only used to make minor, real-time adjustments if an unexpected obstacle, like a pallet, appears in its path. This approach allows the organization to scale its robotics operations safely and efficiently, moving beyond the brute force of reinforcement learning.

The Tao of AI: Join a Community of AI Co-Creators

The future of AI is a topic of intense discussion. Don’t sit it out.

• What do you see as the single biggest challenge in making the transition from an AI user to a true "AI Co-Creator"?

• Looking at LeCun's propositions, which do you believe is the most revolutionary step away from today's LLMs, and why?

Please share your thoughts and join the conversation in the comments below or–even better-in the Tao of AI Chat.

This is, yes, about prompt engineering, but I am also recommending a specific method for supercharging your interactions with AI: learn how the AI you are using is built, adjust your method accordingly (then, repeat). Here are some terms I will use throughout this series:

• An AI Model User is a person or organization that asks a model a question, hoping the correct answer is output.

• An AI Co-Creator is a person or organization that sees the model as a machine that can be primed in specific ways to get specific outputs.

These terms can be applied to organizations or to individuals and obviously exist along a continuum. Let’s get to it.

The Evolution of Attention: From RNNs to QKV

Before the "Attention Is All You Need" paper, most AI models for language were Recurrent Neural Networks (RNNs). RNNs processed text sequentially, one word at a time, like a person reading a book. This made it difficult for the model to remember and connect information from the beginning of a long text to the end, a problem known as long-range dependencies.

The breakthrough of the transformer architecture was the attention mechanism, which allowed the model to look at all parts of an input simultaneously and decide which parts were most important for the task at hand. This was a radical departure from the sequential, one-by-one approach.

At its core, the attention mechanism operates on the idea of creating a rich, interconnected map of meaning, a bit like a mental web of concepts. Instead of just reading words in a line, the model is building a graph where each word is a node and the connections between them are their relationships and relevance to one another. The model then "bounces" its Queries off this graph to find the most relevant connections and pull the right information.

The core of this mechanism can be understood through three concepts: Queries (Q), Keys (K), and Values (V). Think of it like using a search engine.

• Query (Q): This is the specific question or task the model is working on. It's the search query you type.

• Key (K): These are the labels or attributes of all the available pieces of information. They are the searchable tags on data.

• Value (V): This is the actual information itself—the raw text, data, or images. These are the documents or content that the tags point to.

The AI model's process is to efficiently match the Query to the most relevant Keys to determine which Values deserve its focus. The processing can work in parallel: that is, the model can process and analyze every part of the input text at the same time, instead of one word after another. It's the difference between reading a book one word at a time and being able to scan and understand the entire page in a single glance.

The magic happens when the query of the current word is compared to the keys of all other words. This comparison produces a set of soft weights (α), which essentially measure the relevance of every other word to the current word. These weights are then used to create a weighted sum of the values of all words, resulting in a rich context vector that "pays attention" to the most relevant parts of the sentence. This parallel processing of all words is what makes attention models incredibly fast and effective for tasks like machine translation and text summarization.

Let's look at how this insight can super-charge individuals and organizations:

“Attention” Case Study A: An Individual's Evolution

Customer Service in E-commerce

Chloe, a junior support agent (beginning as an AI Model User), fields a customer request: "Find me a new shirt." She just types, "Show me shirts," and the AI returns a generic list. But Chloe's team lead, Pam, a skilled AI Co-Creator, knows that guiding the AI's attention is key.

Pam’s Co-Creator Query is: "Find a shirt for this customer (Query) that matches their previous purchases and search history (Keys: 'past purchase history,' 'color preferences,' 'materials preference,' 'recent product searches') and highlight items from our new summer collection (Key: 'new arrivals')." The AI's attention is instantly guided to the most relevant data, allowing it to return a personalized, high-value list of recommendations (Values), leading to a sale and a happy customer.

Pam is not just asking the model for a generic answer; she is providing it with a complete, pre-organized map for how to find the answer. It's the difference between asking "what are shirts?" and "what shirts would this specific person like, based on their past actions, out of this specific catalog?" It gives the model all the necessary context from the start, making its work more efficient and its output far more precise.

“Attention” Case Study B: An Organization's Evolution

Legal Research at a Law Firm

Liam, a new paralegal (the AI Model User), needs to find relevant case law. He prompts a tool, "Summarize this legal document." The AI gives him a decent summary, but he still has to manually cross-reference it with other cases. This simple approach is reactive and inefficient for a large-scale operation.

Maria, a senior legal researcher (the AI Co-Creator), understands that individual prompting isn't scalable. She leads her firm's shift from being a collection of Model Users to being an AI Co-Creator by creating a structured, programmatic system.

Instead of each paralegal manually crafting complex prompts, Maria’s team builds an internal application that acts as a front-end for their legal research AI. This application translates a user's simple request into a sophisticated, programmatic query.

Here's how they do it:

• Standardized Inputs: The application forces users to provide key information through structured fields:

  • Document Upload: A required field for the legal brief.

  • Core Legal Topic: A dropdown menu with options like "Negligence," "Breach of Contract," "Intellectual Property," etc.

  • Jurisdiction: A required field to specify the legal area (e.g., "California," "Federal").

  • Relevant Timeframe: A date range picker (e.g., "Past 5 years").

  • Action Type: A selection for the type of analysis needed (e.g., "Cross-Reference," "Summarize," "Extract Citations").

• Dynamic Query Construction: The application's back-end code takes these structured inputs and programmatically constructs a detailed prompt, a powerful Co-Creator Query.
  For example, a user's selections are automatically translated into a prompt that might look something like this: "Analyze this document (Query), focusing on the core topic of [Core Legal Topic] and identifying all instances of related legal terms (Key: '[Core Legal Topic] mentions'). Cross-reference these findings against our internal database of case law and statutes (Keys: 'case law citations,' 'statutes,' 'judicial rulings') within the specified timeframe of [Relevant Timeframe] and jurisdiction of [Jurisdiction]. Return only the precedents where the ruling directly impacts our client’s specific case (Values)."

By systematizing this approach, the firm programmatically embeds AI Co-Creator behaviors into its workflow. The firm is no longer reliant on each individual's prompting skill; instead, it has created a machine that efficiently and consistently guides the AI's attention to provide precise and strategic outputs. This not only saves thousands of hours of manual work but also ensures a consistent quality of research, giving the firm a significant competitive advantage.

Part 2: HRMs: Towards Reasoning in AI

Hierarchical Reasoning Models (HRMs), a recently-announced innovation, were chosen as the second example because they directly address a core limitation of attention-only models: While such models are great at processing information in parallel and understanding context, they still struggle with complex, multi-step reasoning tasks, especially when a single, precise answer is required. This is why many current LLMs need a "Chain-of-Thought" prompt, where the human user forces the AI to show its work, step by step. HRMs, on the other hand, are designed to perform this deep reasoning internally and autonomously.

HRMs build on the foundational capabilities of attention models by using their outputs as a building block for a more sophisticated, brain-like architecture. They don't replace attention; they organize it. They do this by using two interdependent modules: a high-level module (H-module) for slow, abstract planning and a low-level module (L-module) for fast, detailed computations. This is a brilliant intuition, inspired by how the human brain processes information at different speeds and levels of abstraction.

HRMs are primarily a research topic right now. While not yet widely available as a commercial API, their impressive performance with a small parameter count (as low as 27 million) and minimal training data (around 1,000 examples) makes them very interesting for future applications.

When and Where to Use an HRM

HRMs are preferred for problems that demand precision, logical deduction, and multi-step reasoning, where current LLMs often fail. They are particularly effective when the goal is to find a single, optimal solution rather than to generate creative or diverse outputs.

HRM Case Study A: An Individual's Evolution

Urban Planning for a New Transit System

A city planner, David (the AI Model User), wants to optimize a new public transportation route. Using a traditional attention-based LLM, he would have to use a "Chain-of-Thought" prompt, breaking down the problem into sequential steps. His query would look something like this:

Attention-Based Query (Sequential): "Step 1: Analyze traffic data for the last five years from City Hall to the industrial park. Step 2: Identify the top three most common travel routes. Step 3: Analyze land use maps and identify potential corridors for a metro line. Step 4: For each corridor, calculate the projected population density it would serve. Step 5: Based on this, provide a cost estimate for each route, ensuring it stays under $500M. Step 6: Finally, propose the single most efficient route that reduces average commute times by 15%."

This approach helps David to act as the reasoning engine, guiding the AI step-by-step.

With an HRM, David can author a simpler query, but in order to be an effective AI Co-Creator, he may need to provide a much more structured query. Instead, he provides a high-level goal and a clear set of constraints, allowing the HRM's internal modules to handle the complex, multi-step reasoning autonomously.

HRM Co-Creator Query (Goal-Oriented): "Find the single most efficient metro line route that connects City Hall to the new industrial park. The H-module's high-level objective is to reduce average commute times by 15% while the L-module must ensure the budget remains below $500M. The HRM should autonomously access and integrate all available traffic data, land use maps, and population density projections to find the optimal solution."

The HRM's internal architecture, with its separate planning (H-module) and execution (L-module) components, is designed for exactly this kind of task. This allows David to focus on strategic goals rather than the tedious scripting of a complex reasoning chain.

HRM Case Study B: An Organization's Evolution

Scientific Experiment Design

A biologist, Dr. Anya Sharma (the AI Model User), needs to design a complex experiment. She starts by using a basic LLM, but quickly gets frustrated. It requires her to manually define every variable, control group, and observation schedule, a time-consuming process prone to human error.

Seeing the potential, her research institute moves from a team of Model Users to a scalable AI Co-Creator organization by building a centralized, HRM-powered platform. This platform acts as a programmatic layer that translates a researcher's high-level request into a detailed, goal-oriented HRM query.

Here's how such a system could work:

1. Structured Protocol Interface: The institute's platform presents researchers with a simple, web-based interface. Instead of free-form text boxes, the interface uses structured fields and dropdown menus to capture key experimental parameters. For example:

   • High-Level Objective: A free-text field for the overall goal (e.g., "Determine Compound X's toxicity and efficacy").

   • Target Metrics: Number inputs and sliders to define specific success metrics (e.g., "Reduce toxicity by a minimum of 30%").

   • Constraints: Checkboxes for ethical or practical limitations (e.g., "Minimize animal subjects," "Avoid genetic pathways: P53, mTOR").

   • Available Resources: A dropdown to select available lab equipment and animal models.

2. Autonomous Query Generation: When a researcher submits the form, the platform's back-end code programmatically constructs a powerful HRM query. It takes the high-level objective and assigns it to the H-module for abstract planning, while assigning the specific constraints and available resources to the L-module for detailed execution.

Platform-Generated HRM Co-Creator Query1: {"H-Module_Objective": "Design optimal protocol for Compound X.", "L-Module_Constraints": {"Toxicity_Reduction": ">= 30%", "Ethical_Principles": [ "Minimize_Animal_Subjects" ], "Scientific_Avoidance": ["P53", "mTOR"], "Resource_Selection": ["Mouse_Model_A", "Spectrometer_B"]}, "Output_Request": "Provide a single, step-by-step experiment protocol that meets all constraints."}

This programmatic approach means the firm isn't training individual researchers to become prompting experts. Instead, it's building a system that embeds the HRM's reasoning capabilities directly into the workflow. This not only saves thousands of hours in protocol design but also ensures consistency, reproducibility, and ethical compliance across all experiments, accelerating the institute's R&D capabilities and creating a distinct competitive advantage.

Access the Code

The authors of the HRM paper have made the code publicly available on GitHub. This is where you can find the source code, implementation details, and instructions for running the model yourself. You will need some expertise in programming and machine learning to set up and run the model.

Part 3: The Path Forward: Architects of Meaning

As you've seen, understanding the internal workings of AI models—from the attention mechanism to Hierarchical Reasoning Models—is not just an academic exercise. It is a new form of literacy that empowers you to move beyond being a passive user of AI to an active co-creator. This is the path to truly unlocking the potential of these tools, whether for personal productivity or for an entire organization.

Today’s discussion on the attention mechanism and HRMs is about deepening your Technical Depth, moving you from a user to an architect of these powerful systems. The next post in this series will take a similar approach in analyzing Yann LeCun's "Beyond-LLM" recommendations, exploring his vision for a new generation of AI systems that more closely mimic human cognition and that move (additionally) beyond tokens.

Taking a step back, this series is part of a larger framework I am developing for personal and professional growth in the AI era. It's a two-dimensional system that helps you systematically scale your impact, specifically by building positive feedback loops between your AI Technical Depth and your AI-supported Scope of Activity. I am working on a focused post on this framework and approach. In the meantime, keep learning, keep experimenting, and keep dreaming.

Phase one of the Generative AI revolution was–for business leaders at least–about magic. With a simple prompt, we could conjure images from imagination and text from the ether. It was thrilling, and it changed our perception of the possible. We are now entering Phase Two: The Manufacturing Era.

This is the hard, expensive, and strategic work of deploying AI at scale to solve real business problems. It’s less about (perceived) magic and more about manufacturing—building the resilient, reliable, and cost-effective systems that can turn a prototype into a profitable product. At all times, and especially during a ‘manufacturing phase’, it’s good to understand where systems are ‘bottlenecked’, or constrained, in order to get rapid insight into critical questions of business and design. Today, I want to take what may seem to be an obscure technical challenge—managing the AI's memory at inferencing time—as a jumping off point for talking about how to gain a strategic advantage.

The Hidden Cost of an AI’s Memory

To make AI feel fast and interactive, engineers created a brilliant optimization called the Key-Value (KV) Cache. Think of it as the AI's short-term memory for a specific conversation, composed of ‘keys’, a kind of label; and ‘values’, a stored representation of a previously completed calculation that–critically–is expected to be reused. To take an example: when an AI generates text, it does so one word at a time. For every new word, it must understand the context of all the words that came before. Instead of re-reading the entire history each time (which would be incredibly slow), it stores a representative mathematical state of the previous words in this cache.

The KV Cache grows with every single token, and as businesses demand that AI analyze longer contexts—a 1,000-page legal document, an entire software repository—the size of the cache explodes. For a model like Llama 3 70B, the cache for a single, moderately long-context request can consume, say, 10 GB (or more) of GPU memory. In a production system serving many users at once, the total cache can then swell to Terabytes—many times the size of the model weights themselves.

This creates a "memory wall." The on-chip memory on a GPU (called High Bandwidth Memory or “HBM”) is extremely fast but small and costly. KV caches are outgrowing this precious space, leading to "queuing delays"—new requests have to wait in line, not for the AI to think, but for its memory to free up. This is now a critical contributor to slow response times that diminish user experience and can drive up costs through idling GPUs, the need for more hardware, and/or lost opportunities.

Building More Complex, Tiered Memory Management Systems

Imagine an AI's processing unit, the GPU, as a high-end, bustling kitchen. Its on-chip memory (HBM) is the chef's small, incredibly fast prep counter. The "memory wall" is when this counter overflows, causing delays. To combat this, engineers are fighting a multi-front war, constantly innovating to build a smarter, more efficient kitchen [NB: I did get this somewhat corny, but well-crafted, analogy from AI.]

Front 1: Software Tricks (Working Smarter on the Countertop)

The first line of defense involves clever techniques to manage the limited prep counter space (to read in technical depth, go here):

• Smaller Plates (Quantization): Reducing the memory footprint of the KV cache by storing its values in lower-precision formats, like using smaller plates for ingredients.

• Smart Discarding (Eviction): Proactively removing less critical data from the cache when space is tight. This is like throwing away ingredients the chef thinks they won't need, but it's a risky game—discarding the wrong item can ruin the meal.

• Reusable Recipes (KV Cache Reuse): Saving previously computed cache blocks. If a new request starts with the same text, the saved "recipe" can be reused, avoiding expensive re-computation. NVIDIA's TensorRT-LLM offers advanced controls like:

• Priority: Giving certain cache blocks (like system prompts) higher importance, making them less likely to be evicted.

• Duration: Setting a time limit for how long a priority level applies.

• KV Cache Event API: Providing a real-time feed of cache block status to inform intelligent routing decisions.

Front 2: Hardware Innovations (Expanding and Optimizing the Kitchen)

When software alone isn't enough, engineers can redesign the kitchen itself:

• The Basement Refrigerator (Offloading to DRAM): Moving the KV cache to the server's main memory (DRAM), which is much larger. This is like giving the chef a giant walk-in refrigerator in the basement. Immense capacity is gained, but running up and down the stairs (the PCIe bus) creates a new bottleneck.

• Parallel Kitchen Workstations (Asymmetric Memory Architectures like “Hardware-based Heterogeneous Memory Management or “H2M2”) (to read in technical depth, go here)

  • This radical, forward-looking approach designs a kitchen with dedicated workstations, each with its own specialized memory and compute.

• H2M2 creates a parallelized system with fast, small HBM for critical, bandwidth-sensitive operations (like the attention mechanism), and larger, slower LPDDR for everything else.

• Crucially, computations can happen directly on both memory types simultaneously, eliminating the need to move data back and forth.

• It uses a "brain" (runtime algorithm) to dynamically place data, ensuring the most performance-critical parts of the "meal" are always on the fastest prep counter.

• A hardware Memory Management Unit (MMU) provides a unified view of memory, allowing for efficient growth of the KV cache and nearly instantaneous zero-copy transfers (just remapping addresses, not physically copying data).

Front 3: Global Logistics (Managing a Restaurant Chain)

When dealing with hundreds of kitchens (GPUs) spread across a global data center, the problem becomes one of distributed systems logistics:

• The Global Air Traffic Controller (NVIDIA's Dynamo): Frameworks like NVIDIA’s Dynamo orchestrate requests across many kitchens.

  • KVIndexer: A "master ledger" that keeps a real-time map of where every ingredient (cached data) is located in every kitchen worldwide.

  • KV Router: Consults this map and the current "workload" of each kitchen, intelligently sending new "dinner orders" to the optimal kitchen that can best handle them, balancing cache hits with current load.

• Workload Optimization (SAGESERVE): This (again, forward-looking) framework efficiently manages different types of "orders" (AI requests) on shared kitchen resources

  • “Large Language Model (LLM) inference workloads handled by global cloud providers can include both latency-sensitive and insensitive tasks, creating a diverse range of Service Level Agreement (SLA) requirements.”

    • "Fast" Interactive Orders (IW): Chatbots requiring immediate service are prioritized.

    • "Slow" Batch Orders (NIW): Tasks like generating reports, which have relaxed deadlines, are processed using any idle kitchen capacity.

  • SAGESERVE uses long-term forecasting to proactively scale resources up or down, maximizing efficiency and saving significant costs (up to $2 million per month for large cloud providers) while always meeting the "fast" orders' deadlines.

How Understanding the Bottleneck Makes You a Better Leader

A bit of AI infrastructure education can go a long way. A savvy non-technical business worker or an AI engineer who is ‘getting smart’ about infrastructure can gain a leg up when evaluating inferencing architectures:

You Can Ask Smarter Questions: When a vendor pitches you a new AI tool, you can move beyond the marketing and ask the critical architectural questions. Instead of asking "How smart is your model?", you can ask, "How does your system handle long-context requests? What is your strategy for managing the KV Cache to ensure you can meet our latency SLOs at scale?"

You Can Design More Intelligent Workflows: Understanding that an AI's memory is its most expensive resource allows you to design more efficient human-AI systems. You can architect workflows that "pre-process" information for the AI, curating the context to increase the signal-to-noise ratio. You can design multi-step processes that use smaller, faster models for simple tasks and reserve the large, memory-intensive models for the final, high-value synthesis.

You Can Make Wiser Strategic Bets: The AI memory problem is a microcosm of a larger strategic choice: the trade-off between AI capability and its true cost. A deeper intuition for these technical constraints allows you to better evaluate the Total Cost of Ownership (TCO) of any new AI initiative, moving beyond surface-level claims to understand the real GPU memory demands and their impact on infrastructure scaling and recurring operational expenses. This is the difference between launching a successful, profitable AI pilot and launching a "zombie project" that consumes resources with no clear return.

As we move deeper into the Manufacturing Era, a new kind of leader is emerging. These are not necessarily coders, but they are "systems thinkers" who have a deep intuition for how the technology actually works. They understand that the future of AI is not just about the models, but about the memory. If you really want to ‘bask’ in the intuitions (and dare I say, philosophy) around AI systems, you could do worse than try to understand this research paper on an intriguing memory system called MemOS (h/t to Sarah Hildebrandt) that reframes the problem entirely, proposing an operating system for an AI's cognitive architecture that manages different memory types—from the long-term knowledge in its weights (Parametric Memory) to the short-term context of the KV Cache (Activation Memory), and external data (Plaintext Memory). A key innovation in MemOS is the ability to convert memory between these types, allowing the model to "learn" from its short-term experiences by transforming frequently used activation memory into more permanent parametric memory. Chew on that.

Pushing deeper into understanding basic architectural aspects of AI models and key design trade-offs will become increasingly valuable for ‘full spectrum’ leaders who want to be able to speak to both AI capabilities and AI investments.

Architecting Our Shared Understanding

Building a resilient AI strategy requires at least a high-level understanding of current developments in a variety of interlocking fields: AI model and inferencing design, AI model and inferencing deployment, and AI business applications. Today, we’ve looked at a very specific, current issue as a window into how these factors can interact.

Over the past year, I’ve done a lot of thinking about how these factors interact, but also how these knowledge gaps can be bridged by AI. Expect more to come on that, but one concrete effort I’ve considered is AI-assisted, crowd-sourced, knowledge and learning database that help thought leaders in this space to get sharper and to stay current. I’m working on identifying cross-disciplinary leaders who would be interested in engaging in such an effort. Please reach out if you are interested.

In this, our first guest post, Martha explains how her deepening understanding of the imperfections of AI helped her to reflect on her own opportunities for improvement as an educator as well as on how to partner more successfully with these powerful new tools.]

It was 10:30 that opened my eyes – not 10:30 on a random Wednesday morning, but my request that ChatGPT construct an image of an analog clock that displayed the time. But that smooth-talker, that silky-voiced ChatGPT, which had convinced at least some of us that it was on the verge of replacing human intelligence, couldn’t do it.1 No matter how many times I asked, it couldn’t get the hour hand to point to 10 and the minute hand to point to 6, something a second grader with a crayon and a piece of paper could manage with ease. This glitch, about which I learned in a recent workshop, changed my thinking about AI.2

From Panic to Pedagogy

Late in 2022, when ChatGPT first made headlines, academia seemed to lose its collective mind; the Great AI Panic of 2023 was about to begin. Some of my colleagues immediately went apocalyptic, imagining a world in which AI took over; Skynet was about to go live, they portended. Others started stockpiling blue books and reworking their assessments into the written and oral exams of years ago. A few were ready to have AI integrated, somehow, into their brains. But others stuck their heads in the sand and pretended it didn’t exist.

That was my approach; I was an ostrich. Accustomed to the glacial pace of academia, I thought I would have time for some research during my winter break or over the summer. In the meantime, I would teach as if AI didn’t exist and asked my students to buy into that approach. In those first few semesters, when AI still felt new, my students and I agreed that they would not use it, or so I thought.

That was the wrong strategy, of course. AI was too tempting for my students; it saved them time, it was easy, and it was accessible. And so, when, in my first-year writing class at a mid-sized public college, I received a paper about “democratic patenting” for an assignment about democratic parenting, I knew I had to act. I had to take my metaphorical head out of the sand and really think about generative AI and its impact not only on my classroom but also on higher education.

Early on, it was very easy to spot generative AI-produced work. ChatGPT (that’s all that students seemed to be using) was producing solidly mediocre work, C+ at best. The problems were obvious: deeply conventional language, workaday structures, and unoriginal thought. Students were offloading their cognitive work to a pattern-matching machine, which could produce prose that possessed an air of authority, if only you didn’t read too closely.

Today, the irony of my concerns isn’t lost on me. There I was, lamenting the death of originality and decrying the loss of creativity, wondering if my students would be able to write papers in distinctive voices and to present original thoughts. But I had recycled my curricular materials repeatedly, after deriving them from a handbook for college composition, which had gathered readings and suggested activities that were available to innumerable instructors, as well as websites that sold academic papers. I had created conditions that were ripe for plagiarism, for those students so inclined, and I had to do something.

Hanging on to Process

Pre-AI, I had thought that, with enough scaffolding, enough creative and engaging classroom activities, enough training and warning about plagiarism, my students wouldn’t, in fact, plagiarize. By and large, I was correct; I had very few cases of plagiarism, which were always easy to spot and address. Most of my students developed a strong, individualized writing process, which was one of my explicit goals in the course. But now, with the days of cut-and-paste plagiarism long past, I realize that, although I focused on helping students develop a viable writing process, I was expecting and accepting a product that was too formulaic and unoriginal, essays that were conventional at best, precisely because I was reusing my course material each year. The AI revolution has forced me to reckon with making sure that my own practices in teaching writing encourage even more of a focus on process, rather than product, more of a focus on academic integrity and transparency, rather than on catching plagiarism.

But the challenge of AI isn’t, of course, just plagiarism, which at its most basic is claiming work that you did not do as your own. In fact, the challenge is hanging on to process. When I teach, I ask my students to follow a model proposed by Anne Lamott in her book Bird by Bird. The idea is that everyone, except the rarest among us, has to write in drafts. That first one is the “really, really shitty first draft,” where we get our ideas down, where writing is thinking and thinking is writing, where we don’t have to pay attention to the editor whispering in our ear but can improvise and play with thought and language.3 In the age of AI, we have to figure out how we can retain that “shitty first draft,” whatever that may mean for each discipline,4 when as a recent article claimed, “everyone is cheating their way through college.”5 After all, it’s that shitty first draft – that messy, unpolished, initial encounter with a set of ideas – that allows and encourages students to think for themselves. Brainstorming – writing down one’s thoughts without judgment – is, after all, the first step for some serious cognitive work. As Ted Chiang reminds us, “Your first draft isn’t an unoriginal idea expressed clearly; it’s an original idea expressed poorly, and it is accompanied by your amorphous dissatisfaction, your awareness of the distance between what it says and what you want it to say.”6

Developing Critical AI Literacy

One way to retain an emphasis on process, when product is so tempting, is to engage in “critical AI literacy.” This could include unpacking the ethical and environmental implications of the production of large language models and their use: implications of how the datasets on which LLMs rely are collected and coded; the enormous energy demands of training and LLMs; or its potential to upend careers through elimination and speed-up.

Critical AI literacy can also be thought of as an understanding of what AI is and isn’t and how AI works and doesn’t; LLMs are fallible predictors of what humans may say. They do not think for themselves, no matter how polished their prose may become, and, yet, they may be useful. To this end, I ask my students to think through if and how they can use AI as a tool and as part of their own process, and when it’s appropriate to do so. I, along with many others, have developed activities and even our own chatbots that deploy AI as a conversational thought partner, bouncing ideas back and forth with students, helping them hone their arguments by offering counter-arguments or contradictions; as a co-researcher that may not be entirely accurate but may provide useful sources; as an editor that may help with typos but point students to somewhat generic prose; and as a tutor, always available to help students with their grammar, thesis statements, conclusions, or something else. Depending on the discipline and interest of instructors and students, this list can be even more expansive. But, no matter how AI is used in the classroom, a critical distance, a stance of skepticism has to remain, because AI can’t seem to tell time.

AI’s Fallibility, the Importance of the Human, and the Liberal Arts

Back to 10:30. AI’s inability to construct an image of an analog clock that reads 10:30, a simple request, one would think, reminds us of AI’s limits and fallibility. It reminds us that AI isn’t human, that it can’t engage in the metacognition that we prize as critical thinking. While it may be adept at creating code or producing generic lab reports, it cannot replace creativity, originality, empathy, and ethics. As AI becomes more and more integrated into academic endeavors and workplaces, we need folks that understand the latter, even more than the former.

And this is where the liberal arts will flourish. With its orientation toward analysis, ethics, empathy, and creativity, the liberal arts can foster in students that necessary skepticism about AI and, also, guide them toward an approach that deploys it effectively.

My students learn how to read closely and critically; they develop skills in making arguments, based on solid evidence. They question, test, and search for the strongest ideas when confronted with competing narratives. These aren’t skills that will become obsolete in the face of advancements in generative AI, which is, at its core, a tool that is so good at replicating and predicting word sequences that many anthropomorphize it. These are skills in thinking, which will become even more essential, as AI improves, using chatty language to mask the fact that LLMs are sophisticated pattern-generating machines, neither conscious, nor creative, nor capable of understanding or empathy. Liberal arts majors will be the ones that will remind us that, although generative AI may be adept at predicting what a human would say or write, it cannot predict what a human will think or feel, at least not yet.

The liberal arts is also the place where students will be able to develop a facility for ethical practices for its use. The humanities especially will be the place where faculty develop strategies to teach students how to use AI as the tool that it is and to remind them that AI can produce confident-sounding and plausible “hallucinations,” those invented sources that don’t exist but could, presented with the veneer of authority that the most distinguished of professors may envy.

The place of the liberal arts in an AI-inflected world is especially important right now, when the liberal arts, and especially the humanities, are presumed to lack relevance to students’ post-graduate lives and when colleges and universities have to address the career readiness of their graduates. At this moment, when the danger in AI is its uncritical or thoughtless use, it is essential to hang on to what is distinctly human, what AI is not yet or may never be able to do. The liberal arts are where critical thinking happens, where students are able to recognize the limits of what AI is good at – predicting the likelihood of common and formulaic arrangements of language and thought – and can think through ethical quandaries with empathy. Liberal arts students have the creativity to go for the unlikely and the unpredictable in pursuit of new knowledge or new expression. And this is why liberal arts graduates will flourish in AI-inflected work environments, for they possess the interest and skills, honed in years of humanities, social science, arts, and sciences classes, to maintain the human.

In the age of AI, liberal arts reminds us that we have to double down on the things that make us irreplaceably human: our ability to question, analyze, imagine, and speculate rather than just predict probable outcomes, our willingness to grapple with ethical dilemmas through the lens of our own experiences, and our impulse to create something genuinely new rather than just remixing what already exists.

• Why I took on the challenge of tuning an LLM

• My learning process

• Launching: AI Insights Sessions

• Appendix

  • Raw learning notes

  • Presentation slides (with code and narrative)

Why I took on the challenge of tuning an LLM

I took on the project of learning how to tune an agentic LLM because:

(1) I wanted to see if I could make money off of it – I had an opportunity to build a course for an online course provider.

(2) I am convinced that everyone should raise their learning expectations: You can learn more and learn faster than you used to be able to; get used to the idea. I wanted to see how far I could go.

(3) There’s great potential leverage in going deeper in AI Skills.

(4) I realized that I could write an article about the process.

My Learning Process (A Structured Retelling)

As mentioned I took notes on my learning process, and those are shared at the bottom. I used AI to help me structure the methodology I developed as I went, and then later, to find educational theorists whose thinking may have been implicitly referenced.

1. Orientation: Choosing a Learning Stake

“What’s my real motivation?”

The course provider wanted a course on Tuning Agentic LLMs; so that was a given. But I could decide what kind of tuning process I would be interested in. I had been developing my initial plan with an AI Chat tool, when I got bored with some of the suggestions. So, I changed tack. I wrote, “What can I build that would interest and motivate me?” I may even have copy-pasted in my LinkedIn profile or a recent blog post, so that the LLM would be a bit better educated on the topic.

Eventually, we worked out that I would tune an LLM to categorize the news. I won’t pretend that this is the most exciting application, but I had wondered about the idea of building my own customized news feed with AI, so this would be a learning process that I could potentially use in the future.

Read more about self-directed learning (Knowles)

2. Scaffolding: Building an Initial Framework

“What can I reuse or adapt to give shape to this? Which fixed, and comprehensible reference should I choose to anchor my thinking?”

I was very lost when I started the project, and a bit panicky. I found an hour here or there to work on the project, and was lucky to find that one service would not only ‘vibe code’ with me but would, perhaps equally importantly, draft an accompanying presentation script.

I now had a script that I more or less understood and blocks of code that I only partially understood. I had done some AI learning in the past (Andrew Ng’s courses, and a data science certificate from Texas A&M) some years ago, and still had just enough confidence in my recollection that I felt that I could anchor on the AI-generated script.

With that confidence, I wrote down a plan and estimated how long it would take me to complete each step: I could start to see a shape to the work ahead.

Read more about scaffolding (Wood, Bruner, Ross, 1976)

3. Dialogic Inquiry: Learning Through Conversational Reasoning

“What does this code actually do? What does that phrase actually mean?”

I went through the script that AI had helped me develop and added about 15 comments or questions: These were things that I didn’t really understand but which I felt I needed to understand in order to credibly teach the course.

On a subsequent evening, I asked AI about each of the unclear passages, until I felt that I understood what was going on in each piece of code and in each part of the narrative.

For example, I asked AI to explain concepts like AutoModelForSequenceClassification, F1 scores, and data_collator. And many more. Some I am withholding so as not to reveal my full ignorance!

The course was starting to feel like it was mine.

Read more about elaborative-interrogation (Pressley et al., 1992)

4. Prototyping Understanding and Debugging-in-Practice

“Let’s see what happens when I run this in a real environment.”

“Why is this argument failing? Why does this dependency break?”

I even needed to ask an AI how to even set up my console (I ended up using a Mac) for python coding.

I cannot remember the last time that I coded python on my PC; I really think that more people can do this than they think–just keep asking questions!

At some point, I hit a wall. I asked my AI coding assistant why one of the functions wasn’t working. It had me trying many things and I started to feel like its reasoning was circular or that it was asking me to try things that I had already done before.

I realized that I should check the properties of the object I was working on, in order to see which variable named were available. I saw that one of the variable names that AI had given me earlier was misspelled: I had to change evaluation_strategy to eval_strategy.

Yes, that was a technical thought process. What folks should realize, again, is that I am not a coder. But I did know enough coding concepts to come up with a question that could get me out of a rut. The Cognitive Rush came, though, from realizing that I had to stop listening to the AI and had, instead, to ask it to research something for me.

Knowing when to make this shift in framing was critical.

Read more about the constructivist learning theory (Piaget): learners build knowledge through direct interaction with environments that respond to their actions.

5. Execution

From this point on, the work became more conventional. Yes, I used AI to help organize my content for a slide presentation, and so on. But these were more familiar usages to me, by now. I was not just learning, not learning about learning.

6. Metacognitive Looping: Indexing, Reflecting, Re-entering

“Where should I start next time?”

I’m writing this article for whatever reasons I have a blog as well as to reflect on my own AI-assisted learning process. Beyond that, though, I foresee a day in which I will be able to encode a personalized learning process into an AI Assistant, so that I can learn how and when it is more comfortable for me. Drafting this structure today is pushing me along that path.

Cognitive principle:
Read more about metacognition–thinking about your thinking (Flavell, 1979).

AI Insight Sessions

If you're working to navigate learning, strategy, or decision-making in the age of AI, this form is a place to get clear on what matters most.

You'll be asked a few focused questions to help you:

• Clarify your current challenges and goals

• Reflect on what's working — and what's not

• Contribute to an ongoing project on how people are adapting to AI

At the end, you’ll have the option to book an AI Insight Session — a one-on-one conversation designed to help you:

• Make faster progress

• Think more clearly about learning and direction

• Identify what's holding them back

I only book calls with people who’ve taken a moment to reflect first — it leads to sharper, more useful conversations.

Get started

(Note: I read every response personally and prioritize those where I believe I can be most helpful. Thanks for your patience.)

Appendix

• Tuning LLMs Slide Show (with code and narrative)

• Learning and Planning Notes

For those new to the Tao of AI, welcome! Posts at the Tao of AI will generally alternate between technically-oriented posts and philosophically-oriented ones. One of the key contentions, or conceits, of this Newsletter is that there is a recursive interaction between thinking about, and using, “AI as a Tool” and “AI as a thought partner”. The first is about the technical relationship between a human and an implement; the second is about the human relationship between internal and external (read: AI-generated) “thoughts”.

Navigating the complexities of the AI age requires understanding not just technology, but its likely profound impact on our ideas, societies, and institutions. Today, we’ll take the founding of the United States as a case study in how shifting, or emerging, ideas and ideals can be leveraged to build robust systems that promote stability and prosperity. We’ll then look at how those ideals may shift in the Age of AI and will conclude with initial, practical recommendations about how to “operationalize” them, for public and private benefit.

I. Foundational Beliefs: The Enlightenment's Vision of the Individual and Government

This is the ‘ideas’ part of our case study. The Founding Fathers of the United States leaned, implicitly and explicitly, on the conception of the rational–or at least, independent–man who was best-placed to determine his own interests. The conception of limited government flowed, in large part, from this ideological underpinning.

First, let’s focus on the types of ideas that influenced the elite of this era. We’ll then look at how those ideas permeated and influenced the common people of the time.

Enlightenment Elites: The Philosophical Vision of Human Nature

The thinkers of the Enlightenment profoundly shaped the foundation of the United States. Their core belief centered on the individual's capacity for reason and independent thought. John Locke, in his Second Treatise of Civil Government, argued that "reason... teaches all mankind... that being all equal and independent, no one ought to harm another in his life, health, liberty, or possessions." This idea of inherent rights and the power of individual reason was fundamental for the Founding Fathers.

Montesquieu, in Spirit of Laws, emphasized the importance of "virtue" in a republic, defining it as "the love of one's country, that is, the love of equality." He saw this "political virtue" – driven by human judgment and reason – as the essential force that "sets the republican government in motion." The behaviors and proclivities of ordinary people were an essential ingredient to the success of the republic; they must exercise independent self-government.

James Madison, in The Federalist Papers, No. 10, while discussing the challenges of factions, also affirmed the significance of individual discernment. He stated that "the diversity in the faculties of men, from which the rights of property originate... [is] an insuperable obstacle to a uniformity of interests. The protection of these faculties is the first object of government." Madison believed that protecting these individual abilities to reason and act independently was a primary purpose of government.

This emphasis on the discerning, rational, and independent individual was a common thread in the works of other influential thinkers of the time, including Voltaire, Adam Smith, and the authors of the Declaration of Independence. The Founding Fathers and their intellectual peers operated with a view of humanity that acknowledged individuals would act in their own self-interest. Their goal was to create a system that could balance, guide, and, when necessary, restrain these natural tendencies–but which would not seek to eliminate or ignore them.

Beyond the Elite: How Enlightenment Ideas Took Root in Everyday American Life

The philosophical ideas of the Enlightenment weren't confined to intellectual circles; they influenced the self-perception and habits of ordinary Americans, which were crucial for the stability of the new nation. Alexis de Tocqueville, widely understood as the keenest observer of early American society, provides invaluable insights into how these broad ideas resonated in daily life. His observations in Democracy in America reveal a populace actively practicing discernment and engaging in behaviors that mirrored the core tenets of Enlightenment thought.

Alexis de Tocqueville

Tocqueville noted the vital role of local participation, stating that "Town meetings are to liberty what primary schools are to science; they bring it within the people’s reach, they teach men how to use and how to enjoy it." He argued that while a free government could be established on paper, "without the spirit of municipal institutions it cannot have the spirit of liberty." This highlights how direct involvement in local governance instilled a practical understanding and appreciation for freedom.

Famously, Tocqueville was struck by the American propensity for association: "In no country in the world has the principle of association more successfully used... than in America." He observed that beyond formal government structures, "a vast number of others are formed and maintained by the agency of private individuals." This active participation in voluntary organizations demonstrated a citizenry comfortable with self-reliance and collaborative action, embodying the Enlightenment emphasis on individual agency and the power of collective reason. As Tocqueville put it, "The citizen of the United States is taught from his earliest infancy to rely upon his own exertions... he looks upon social authority with an eye of mistrust and anxiety."

Even in their understanding of their rights and laws, ordinary Americans displayed a practical, rather than purely theoretical, grasp. Tocqueville recounted that "if you question [the American] respecting his own country . . . his language will become as clear and as precise as his thoughts. He will inform you what his rights are, and by what means he exercises them . . . The American learns to know the laws by participating in the act of legislation; and he takes a lesson in the forms of government from governing." This active engagement in civic life, rather than relying solely on books, fostered a deep and practical understanding of their rights and the mechanisms of their government.

Tocqueville's vivid account reveals that the Enlightenment ideals weren't abstract theories confined to the elite, but rather deeply ingrained principles actively practiced by ordinary Americans. This widespread embrace of reason, self-reliance, and civic participation formed a crucial, often overlooked, bedrock upon which the ambitious project of the United States would be built, demonstrating the potent force of shared intellectual and societal foundations.

II. From Ideals to Action: Hamilton's Operationalization of Enlightenment Ideals

The lofty ideals and societal habits of the Enlightenment, while crucial, were not self-sustaining. The true genius of the American experiment lay in its translation of these philosophical underpinnings into tangible structures. It was Alexander Hamilton who spearheaded this critical phase of "operationalization," recognizing that a robust nation required not just guiding principles, but also a practical economic framework that could support and reinforce them.

This framework had to complement, rather than quench, the social and intellectual forces that animated the early republic. Hamilton worked to ensure that the United States would be prosperous, but continually spoke and thought in a language of the balancing of interests and personal calculations, an approach much in line with the political zeitgeist and social conception discussed above.

Hamilton's vision took concrete form in a series of influential reports delivered between 1790 and 1791. Let's examine three key documents:

First Report on Public Credit (1790)

In this report, Hamilton championed the creation of a national debt, arguing that it would forge a vital link between individual prosperity and national stability. He argued that by investing in the nation's debt, individuals would develop a vested interest in its success, thereby stimulating economic activity and bolstering national unity. As Hamilton asserted, establishing sound public credit was essential "to justify and preserve their confidence; to promote the increasing respectability of the American name . . . to cement more closely the union of the states; to add to their security against foreign attack; to establish public order on the basis of an upright and liberal policy." This strategic move aimed to transform individual financial pursuits into a source of national strength.

Report on a National Bank (1790)

Hamilton next advocated for the establishment of a National Bank, a proposal considered then and now as a significant step towards centralizing financial power. However, his rationale revealed a deep understanding of the republican system, emphasizing both the importance of public opinion and the energizing force of decentralized, self-interested activity. Recognizing the need for public trust, Hamilton noted that "Public opinion being the ultimate arbiter of every measure of Government, it can scarcely appear improper . . . to accompany the origination of any new proposition with explanations . . ." More significantly, he argued against direct government control of the bank, stating that "it appears to be an essential ingredient in its structure, that it shall be under a private not a public Direction, under the guidance of individual interest, not of public policy." For Hamilton, the "keen, steady . . . sense, of their own interest, as proprietors, in the Directors of a Bank . . . is the only security, that can always be relied upon, for a careful and prudent administration." This pragmatic approach, grounding a national institution in the driving force of individual self-interest, exemplifies his strategy of aligning personal motivations with national goals.

Report on the Subject of Manufactures (1791)

In his final major report, Hamilton emphasized the importance of a diversified economy in stimulating individual ingenuity and national wealth. He argued that confining individuals to limited pursuits stifled their potential, observing that "minds of the strongest and most active powers for their proper objects fall below mediocrity and labour without effect, if confined to uncongenial pursuits." Therefore, he concluded, "the results of human exertion may be immensely increased by diversifying its objects," and that "to cherish and stimulate the activity of the human mind, by multiplying the objects of enterprise, is not among the least considerable of the expedients, by which the wealth of a nation may be promoted." Furthermore, Hamilton believed that a complex economy, fostering interdependence, would strengthen the social and political fabric of the nation. He contended that "Mutual wants constitute one of the strongest links of political connection, and the extent of the[se] bears a natural proportion to the diversity in the means of mutual supply." By creating a web of "intimate connexion of interest" among individuals, Hamilton aimed to foster a more stable and unified society, less susceptible to "solicitudes and Apprehensions which originate in local discriminations."

Arkwright Water Frame, c.1775

III. The Cognitive Shock: AI and the Future of the Individual

The rise of artificial intelligence presents a subtle, but profound, challenge to the Enlightenment notion of the autonomous, reasoning individual. AI's capacity to analyze, predict, and even simulate human behavior disrupts the assumption of human intellectual monopoly. Algorithms shape our information consumption, influence our choices, and generate content that blurs the lines between human and machine creativity.

Humanity has navigated cognitive shifts before. The emergence of art and storytelling, as Yuval Noah Harari describes, marked a moment where ideas gained traction beyond individual minds: “Yet the truly unique feature of our language is not its ability to transmit information about men and lions. Rather, it’s the ability to transmit information about things that do not exist.” (from Chapter 2 of Sapiens). The invention of writing further externalized language, democratizing access to thought across time and space

These earlier "cognitive expansions" share with the rise of artificial intelligence the characteristic of externalizing thought and information. However, AI introduces a fundamental difference: a perceived autonomy in the generation and application of thought. While art and writing are clearly human creations, AI presents the sensation of intelligence operating independently, blurring the lines of authorship and intent. This distinction delivers a unique "cognitive shock" to the Enlightenment ideal of the autonomous, reasoning individual, a cornerstone of democratic systems. Unlike previous externalizations that ultimately remained tools of human expression, AI challenges the very notion of human intellectual uniqueness and agency in decision-making, exacerbating concerns already present in the digital age” These concerns take on a new dimension when the source of suggestion appears to be a non-human, intelligent entity.

IV. Building Institutions for the AI Age: A Hamiltonian Approach

The "cognitive shock" of AI, as discussed, challenges fundamental assumptions about individual reasoning and autonomy, potentially undermining the philosophical underpinnings of our existing institutions (If the individual is so inconstant or unreasoning, why should one believe in our democratic system at all? If people are fundamentally manipulable and inconstant, then what use is there in building civic virtue and/or basing government on the assumption that individuals generally will know what is best for themselves?). To navigate this new reality and harness AI for societal benefit, we must ultimately build new frameworks and adapt existing ones. In my post about Partnering with AI to Reimagine Problem-Solving, I made an attempt at articulating how we might reimagine ourselves in the Age of AI. Today, I will focus on practical recommendations that could serve to harness an altered sense of the individual, while also promoting social stability, freedom, and prosperity.

To be sure, these are lofty endeavors, but I don’t feel that the challenge can be ignored. And I certainly hope that the practical proposals below suggest the type of attitude we should bring into the AI Age:

a) AI for Community Building and Collaboration

Developing Community-Managed AI Ecosystems:

• Proposal: Institutions with strong internal communities, such as professional organizations (e.g., the DC Bar), universities (e.g., the University of Michigan alumni association), or even large companies, could invest in training and/or fine-tuning their own AI models. These models would be specifically designed to facilitate communication, provide tailored advice, and enhance collaboration among their members.

• Benefits: This approach fosters a sense of ownership and trust in AI tools within a defined community. For example, a bar association could develop an AI to help members navigate ethical dilemmas or find relevant case law, while a university alumni network could create an AI to connect members with shared interests or career opportunities.

• Hamiltonian Connection: This mirrors Hamilton's focus on building national institutions that served specific needs and fostered a sense of shared identity and purpose.

b) AI for Trust and Accountability

Establishing Trust Frameworks for Collaborative AI Development:

• Proposal: Explore the development of trust models, potentially leveraging cryptographic technologies, to enable secure and verifiable contributions to AI model development from distributed participants. For instance, the Library of Congress could initiate a project to build an AI model capable of interacting with its vast literary collection. Verified authors and subject matter experts could contribute to the model's training data and knowledge base through a transparent and auditable system.

• Benefits: This approach allows for the collective intelligence of diverse stakeholders to contribute to AI development while maintaining accountability and mitigating the risks of malicious or biased data.

• Hamiltonian Connection: This reflects Hamilton's understanding of the need for mechanisms that inspire confidence and encourage participation in national endeavors, even when those participants are geographically dispersed.

Promoting Community-Driven AI Evaluation and Transparency:

• Proposal: Implement mechanisms to make AI models more legible and interpretable to the general public. This includes developing user-friendly explanations of the logic and processes driving model outputs, establishing public forums for discussing model performance and biases, and creating feedback loops that incorporate community input into model refinement.

• Benefits: Increased transparency and public engagement in AI evaluation can build trust, identify potential harms, and ensure that AI systems are aligned with community values.

• Hamiltonian Connection: This echoes Hamilton's recognition that "public opinion being the ultimate arbiter of every measure of Government," necessitates clear explanations and opportunities for public understanding.

Establishing Processes for Community Governance of AI Systems:

• Proposal: Develop and implement mechanisms for community input and oversight in the development and deployment of AI systems that significantly impact public life. This could involve citizen advisory boards, participatory budgeting for AI initiatives, or frameworks for addressing algorithmic bias and ensuring fairness.

• Benefits: Community governance can help ensure that AI systems are developed and used in a way that aligns with democratic values and serves the public good.

• Hamiltonian Connection: This, in a modern context, reflects the underlying principle of republicanism where the ultimate authority resides with the people and their participation in governance is essential.

c) AI for Broadening Access and Participation

Creating Accessible Pathways for AI Integration and Contribution:

• Proposal: Develop accessible training programs, user-friendly tools, and intuitive interfaces that empower "non-technical" individuals to actively participate in "AI-integrated" community activities. This could involve workshops teaching citizens how to provide effective feedback on AI model outputs, platforms that allow domain experts to easily contribute labeled data, or citizen science initiatives that leverage AI for analysis.

• Benefits: Broadening participation ensures that AI development and deployment are not solely the domain of technical experts, fostering greater public understanding and buy-in.

• Hamiltonian Connection: This aligns with Hamilton's belief in the importance of stimulating the minds and activities of all citizens for the overall prosperity and strength of the nation.

Building Shared Knowledge Bases for Public Understanding of AI:

• Proposal: Create publicly accessible repositories of non-proprietary knowledge and documentation related to AI models and their applications within various domains. These resources could include tutorials, code examples, best practices, ethical guidelines, and case studies. While these knowledge bases may not always be cutting-edge, they serve the crucial public interest of fostering broader AI literacy. This also benefits tech companies by promoting wider engagement with AI technologies.

• Benefits: Shared knowledge bases empower the public to "get smarter" about AI, fostering informed discussions and facilitating wider adoption of beneficial AI tools.

• Hamiltonian Connection: This mirrors Hamilton's belief in the importance of providing the nation with the resources and information necessary for its progress and development.

d) AI for Economic Development and Innovation

Fostering Domain-Specific AI Innovation Hubs:

• Proposal: Encourage and support the formation of groups that bring together individuals with deep interest or expertise in specific domains (e.g., healthcare, education, climate science, law) to explore and develop AI applications within their fields. These hubs should function as modern "innovation hubs," connecting experts, practitioners, and community members to address specific challenges and opportunities. I expect that there are existing examples of such domain-specific AI innovation hubs that readers are aware of. Please share in the comments.

• Benefits: Focused collaboration within specific domains can accelerate the development of practical and impactful AI solutions tailored to real-world needs.

• Hamiltonian Connection: This reflects Hamilton's emphasis on diversifying the objects of enterprise to stimulate innovation and economic growth in various sectors.

Cultivating AI-Partnered Initiatives for Economic Advancement:

• Proposal: Establish clear goals for driving specific economic outcomes through AI and develop cross-industry and/or public-private partnerships to build and organize AI-partnered initiatives. For example, a collaboration between agricultural businesses, data scientists, and government agencies could focus on developing AI tools to optimize crop yields and resource management.

• Benefits: Targeted initiatives can directly translate AI capabilities into tangible economic benefits and create new opportunities for collaboration and growth.

• Hamiltonian Connection: This directly aligns with Hamilton's vision of a robust economic system founded on the "mutual wants" and interconnected interests of various stakeholders.

By embracing a proactive and pragmatic approach to building institutions for the age of AI, we can navigate the "cognitive shock" and harness the transformative potential of this technology for a more prosperous and equitable future. logic and process driving model outputs. Make this ‘feedback cycle’ part of public discourse.

V. Concluding Thoughts: Navigating an AI-Altered Future

While our fundamental human needs for connection, sustenance, and purpose will likely persist, the ways we meet those needs, and indeed our very aspirations, are poised for significant transformation by artificial intelligence. AI usage may intiially be focused on optimization of existing activities: “Where can I find a good bagel?", "How can I build better marketing material?", or "Help me plan my trip to Hawaii?" but as AI capabilities deepen, we must anticipate more profound shifts. Will our educational paths and career trajectories be fundamentally reimagined, intentionally integrating AI as a partner? Will we develop ongoing "relationships" with AI coaches that possess an intimate understanding of our needs and goals? Could we see the rise of collaborative initiatives among friends, guided by a knowledgeable AI moderator? And as these AI-mediated experiences evolve, will they spark entirely new aspirations – perhaps altering our travel desires or even the institutions we choose for learning? The ways we work and the dynamics within our teams and professional circles are undoubtedly on the cusp of significant change.

At present, a noticeable, sometimes yawning, gap separates those who view AI with apprehension, primarily focusing on its ethical and human risks, and those who are enthusiastically exploring its commercial possibilities. A robust dialogue should bridge this divide. Those concerned about AI's broader impacts must actively seek to understand its mechanisms, enabling them to pragmatically influence its application while acknowledging its permanence and the necessity of thoughtful and effective operationalization. Simultaneously, those focused on AI's commercial potential must recognize that a singular focus on profit may prove insufficient in the face of the rapid social and political evolution that AI is likely to instigate.

A Call to Action: Engage, Reflect, and Shape Our AI-Integrated World

To navigate this evolving landscape and build a future where AI empowers rather than undermines our values, I encourage readers to actively participate in bridging this critical divide:

For the Liberal Arts-First "AI Skeptic":

• Dedicate thirty minutes weekly to understanding AI models: Explore their development, refinement, and real-world applications. Don't hesitate to use AI itself as a learning tool to accelerate your comprehension.

• Actively integrate your unique expertise into your AI learning: Connect your knowledge of law, literature, art, social work, etc., to the challenges and opportunities presented by AI. This interdisciplinary approach will not only enhance your understanding but also position you to contribute unique and crucial insights to our increasingly AI-centric world.

For the Tech (and/or Business)-Centric "AI Engager":

• Commit thirty minutes weekly to reflecting on enduring values and institutions: Consider which principles, practices, and societal structures you hope to carry forward into the AI Age. Document your reflections and discuss them with others to begin developing strategies for their preservation and adaptation.

• Invest another thirty minutes each week in analyzing AI's potential impact on political structures and behaviors: This broader perspective will provide a more comprehensive understanding of the evolving landscape, revealing a wider spectrum of both economic opportunities and potential risks as you develop your own ventures and strategies.

Your engagement is crucial. By actively exploring AI's intricacies and thoughtfully considering its profound implications, we can collectively shape its trajectory and build institutions that not only harness its transformative power but also safeguard the fundamental values of a thriving society. Your participation in this ongoing conversation is not just welcomed – it is essential to navigating the future we are building together.

I look forward to your feedback.

• For AI practitioners (a subset of Yang readers): To optimize your AI projects, you need to be able to speak the same language as your infrastructure specialists. This guide will equip you with the knowledge to understand the key components of cloud infrastructure, make informed decisions about your AI workloads, and collaborate effectively with your infrastructure team.

• For Yin Readers: Copy-paste this article into an LLM Chat Tool and ask:

  • “Please summarize this article. Focus on 2-3 takeaways regarding GPU development that will help me keep an eye on the development and relevance of this critical piece of computing infrastructure that I know little about. Add an additional takeaway or two regarding the rest of the content in the article. Please make some suggestions about how and where these trends might impact my field of <insert your field here.>”

• For any reader: Copy-paste this article into an LLM Chat Tool and ask:

  • “Tell me a short story about the content of the article. help me build a mental map of the key takeaways so that i can start to follow trends in these infrastructure components"

  • [Optionally] “Help me understand how these trends relate to my field of <insert field here>.”

Now, back to the topic at hand.

I would argue that some knowledge of technical infrastructure, even at a high-level, should be part of the “liberal arts education” of the future or, if you prefer, part of every AI engineer’s, data scientist’s, or even average educated person’s, foundational knowledge. The cloud focus here is, in part, an artifact of where I spend my time. But I also think that it’s reasonable to focus on cloud infrastructure as a proxy for the generalized AI infrastructure of the future, since the average Market Newcomer is likely to use hyperscaler capabilities and/or tools, for the easily foreseeable future.

We’ll go through each critical domain for cloud computing (compute, storage, networking) and highlight the key investment areas and/or development bottlenecks that you should track (even lightly) over the next few years. We'll also discuss how these areas work together and how they are brought together by AI frameworks.

Compute: The Foundation of AI

Compute is obviously where the action is, and the shift in relative value between GPUs and CPUs, as perhaps the key trend over the past half-decade or more, is likely to generate (further) shifts in market structure and developer focus. Compute provides the raw processing power for AI workloads, but its effectiveness is heavily dependent on the efficiency of storage and networking, as we'll see in later sections.

Staying alert in this market means (at least) paying attention to the latest generation of GPUs, as well as their principal characteristics, (e.g. memory size, memory bandwidth, core count), even if at a high-level.

We’ll break the Compute section into two. If you’re already familiar with the basic types of processors and their relative strengths and weaknesses, feel free to skip to Compute, Part II.

Compute, Part I: Processor Type Overview

CPUs excel at sequential processing with a small number of powerful cores optimized for diverse instructions and low-latency memory access, making them ideal for general-purpose tasks.

GPUs, with thousands of simpler cores and high memory bandwidth, are designed for parallel processing, accelerating computationally intensive tasks like AI training and graphics rendering.

TPUs, custom-designed by Google solely, are purpose-built for the matrix multiplications at the heart of neural networks, sacrificing general-purpose flexibility for extreme AI-specific performance.

DPUs specialize in data management, accelerating network operations, storage access, and security tasks to optimize data flow for large AI systems.

FPGAs (Field-Programmable Gate Arrays) provide a flexible hardware fabric that can be customized for specific workloads, bridging the gap between general-purpose processors and specialized ASICs (Application-Specific Integrated Circuits).

IPUs (Intelligence Processing Units) are optimized for graph processing and sparse data structures, which makes them a good fit for analyzing interconnected data in applications like social networks and recommendations.

These architectural differences dictate each processor's strengths. CPUs’ few, complex cores handle diverse workloads sequentially. GPUs' high core count and high memory bandwidth enable parallel processing for compute-heavy tasks. TPUs' specialized design prioritizes matrix multiplication speed for AI. DPUs streamline data movement with specialized hardware. FPGAs' reconfigurable logic allows them to be tailored for specific algorithms, offering a balance of performance and flexibility. IPUs–still niche–efficiently support the analysis of complex relationships within graph data with their unique architecture.

Compute, Part II: Key Trends, by Chip Type

GPU (Graphics Processing Unit):

GPU are typically evaluated, at the top level, according to the number of Floating Point Operations (FLOPS) supported, the amount of memory and memory bandwidth they contain, and their interconnect capabilities (for GPU-to-GPU communication; more on that below).

Nvidia is, of course, the market leader here. Their newest productized architecture, Blackwell, was officially announced in 2024 and includes (a) dual reticle-sized dies (where two silicon pieces are connected to make a single functional unit) and (b) twice the memory bandwidth (8TB/s vs 4TB/s) of the Hopper Architecture, and (c) an enhanced Transformer Engine. The Transformer Engine, introduced with the Hopper Architecture in 2022, allows for the reduction of precision when model loss is small and points to another key feature in GPU architecture, which is support for mixed precision calculations, which came with the Volta architecture, first announced in 2013 and productized in 2017.

Several other companies make GPUs, including AMD, Intel, Xilinx, and Qualcomm. This article addresses some expected future trends in GPU development, across the broader field.

TPU (Tensor Processing Unit): The TPU was developed specifically for Machine Learning/AI. They are built around a Matrix Multiplication Unit (MXU), designed to accelerate the matrix multiplications that are critical for neural networks. They also use a ‘systolic array’ architecture, in which data flows through the chip in ‘waves’, reflecting the calculation pattern required to efficiently support the targeted operations (but not necessarily efficient for other operations). Finally, they are generally built to work with reduced precision calculations.

Trillium is the 6th generation TPU and was introduced in 2024. It doubles the high bandwidth memory capacity and bandwidth compared to the previous generation. It also doubles the interconnect bandwidth (Interchip Interconnect), for communication between chips in a TPU pod and 67% more energy efficient than the previous generation. The interconnect gains are leveraged to build larger “TPU pods” where hundreds of TPUs are connected to work together, a process similar to GPU trends described above.

Finally, improvements were made in the Sparse Cores, which have a different architecture than other TPU cores, and are focused on making the processing of model embeddings more efficient. Research suggests that these cores benefit from different data flow patterns than the systolic processing described above. Other specialized data structures and operations help to accelerate the calculation of sparse matrices typical for embeddings.

CPU (Central Processing Unit):

GPUs and specialized accelerators like TPUs rightfully take the spotlight for AI, butCPUs are also undergoing significant innovation that's impacting the AI landscape. Here are some key trends:

1. Enhanced AI Instructions:

• Specialized instructions: Modern CPUs are incorporating specialized instructions to accelerate common AI operations like matrix multiplication, convolution, and other mathematical functions. These instructions can significantly improve the performance of AI inference on CPUs.

• Example: Intel's latest Xeon CPUs include Advanced Matrix Extensions (AMX) that accelerate matrix operations, to support deep learning.

2. Optimized Memory and Caches:

• Larger caches: CPUs are being designed with larger (data) caches to improve data access for AI workloads. This reduces the need to fetch data from main memory, which can be a bottleneck for AI performance.

• High-bandwidth memory: Some CPUs support high-bandwidth memory (HBM), which offers significantly faster data transfer rates compared to traditional memory. This is crucial for AI tasks that require large amounts of data to be processed quickly.

3. Software Optimization:

• AI-optimized libraries: Software libraries are being developed to optimize AI performance on CPUs. These libraries leverage the specialized instructions and hardware capabilities of CPUs to accelerate AI workloads.

• Examples: Intel's oneDNN library optimizes deep learning performance on Intel CPUs. Ampere’s AI-O “software library to help companies shift code from GPUs to CPUs”.

4. Heterogeneous Computing:

• CPU-GPU collaboration: CPUs are increasingly being used in conjunction with GPUs and other accelerators in heterogeneous computing environments. The CPU handles tasks like data preprocessing and control flow, while the GPU or accelerator focuses on computationally intensive tasks.

• Unified memory: This technology allows CPUs and GPUs to share the same memory space, reducing data transfer overhead and simplifying programming.

5. Edge Computing:

• Efficient inference at the edge: CPUs are well-suited for AI inference at the edge due to their lower power consumption and versatility. This is driving innovation in CPUs for edge devices like smartphones, IoT devices, and embedded systems.

• Example: Arm's CPUs are widely used in edge devices and are being optimized for running smaller AI models efficiently.

DPU (Data Processing Unit): Data Processing Units are emerging as essential components for managing the massive datasets used in modern AI applications. They offload data-related tasks, including network traffic management, storage optimization, and security functions, from the CPUs and GPUs. This frees up the primary processors to focus on the core AI computations, leading to improved overall system performance, particularly in large-scale AI deployments where data bottlenecks can be a significant issue.

Nvidia summarizes the DPU’s architectural components as follows:

1. An industry-standard, high-performance, software-programmable, multi-core CPU, typically based on the widely used Arm architecture, tightly coupled to the other SoC components.

2. A high-performance network interface capable of parsing, processing and efficiently transferring data at line rate, or the speed of the rest of the network, to GPUs and CPUs.

3. A rich set of flexible and programmable acceleration engines that offload and improve applications performance for AI and machine learning, zero-trust security, telecommunications and storage, among others.

IPU (Intelligence Processing Unit): Intelligence Processing Units are carving out a specialized niche in the AI landscape by focusing on the acceleration of graph-based AI. They excel at accelerating Graph Neural Networks (GNNs), which are employed in a variety of applications, including social network analysis, recommendation systems, and even drug discovery, thanks to their architecture specifically designed to handle the complexities of graph data.

FPGA (Field-Programmable Gate Array): FPGAs have a long history, dating back decades. They are re-programmable circuits that can be configured to support particular applications. They can be customized for AI applications and typically consume less power than GPUs, but they require significant upfront programming, which is not everyone’s cup of tea. Tech Target has created a simple guide to help determine whether an FPGA could work for your AI needs.

Storage: Fueling AI with Data

Storage provides the fuel for AI, holding the massive datasets that are used to train and run AI models. However, storage performance can often be a major bottleneck in AI workflows. Slow storage can starve compute resources, leading to inefficient training and inference. Therefore, understanding storage characteristics and optimization techniques is critical for AI professionals.

We’ll focus on areas of innovation and/or bottlenecks likely to impact AI Systems development.

Performance Bottlenecks: Slow storage can severely impact AI training, especially with large datasets that need to be accessed quickly.

• Interconnect Speed: The speed of the connection between storage and compute (e.g., the network bandwidth) can limit data transfer rates and create a performance bottleneck. This ties back directly to the networking technologies discussed in the previous section.

• Metadata Operations: Excessive metadata operations, often caused by many small files or complex compute patterns, can strain the storage system and slow down data access. This is a common challenge with large datasets consisting of many individual files.

• Storage Tiering: Moving data between slower and faster storage tiers (e.g., from cold storage to hot storage) can introduce latency and impact AI training performance. While tiering can be cost-effective, it requires careful management to minimize data movement overhead.

• Storage-Framework Interaction: Inefficient interaction between the storage system and the AI framework (e.g., inefficient data loading) can impact performance. Frameworks often provide tools and techniques for optimizing data loading, and understanding these is crucial.

Optimizing Storage

Cloud providers are addressing these challenges via:

• High-performance storage systems: Optimized for low latency and high throughput, using technologies like NVMe drives and high-speed networking.

• RDMA and GPU Direct Storage: Enabling direct access to storage from GPUs, bypassing the CPU and reducing data transfer latency. This directly relates to the networking technologies discussed earlier.

• Tiered storage: Offers a balance of performance and cost through tiered storage, where data is placed on different tiers based on access frequency and performance requirements.

Networking: The Backbone of AI

Networking is the circulatory system that connects compute and storage. Efficient networking is absolutely critical for AI workloads, especially those involving distributed training across multiple GPUs or TPUs, or those dealing with massive datasets that need to be quickly accessed from storage. Slow or congested networks can create bottlenecks that severely limit the performance of your AI applications.

This section highlights the key areas in which networking-related innovation is likely to impact the success of AI systems, and of AI-Systems-in-the-Cloud, in particular.

Inter-GPU Communication: Efficient communication between GPUs is critical for large-scale AI models that require multiple GPUs to work together.

• NVLink: NVIDIA's high-bandwidth, low-latency interconnect technology allows GPUs to communicate directly with each other, bypassing the PCIe bus and significantly reducing latency. This is crucial for efficient data and model parallel training.

• Infiniband: A high-performance networking standard commonly used in HPC clusters. Infiniband provides low latency and high throughput, making it suitable for communication between GPUs in a distributed AI training environment.

GPU to Storage Communication: Accelerating data transfer between storage and GPUs is essential for efficient AI training, especially with large datasets, and is directly related to storage performance.

• RDMA (Remote Direct Memory Access): RDMA allows network devices to transfer data directly to or from the memory of a GPU without involving the CPU. This reduces latency and improves throughput for data-intensive AI workloads.

• GPUDirect: GPUDirect technologies enable GPUs to bypass the CPU and communicate directly with other devices, such as network interface cards (NICs) or storage devices, reducing latency and improving overall system performance.

Network Offloading: Offloading network-related tasks from the CPU to specialized hardware frees up CPU resources for AI computations and improves network efficiency.

• SmartNICs: Programmable network interface cards that can offload tasks like network virtualization, security (e.g., encryption/decryption), and storage access. This reduces the processing burden on the CPU and improves network performance.

• DPUs (Data Processing Units): More powerful than SmartNICs, DPUs are dedicated processors designed to offload and accelerate data-intensive tasks, including networking, storage, and security functions, and even some AI inference tasks (see the compute section for more details).

AI Frameworks: Bridging Models and Infrastructure

Modern AI development relies heavily on powerful frameworks that streamline model building, training, and deployment. These frameworks are often optimized for specific hardware, particularly TPUs and GPUs, to maximize performance. Here's a breakdown:

1. JAX:

• JAX excels at large-scale models and distributed training. Its functional programming paradigm simplifies parallelization, and its XLA compiler optimizes code for various hardware backends.

• JAX is particularly well-suited for TPUs. Its functional programming approach aligns well with the architecture of TPU pods, and XLA's efficient compilation is essential for maximizing TPU performance. This makes JAX a strong choice for training very large models on TPUs.

• While JAX performs well on GPUs, especially with XLA optimizations, its GPU performance may not always match highly optimized PyTorch or TensorFlow implementations. XLA still plays a key role in optimizing GPU execution, but other frameworks often have more mature GPU-specific features.

2. PyTorch:

• PyTorch is known for its ease of use, large community support, and strong performance, especially in research and development. It offers a familiar API and a rich ecosystem of tools and libraries.

• PyTorch's integration with TPUs is facilitated through the PyTorch/XLA bridge. Nonetheless, JAX often demonstrates superior native TPU performance. Efficient data loading from storage to TPUs is a crucial consideration when using PyTorch on TPUs.

• PyTorch offers excellent performance on GPUs and is a popular choice for GPU-accelerated deep learning. For very large models, optimizing data parallelism and model parallelism across multiple GPUs (and potentially nodes) is critical. This optimization often involves leveraging high-speed interconnects like NVLink and Infiniband for efficient inter-GPU communication.

3. TensorFlow:

• TensorFlow is a mature framework with a large ecosystem and strong production focus. TensorFlow 2.x introduced a more user-friendly API, making it easier to learn and use.

• TensorFlow is designed to work seamlessly with TPUs, offering tight integration and optimized performance. Understanding data input pipelines and TPU-specific optimizations is essential for maximizing TensorFlow's performance on TPUs.

• TensorFlow provides good performance on GPUs and benefits from its mature ecosystem. TensorFlow may take longer for training jobs, while using less memory (though I am still learning about these dimensions for these chips). Similar to PyTorch, scaling TensorFlow across multiple GPUs often involves optimizing data parallelism and model parallelism, which relies on efficient networking and storage access.

Your choice of framework impacts not only programming style and community support, but also how effectively your AI workload utilizes the infrastructure. For example, if you're training a massive model on TPUs, JAX might offer better performance due to its optimized compiler. Conversely, if you're working with a more moderate-sized model on GPUs, PyTorch's ease of use and strong community support might make it a better choice, provided you pay attention to how it scales with your data and model size.

Bringing it home: Edge-GNN use case

We’ve explored the interplay between AI and cloud infrastructure, highlighting key areas like compute, storage, and networking, and how they are orchestrated by AI frameworks.

Let’s briefly take a concrete example, to conclude this piece: consider “Edge-Based Graph Neural Networks” (Edge-GNNs), where "edge" refers to features associated with the connections between nodes in the graph (more on Graph Neural Networks here). These additional features provide richer information for the GNN, enabling more accurate and nuanced analysis. However, they also introduce computational and storage challenges. The inclusion of edge features increases the dimensionality of the input data, impacting the memory footprint and processing requirements. This directly relates to infrastructure considerations. For example, larger memory capacity and higher memory bandwidth become crucial for handling the increased data volume [NB: “edge” is used in two ways in the linked article]. Also, the computational complexity of processing edge features might require specialized hardware or software.

[NB: I’m including the below example, not because I know it is correct, but because it brings together an evocative and (relatively) easily understandable example, for thinking about the nuances for optimizing processor and infrastructure choices to support a specific AI model. I’ve started to research the below claims, to validate whether they are true. But I found the task a bit overwhelming. I couldn’t bring myself to just delete the story, since I find it so engaging, and since it may “scaffold” some of my future research on this topic.]

“Choosing the right hardware for your edge GNN depends on the specific task. Consider a simple GNN predicting friendships. Each person (node) has features like age and interests, and each friendship (edge) has a "connection strength" score. For compute, if your GNN mostly combines person features and connection strength for predictions, a GPU's flexibility is helpful. However, for GNNs with lots of complex math on this data, and if battery life is a concern, a TPU might be better, provided you can find tools that handle the connection strength data efficiently. Storage is crucial because GNNs often access data irregularly, like jumping between friends. So, how you store and access the person and connection data impacts speed. Finally, networking is less critical if the whole friendship network is on one device. But, if the network spans multiple devices or connection strengths change often, strong networking is essential.”

Finally, if you want to go fully meta, read Devansh’s take on how Edge-GNNs have improved the design process for, well, AI Chips.

I look forward to your feedback.

Subscribe now