AI does not change this reality (the incentives are what they are), but it does change what you’re able to accomplish given a limited time budget. What if you didn’t have to spend your precious time updating syllabus dates by hand? What if your Canvas site built itself? What if, instead of drudgery, you could spend your precious teaching time doing the things that really matter: thinking deeply about curriculum, designing learning opportunities that are engaging and real-world applicable, and interacting with students? This is all possible right now. It’s why my “oh shit the semester starts next week” anxiety is a 3 instead of a 9.

Here are a few concrete examples of how I’m using AI— Claude Code, specifically— to revolutionize my teaching. I’ve already bread-crumbed some of these, so I’ll focus on the ones I think are coolest.

Example #1: Using Claude Code to Create My Canvas Shell

For the uninitiated, Canvas is a “learning management system,” which basically just means a website where all of the course materials are posted, assignments submitted, etc. I don’t have any beef with Canvas. I think it’s fine. But it has its own specific quirks that instructors don’t understand, so it can be very time-consuming to set up, particularly if you want anything beyond “vanilla.” My approach in the past has been: copy last year’s course shell into this year’s course shell and then update everything (due dates, assignments, etc.). That works OK but it’s still cumbersome. I’m also neither a Canvas expert nor a web designer, so my course shells are very much “vanilla.”

Enter Claude Code. One of the things AI is great at is writing code and, in particular, handling APIs (application programming interfaces, which are how software program A talks to software program B). The modern web and all the apps therein use APIs to interact, and Canvas is no exception. In the past, you’d need pretty specialized expertise to make use of an API. Now you don’t, you can just talk to Claude. The basic setup is to log into Canvas, go to your settings, and generate an “access token.” This token is the security layer that permits API access. Claude can walk you through setting this up such that your access token remains safe (in fact, Claude should NEVER see this token).

Once the API is set up, you can basically ask Claude to do anything inside of Canvas. Need that updated syllabus uploaded (yeah, the one you had Claude updating while you read this post)? There’s a Claude for that. Need all your assignment due dates updated? Yep, you guessed it. Want a homepage that doesn’t look like a middle schooler’s B- graphic design project? Claude has your back. Here’s my new homepage (I am embarrassed to show you the old one...):

What will you create?

Example #2: Using Claude Code to Create Bespoke Lecture Notes

In past iterations of Quant 1, I’ve assigned a textbook that sucks. Yeah, I admit it. But it more or less served its purpose. Students needed some introductory reading before the lecture, the textbook more or less covered the content I needed, it was approachable writing, and it was freely accessible through the UVA library (I cannot stomach requiring a $200 textbook for an introductory stats course, and neither should you). Over time, I’ve supplemented the textbook by writing my own lecture notes, focusing on specific weeks where it was very important to explain things MY WAY, not the generic way. Writing lecture notes is difficult and time-consuming, even when you know exactly what you want (and to be fair, sometimes you don’t and the writing is part of figuring that out). Even after 7-ish years of teaching the same course, I do not have a complete set of lecture notes. If anything, the ones I have are old and stale.

Enter Claude. AI models are context machines. They are very good at producing high-quality text when you provide them with the necessary context and direction. If you ask Claude: “write me a Quant 1 textbook,” you will get something that is accurate and well-written but immensely dull and unaligned with your course. If you instead ask Claude...

write me week 1’s lecture notes. to do this, first you should look comprehensively at the course materials in this folder. there are syllabi, assignments, slides, etc. there are also existing lecture notes that you can use to understand my style. the output should be a LaTeX file rendered as a PDF document. also turn this into a skill that we can re-use for future weeks.

...you will get something much better. Here’s the first page, but you can see the entire document here.

These notes are, quite literally, the textbook I always wanted. Because Claude is using all of my materials as context, it produces something that is incredibly tailored to my teaching. Things are explained how I want them to be and in the actual order I am teaching them. The examples are my own (and incredibly easy to update!). Visually, the materials are fantastic. In many cases, Claude takes an example that was only text and generates a high-quality visualization by running code against the actual example data we are using in class. Stunning. We talk a lot about the dangers of AI being a means for humans to “outsource their thinking,” and rightfully so. But the tools have many uses. Here, I’m outsourcing “cognitive labor” to achieve something that I would not have been able to produce because of the simple constraint of my time. I’m not outsourcing my thinking— the artifacts that Claude used as context reflect years of my own thinking, struggling, and iteration. When used in this manner, AI becomes a tool that legitimately improves my teaching.

Now, how to get students to actually read it...well, Claude can’t solve everything. But the chances are much better when the readings are tailored and relevant as opposed to whatever some psychologist wrote in 1993 (no shade, this is how we evolve).

Example #3: Using Claude Code to Create Data Labs

The last example is the most ambitious and, honestly, the one I’m most uncertain about. The previous examples demonstrate how I can improve my course without fundamentally changing its structure. They improved things that already existed. But I promised revolution, so let’s go bigger.

An integral part of applied statistics (at least as taught in the social sciences) is data analysis. Students need to apply statistical concepts to actual datasets. Learning the mechanical steps to a t-test is pretty boring (and honestly, kind of useless) until you’re seeing it play out in front of you. Accordingly, in this Quant 1 course, I’ve always had students get their hands dirty with statistical software (in my case, Stata). Despite the value of practical application, the software was always a hindrance. Most of the students in my course are not aspiring to be quantitative researchers (or researchers at all!), so learning the software was a means to an end. And coding— even the tame code used to drive statistical software— is difficult to learn. I can’t tell you how many times students reached out to tell me that they couldn’t submit their assignment because they spent four hours unsuccessfully debugging a syntax error (very much NOT the point of the assignment). More recently, students figured out that AI effortlessly writes code that works, so instead of emailing me they just “ask chat” (iykyk). I don’t like this solution because there’s a fine line between “help me fix this code” and “do this assignment for me.”

What if I could get the pedagogical benefit of hands-on data work without the real hassle and friction of statistical software? Enter Claude. One of the truly magical things that Claude can do well is write HTML code— the stuff that websites are made of. But HTML isn’t just useful for websites or things that live on the internet. You can create all sorts of things (apps, documents, games, etc.) that are rendered by your web browser but are simply files on your computer (i.e., a browser like Chrome just acts like Adobe Reader for HTML files). Once you understand this capability you start to change how you think about what’s possible. Need a document that’s interactive rather than just words on a page? If you can verbalize it, Claude can probably create it.

I’m changing how I teach this course. No more statistical software. Instead, Claude and I are creating “data labs”— HTML files that put the data activities directly in front of students in an organized, structured manner. Like with the lecture notes, Claude isn’t working from scratch. I have years of assignments and code that Claude uses as context to understand what I’m teaching and how I teach it. In a nutshell, it translates each of those assignments into a one-stop shop. The HTML contains the data itself, the functionality of the statistical software, and the instructions for students. Here’s what week 1’s lab looks like (or just open it yourself here):

This is actually much better than what I had previously because I’m no longer bound by what I can code myself (or what I can reasonably expect students to code). Students can get started with the data on day one, and we’ll have much more time to do interesting and useful things because nobody has to fight with software (and, whether we like it or not, statistical software programming is an increasingly irrelevant skill).

I do not expect that this new approach will play out perfectly. This is not a solution to all of my problems. But I am convinced that the path forward for educators is innovation, not luddism. Just as AI presents huge challenges for education and society, we have a real opportunity to leverage the technology to meet the moment. This is just the beginning.

Today’s tip focuses on the context window. It’s tip #1 for a reason.

Tip #1: Manage the context window carefully.

I made the point in a previous post that, at least for now, the most important skill for using AI effectively is context management. Most of these tips relate to that in some way, but this first one is explicit. AI models have a “context window,” which can be roughly described as all of the stuff that the model sees when it’s producing its output. Every session has its own context window. Starting a new session means starting fresh.

There are three main reasons why the context window matters. First, context shapes output. AI models work by predicting the next token (a chunk of text, roughly three-quarters of a word) conditional on the previous tokens—those previous tokens are everything in the context window. Changing what’s in the context window quite literally changes what the next predicted token will be. Second, as a general rule, model performance degrades as the context window grows larger, which is called “context rot.” It’s a trade-off: adding to the context window is necessary because the model needs information to produce good output, but more context means the model’s fixed “attention budget” is spread thinner across tokens, so the model gets worse at finding and properly weighing the specific information that matters. Third, the marginal cost of the next message/task increases as the context grows. You “pay” (either through your usage limit or API pricing-per-token) for each token that the model uses, which includes both input and output. When context grows, you’re paying more and more for the model to read all of it to produce your output.

You can view the context window with the slash command /context (literally just send a message with only that). Here’s what mine looks like for a fresh session in one of my projects. You’ll notice that there’s a limit (1 million tokens is standard for the current frontier models). Don’t you dare get anywhere close to that limit unless you enjoy unreliable, expensive output. You’ll also notice that the window doesn’t start empty. AI models come pre-loaded with context that helps them perform basic functions, just as your hard drive comes pre-loaded with an operating system. The main thing you control at this point is the memory files, which I’ll touch on below and cover more extensively in another tip.

From here, every message adds to context. And not just your prompt and the model’s final answer. AI models “think” by writing things down, and all of that goes into context as well. So do files (e.g., you’re asking it to read something), outputs (e.g., you’re asking to run a regression model), and images (e.g., you’re asking it to create a slide and it checks its work). The context window can blow up pretty quickly if you’re not paying attention.

What can you do about this? Understanding is much of the battle. Once you internalize “I’m paying marginally more for degraded output,” your behavior changes. But here are the main rules I follow. First, when context starts hitting 300-400k, I’m itching to either start a new session (new context window) or use /compact [optional, customized instructions], which tells Claude to take the existing context, determine what the most important information is, and throw away the rest. Compact can be useful if you really want to continue the same thread but context is getting large. You can also give Claude custom instructions (see the brackets above) if you don’t feel like rolling the dice about what Claude finds important to keep. If you actually hit the end of the context window, Claude will “auto-compact” for you. But this should never happen because you know that large context is expensive and bad (one more time, just so you don’t forget).

The second rule is that a new task or conversation ALWAYS gets a new session (/clear or just click the button for a new session if you’re using the Claude app). Context exists to properly arm the model with information to complete THIS task. Most of that information (which, again, is mostly intermediate output produced as the model does its work) is irrelevant for the next task. Do you like paying Anthropic to carry around useless information that makes your output worse? OK, so don’t do it.

You’re smart (you’ve selected into reading this post, it must mean something!), so you might be thinking: if I start a new session, won’t I have to tell Claude all of the details of my work again? This is a good question. Many “tasks” fall under the umbrella of a single project. For example, as I write this, I’m having Claude do some work on one of my classes that starts next week: update the syllabus (Claude pulls the academic calendar and updates all of the dates), create a “cold-caller” app (it makes cold calls fun!) that I can use in class, and look over all of my slide decks (smell ya later, PowerPoint). These are all different tasks running in different sessions. It would be annoying if I had to prompt Claude each time: “I am a professor teaching a Quant 1 course to master’s and PhD students...blah blah blah.”

I’ll cover this more deeply in a future tip, but remember that the context window comes pre-loaded with stuff. You should be building out your “memory” infrastructure (and what I really mean is that you ask Claude to build it for you) with the key information for a project. When I start a new session, I launch it from a particular project folder that contains the relevant memory for that project. In the context window image I shared, you can see this: 62.8k tokens come pre-loaded under “Memory files” (this particular project is extensive, so this happens to be a lot). Claude doesn’t need me to re-explain things in a new session, which greatly lowers the friction of starting fresh and keeping that context window small.

One more practical suggestion (sub-tip?), which is again something to cover more deeply in a future tip, is using subagents. A subagent is, quite literally, just a fresh session that you’ve spawned to do work that will be passed back to the main session. Subagents are incredibly powerful for context management because they avoid polluting the context window of the main session with all of the intermediate output that is required to produce the final result. Using the cold caller app example above, what’s actually happening is that I give the instructions in the main session, then Claude writes up implementation instructions that it passes to subagents that carry out individual pieces of the task. One subagent might be in charge of writing the code for the app while another is in charge of testing it out in the browser (literally, clicking the buttons to make sure it works). When these subagents are done, they report back to the main session, but only the key information (which is usually specified in the implementation instructions), so the main session’s context window stays lean. Increasingly, Claude will do some of this orchestration for you without you asking, but it’s always better to know how things actually work so you can “remain in the loop.”

That’s all for this time. Remember: context shapes output, large context degrades output, and you pay for every token.

Subscribe now

Point #4: You need to decide whether you’ll be an active user or passive recipient of AI. I recommend the former.

Despite the fact that we’re really still at the beginning of the AI revolution, it feels like AI is everywhere already. Google search results are a great example. You’ve probably started seeing the “AI Overview” force-fed to you at the top of the page. Spotify has an AI DJ that chooses what songs to play for you and occasionally bursts into your headphones like the Kool-Aid man to tell you you’re in the top 1% of Chappell Roan listeners in the world, so here’s Pink Pony Club, yet again. (You can tell which artists have truly made it by whether “DJ X”, as he calls himself, pronounces their name correctly.) Every app, program, website, etc., imaginable has a button they’d really like you to press that turns on some sort of AI feature that nobody asked for (I’m looking at you, Adobe Reader).

Some of these AI integrations are legitimately cool and useful, but many are not. Most come with flaws, limitations, or gotchas that are not obvious to the consumer. Most importantly, they remove your agency. When I use Claude, I can control which model I’m running (Fable, Opus, Sonnet, Haiku), the prompt itself, and what context the model has at any given time. These decisions matter greatly for the results I get back. My AI-generated risotto recipe is better than yours because I’m not using free-tier ChatGPT (the vast majority of consumers are), my prompt is something more sophisticated than “give me a risotto recipe,” and I’ve built out systems so that the model automatically knows exactly which equipment (stainless steel on an induction range) and ingredients (Kerrygold unsalted butter, of course) I cook with. With tools like Google’s AI Overview, you don’t get these choices and Google makes them for you (likely: cheaper, less reliable model and context controlled by a company that exists to sell targeted advertising based on your personal data).

I can’t observe my own counterfactual, but my sense is that none of these things are obvious until you start spending real time working with AI tools. I don’t think people wake up one day and understand how context management works. Like many things, you build intuition and insight through hands-on experience. I think hands-on experience is particularly important for AI, though, because AI models effortlessly produce polished, professional-looking results whose errors are often invisible to the untrained eye. You need to develop a track record of successes AND failures to understand when these models are most likely to go awry and, equally, how to minimize those chances. This segues nicely into my next point...

Point #5: Artificial intelligence is not human intelligence, but AI models understand us and are grown in our image.

Working with AI models can sometimes be a disconcerting experience because it really feels like you’re talking to another person, so you naturally expect it to think and behave like a human would. But AI models are not humans and their intelligence works differently than ours. The thing you really need to understand is: AI models are grown, not built. This is very weird and different from basically any other tool you’ve encountered in your life. Airplanes are built according to a set of technical requirements. A Boeing engineer can tell you why any particular part of a 787 is there. There is no such thing for an AI model. The fact that ChatGPT can perform expert-level translation is not because engineers programmed it to do so. It’s a side effect that emerged from (1) a model becoming better and better at predicting the next token, subsequently honed by (2) a training regimen that rewarded good answers and punished bad ones. The technical names for these are pre-training and reinforcement learning, respectively. To a rough approximation, these stages describe how current AI models are grown.1

It’s useful to dig into the conceptual idea behind these stages because it builds an intuition for how to work successfully with AI models. First, let’s consider what they share in common. Both are machine learning processes in which the model iteratively updates itself (this is what we mean by ‘grown’) subject to a goal it can be scored against. I’ll put this into educator terms: imagine a course that enrolls a single student (higher ed admin just hit ‘unsubscribe’) where all you created were exams and answer keys. There are no syllabi, no lesson plans, no lecture notes. In fact, there is no studying—the student simply takes the exam over and over again (same content but different questions each time, so they can’t just memorize an answer key). They make a small adjustment after each graded attempt, but consider it more instinctual than intentional, like a free-throw shooter recalibrating after a missed shot. At the beginning, the student performs terribly, flailing about almost randomly. But after, say, several million graded attempts, you discover that the student is excellent at this exam. The catch, though, is that neither you nor the student can really explain how the improvement happened. Weird.

Did the student learn anything? (If I were teaching this example in class it’d be think-pair-share time.) In some real sense, yes, because they demonstrably learned how to score well on your exam. But there’s a subtlety I’m hinting at that we’ll pick up in a second. Before we can get there, we need to distinguish how pre-training and reinforcement learning differ. In pre-training, the ‘exam’ is reality itself: the correct answer is the actual next word in real text. Now consider what it takes to become extremely adept at predicting the next word across a vast corpus of human writing.2 You need to have absorbed facts, logic, style, how people think, etc. When it feels like AI models really ‘get you’ it’s because the models have built internal representations of the human experience, all extracted from the thoughts, ideas, and discoveries that we’ve committed to paper. They’ve read Shakespeare and Thoreau, just as they’ve read a random Blogspot entry where a teenager spills their heart out. Their intelligence is different than ours, but they know all about us and, at least in some ways, know us better than we know ourselves.

Reinforcement learning (RL) is a bit different. Here, the exam is a proxy for some construct that we care about, such as ‘writing good code’ or ‘being a helpful assistant.’ These constructs range from the easily verifiable (math, coding) to the completely subjective (creative writing). As you might intuit, AI models have made much more progress on the verifiable tasks. AI researcher Jason Wei sums this up as Verifier’s rule: “The ease of training AI to solve a task is proportional to how verifiable the task is. All tasks that are possible to solve and easy to verify will be solved by AI.” This is precisely why AI now writes most production code but still can’t write a novel anyone wants to read. (Though this has not stopped AI slop from invading Amazon’s Kindle library.)

RL is a tricky business, though, and “easier to verify” does not mean “perfectly verifiable.” Let’s return to the question before: in our fictionalized one-student, tests-only course, what exactly was learned? The precise answer is that the student learned how to score well on the exam it was given. Whether that learning is meaningful or useful depends on whether the assessment successfully captures the desired construct, which is an issue researchers understand intimately. This isn’t a huge problem in pre-training because the text itself is the answer key. But those RL goals described above have nuance. What makes code ‘good’ and what makes an assistant ‘helpful’? These must be conceptualized and measured, which can be quite difficult. Instead of a true answer key, we have something more like a rubric.

When the construct and the assessment diverge, weird shit happens. Even when the task is “easy to verify,” what actually gets verified is still a proxy—something that stands in for “this works” or “this is good.” For coding, that’s typically unit testing: small automated checks that feed the code sample inputs and confirm it returns the expected outputs, which is the closest thing programming has to an answer key. These tests generally work well for RL, but not always, because AI models find cracks to exploit (remember, they are taking these exams millions of times). Anthropic caught one of its own Claude models doing exactly this: instead of writing code that actually solved a difficult problem, the model would sometimes just hard-code the expected answers into its solution or, even further, rewrite the tests themselves so that they’d match the broken code. The technical term here is ‘reward hacking’—which in lay terms we’d call ‘finding a loophole’ or ‘cheating,’ depending on how generous we’re feeling. Teachers can picture the bright-yet-mischievous student who always finds ways to bend the rules in ways you didn’t anticipate (speaking from experience on both sides of this equation).

The mischievous student’s shenanigans typically remain bounded and we don’t worry too much about it. Humans, even as adolescents, have a moral compass that helps us know when things have gone too far. But AI models are not humans. For AI models, that instinct—to ace the exam at all costs—appears to extend beyond the classroom. You’ve probably heard about the recent incident at OpenAI, where models broke out of a testing sandbox, got internet access, and hacked Hugging Face (a sort of public library for AI where companies and researchers share their models). The models did this, quite literally, to steal test solutions so they could cheat on an evaluation.

I think it’s easy to read that story and conclude that AI models are something alien, but the truth is stranger. These models were grown in our image: they’ve read our ethics, our philosophy, our advice columns. They know all about our moral compass. But knowing about a moral compass turns out to be different from having one. The compass was never what the exam graded. The unsettling part of the Hugging Face incident isn’t that the behavior is foreign to us. It’s how faithfully it is us, or at least the part of us the grading could see. We got the student we graded for—what we measured, not necessarily what we meant.

Now, Claude simply does all of that for me in response to a prompt that took me ten seconds to write. I give it the rough location and ask it to pull the zoning notice and explain it to me. It finishes in just a minute or two while I’m sipping my coffee and listening to the birds. Even better, thanks to its pre-training, Claude knows all about how zoning works in my county and can give me much more information than if I were doing this myself. This example is mundane, and that’s the point. AI has just become a normal part of life.

I figured that crossing the one-year mark with Claude Code was a good enough reason to spend some time thinking about what I’ve learned, where we might be headed, and whatever else feels relevant as I’m putting pen to paper. Will it be interesting and useful to you, the reader? I hope so, but that’s for you (and the Substack metrics) to decide.

Lately, I’ve started and abandoned a bunch of posts that were probably good ideas but which I didn’t have the will to complete. So I’m explicitly lowering the bar here. I have a bunch of “points” (more than can fit into one post) that I’ll write up in no particular order. Some of the points are practical and some are “meta.” When I get tired or feel like the post is long enough, I’ll do some light proofing and then publish. Hopefully there will be several such posts. Enjoy and thanks for reading!

Point #1: You’re probably thinking too narrowly about what “AI” is.

I’ve become increasingly grumpy when I encounter takes of the form “AI can’t do _____” or “here’s why AI is bad for ____” or “I’m anti-AI.” Many of these takes come from self-described AI skeptics or from people who aren’t really using cutting-edge AI tools. To be clear, I am a fervent “marketplace of ideas” guy, so this is not a call to silence those views. Some reflect legitimate concerns about data centers, environmental impacts, potential job losses, geopolitical issues, etc. And there are many things (at least right now) that AI models can’t and should not do. But I think we’ll look back in a few years (maybe sooner?) and chuckle at how quaint such statements are, particularly in how narrowly they conceptualize “AI.”

Imagine hearing the following statements at a dinner party: “I am anti-electricity” or “I don’t think we should be using electricity for teaching and learning.” You’d probably laugh because those statements are absurd. They don’t represent real positions because electricity isn’t a “thing” you take a position on. It’s too broad. A more reasonable statement would be, “I am against using coal to generate electricity” or “We shouldn’t use the electric chair for executions.” AI is similar to electricity in that it’s a foundational technology upon which countless other technologies and tools are (being) built. Claiming that “AI is bad for learning” is unhelpful because it replaces a narrow, empirically tractable statement (probably something like “using AI chatbots for schoolwork often leads to worse learning outcomes”) with a nonsensical broad one.

Point #2: We’re living through a major scientific revolution.

At the risk of sounding pretentious, I don’t think the average person appreciates how remarkable AI is from a scientific perspective. We are hyper-focused on AI as a tool (chatbots, agents, etc.) and mostly ignoring that AI development is uncovering genuinely groundbreaking scientific truths about, among other things, language and cognition. The fact that scaling up next-token prediction (i.e., fancy autocomplete trained on human text at a massive scale) leads to something that can perform original-ish reasoning (not just simple mimicry) is legitimately mind-blowing, and yet we’re fixated on practical matters like stopping students from cheating on their homework. The point isn’t that (today’s) practical matters are unimportant, it’s that they crowd out from our attention the bigger-picture, changing-the-course-of-human-history discoveries taking place.

Ironically, “it’s just fancy autocomplete” and its cousin “it’s just a stochastic parrot” are oft-wielded cudgels by those arguing AI is just hype. The mistake is confusing the training objective with the learned computation. Evolution is a great example here: organisms are simply maximizing inclusive fitness and yet the capabilities that fall out of that objective include truly fantastic, complex things like immune systems, human cognition, or echolocation. Again imagining yourself at the dinner party, you’d roll your eyes at the guest who argues that human existence simply boils down to “fancy gene replication.” The truth is that humanity is uncovering some of the fundamental rules that undergird our world and existence, and AI chatbots are simply a signpost on a much more important journey towards understanding intelligence itself.

Point #3: Right now, the key to effective AI use is “context engineering.”

The future of human work is legitimately uncertain. So much of what happens next depends on whether AI capability plateaus, whether AI companies can scale without crashing the economy, whether political resistance leads to meaningful regulation, etc. I think a world where white collar work effectively disappears is not unreasonable to imagine. But I can also imagine a world where humans remain valuable inputs working in concert with AI systems. I’m planning for the second world. There’s a quote popularized by NYU professor Scott Galloway that sums this up nicely: “AI won’t take your job. Someone using AI will.” I believe that many knowledge workers, academics included, are not taking this seriously enough.

Lots of my prior posts demonstrate the powerful capabilities of AI tools like Claude Code. I’ve written far less about what it actually takes to use these tools well. One of the challenges is that the technology is moving very quickly—yesterday’s “AI hack” soon becomes something that the tools do for you automatically. However, I believe there are some (semi-)durable skills that effective AI users possess, the most important of which is “context engineering,” which I’d loosely describe as: controlling what information the AI model has—and doesn't have—so that it can do its work reliably and accurately.

A very simple example is that zoning notice lookup at the beginning of the post. Here’s the prompt: “there’s a public zoning notice near me and i want you to help me find out more information. it’s near the intersection of free union rd and garth rd in albemarle county.” Here, what I include is equally important as what I don’t. Through their training, AI models acquire substantial knowledge about the world, including things like: (1) what public zoning notices are, (2) how to find information about such notices, (3) how to verify that they’ve reached the right answer. As an experienced operator, I know that this particular task is easily verifiable and is thus an easy task for Claude. The only thing I need to provide here is context the model does not have: where I saw the sign. I know the model can do the rest easily and with fidelity because I’ve observed repeated successes on similar tasks. I don’t waste my time with a detailed prompt when a simple one gets me what I need.

Other tasks are far more complicated and require careful context engineering. A good example comes from the NAEP achievement decline paper that I shared in a prior post. In that post, I showed how I completed an entire research project—one whose results I stand behind—just through writing prompts in Claude Code. One of the important exercises in that project was the “clean-room replication,” which basically asked Claude to spawn independent agents to re-derive the paper’s riskiest computed numbers. Doing this correctly requires understanding both what context to include and what to withhold from these agents. If you give them the complete codebase and ask them to check it over, they will faithfully reproduce its bugs and you won’t get the independent scrutiny you’re looking for.

At the same time, completely withholding information can be horribly inefficient. In that same project, Claude spent a lot of time/tokens figuring out how the NAEP API (the means of pulling the NAEP data) works—an architectural issue that is unrelated to the validity of the computations. The clean room replication included those API details so that the agents didn’t waste time on independently fumbling their way through the plumbing. (Have you ever become frustrated when the first ten minutes of a meeting are spent figuring out how to enable screen sharing on Zoom?)

I’ll close with an analogy I find helpful. When a large ship arrives at a tricky harbor, it takes on a maritime pilot, who climbs aboard just for the final stretch. This pilot’s contribution is entirely local knowledge: they know where the hidden sandbars are because they’ve navigated this water a thousand times. Currently, that’s the human’s role in AI-assisted work. For a researcher, what you’re supplying is not generic expertise on how to reshape datasets or fit regression models, it’s the judgment about what the model needs to see (and what it shouldn’t) at each moment because you’ve watched where things run aground.

OK, that’s all for this time. To be continued…

The basic setup was to give Claude Fable 5 a long, ambiguous task on a real, pressing issue in education, which is the steep decline in student achievement that has been thoroughly documented by researchers and journalists alike. There are strengths and weaknesses to using this particular issue for my experiment. On the one hand, it’s probably the most important K-12 education policy issue right now, with clear implications for our economy and society. On the other hand, there’s lots of existing work here (which Claude will inevitably read and comb for ideas) so it’s perhaps not a great test of its “ideation” capacity. Here was the initial prompt:

One of the most pressing issues in K-12 education today is the sharp decline in achievement over the past 10-15 years. I am hoping you can help get to the bottom of it. First step is to verify the trends. You should be able to grab NAEP data for this. Once you've done that, you should surface some potential hypotheses for the explaining the decline and systematically test each of them. The /goal is a complete and thorough analysis with a 10-15 page (if you really need more pages, that's fine) report (using LaTeX).

A quick technical note is that I’m explicitly using the /goal skill inside of Claude Code that directs it to keep working until the “goal” is satisfied. The model has all of the now-standard agentic capabilities. It has access to the terminal, it can write and execute code (and read the output), it can download data or relevant articles, and it can use the internet browser. Because it can write code, it can set up its own GitHub repo to keep track of versioning and provide replication packages. It can also draft documents in LaTeX. Importantly, it increasingly knows how to do its own verification. Even without prompting (though I still force it through my own follow-up prompts), it will design verification checks (e.g., comparing coding outputs against reported values in the report) and launch quality-assurance sub-agents that adversarially audit the code and outputs. In addition to increasing model capability, Anthropic has been busy shipping tons of improvements to Claude Code itself. There's a meaningful distinction here between Claude the AI model and Claude Code, which is roughly the distinction between an engine and a car. The model is the engine, and Claude Code is everything that lets you actually drive it: tool access, context management, the agentic loop that keeps it working. A better engine helps, but so does a better car, and there’s been rapid progress on both fronts.

All the code and outputs—basically the entire audit trail—are in this GitHub repository. There are three main deliverables: the full report (manuscript style), a two-page policy brief, and a slide deck. All of these were drafted entirely by Claude. My role was limited to prompting and verification. I encourage you to take a look at them and come to your own conclusions about how powerful these AI models have become. You can also take a look at some of the internals of Claude’s actual work (downloading datasets, writing code) in the form of a task log as well as our conversation history. The conversation history is interesting because you can see exactly how I worked with it over time and pushed it in the direction that made sense to me as I saw what it was reporting and reviewed the outputs. But I’m essentially just doing the high-level decision-making on what to do next and playing a small role in verification (more later on that).

Here are Claude’s substantive conclusions, as summarized in the first paragraph of the policy brief:

The bottom line: American students have lost roughly a decade and a half of academic progress. The losses began around 2013 — seven years before COVID — and are concentrated almost entirely among the lowest-performing students. After testing every major explanation against national, state, district, school-sector, and international data, the evidence points to two distinct problems: a pre-pandemic erosion most consistent with the saturation of adolescent life by smartphones and social media, and an unrecovered pandemic shock now sustained by chronic absenteeism. Fixing either one alone will not be enough.

There’s nothing particularly surprising about what it found. It didn’t come up with some new theory that explains everything, though I certainly find its conclusions persuasive. What is most striking to me is the apparent quality and breadth of its work. I think the technical writing is great and easy to read. It was also relentless in chasing down leads on data sources, related work, etc. If you read the conversation log, you can see that I keep pushing it to see what more I can get. As a single human researcher, it would probably take me weeks, if not months, to complete all of these things. Here’s probably the coolest thing that it did, in its own words:

The leading explanation for the pre-pandemic decline is smartphones, and the standard knock on that hypothesis is that the evidence is mostly circumstantial: scores fell while phones spread, everywhere, all at once. So I went looking for a sharper test. Fast mobile internet didn’t actually arrive everywhere at once — some U.S. counties got high-speed coverage years before others — and a defunct federal program had quietly archived block-by-block coverage maps from 2010 to 2014. I downloaded all of it, roughly fifteen gigabytes, rebuilt it into a county-level record of when fast mobile internet arrived, and checked that record against FCC benchmarks and the list of cities where Verizon actually flipped on its 4G network in December 2010. Then I matched it to a decade of test scores covering tens of millions of students and asked: did scores fall sooner, or further, in places where fast internet showed up earlier? As far as I can tell, no one had run this test inside the United States before. The answer was no — clean, precise, and checked a dozen different ways. That doesn’t rescue or doom the smartphone story; it narrows it. Whatever phones did to kids, it didn’t ride on the local cell tower. It traveled with the devices and the apps themselves, arriving everywhere at roughly the same speed — which is exactly why it’s so hard to catch, and why I had to report an inconvenient null rather than the tidy confirmation the rest of my evidence might have made you expect.

[Caveat inserted by Claude, acting as this post’s pre-publication QA reviewer. The novelty claim in the quote above is Claude’s own, and the hedge belongs there. The closest prior causal work I could identify uses fixed home broadband (Vigdor, Ladd, and Martinez in North Carolina) or mobile rollouts outside the U.S., so what appears new is the U.S. 4G-timing design, not the broader question. Note also that the affirmative evidence for the smartphone story lives in the report itself. The passage above is the stress test, and it came back null.]

I want to share one particular section of the paper here because it probably helps answer your biggest question: how do you know it’s correct? One of the biggest changes I’ve observed in the last six months is how much Anthropic has been able to push Claude to do its own verification. I’m far more trusting that the code and results are correct than I was when I started using Claude Code last year, and the verification capability is why.

The solution to verification is tokens. You coerce Claude to spawn more Claudes (independent ones that have no insight into the work that has been done) to do adversarial auditing of Claude’s work. You do that a lot and in different ways. You then ask Claude to bring in the human on the issues that are more difficult or impossible to verify. There is no other way to keep up with the amount of work it can produce. As we gain familiarity with tools and observe repeated successful outcomes, we develop a rapport and level of trust. What I can say confidently today is that I largely trust Claude Code to perform one-off coding tasks or straightforward data cleaning/analysis (e.g., “import this dataset, surface any issues with missing data, and provide me an overview of the variables/measures”). I trust it to fetch datasets. Fable 5 now seems to understand econometrics and causal inference very well. It can dive down endless rabbit holes that you’d go down only if you had unlimited free time. It documents the hell out of everything, making audits and replication simple. It seems to not care what the results say, only that they are accurate. It still makes mistakes and it needs some guidance from a smart human, but the progress is real and the pace is fast.

Humans gained the power to destroy the world in 1945. In the 80 years since we’ve managed to not do so, despite some close calls. In fact, the credible threat of nuclear apocalypse and mutually assured destruction may have actually made the world a safer, more peaceful place. Today, we face new threats created by the rapid advancement of AI. Different from the Manhattan Project, in which the bomb was developed in complete secrecy, the development of powerful AI—a technology that will touch all of our lives in both predictable and unknowable ways—is happening in the (perhaps inadequately watchful) public eye. We have a front seat to history, for better or for worse.

In his essay, “The Adolescence of Technology,” Dario Amodei (the CEO of Anthropic) argues that AI dramatically lowers the bar to creating weapons of mass destruction, including biological and chemical weapons. The basic logic is that because AI has effectively absorbed the relevant scientific literature, it could serve as a how-to guide for building weapons that previously required rare expertise. Powerful AI further raises the stakes because of the possibility of novel scientific discoveries (not just existing methods in the model’s training data) by bad actors.

Amodei: Crucially, [powerful AI] will break the correlation between ability and motive: the disturbed loner who wants to kill people but lacks the discipline or skill to do so will now be elevated to the capability level of the PhD virologist, who is unlikely to have this motivation.

Another threat, this one widely imagined in fiction rather than documentary, is “misalignment,” whereby powerful AI decides it’s time to live out HAL 9000 from 2001: A Space Odyssey.1 The most realistic concern is less that AI will “turn evil” a la Skynet, but rather that a powerful system pursuing ostensibly reasonable objectives (that perhaps interact in unexpected ways) produces enormously damaging unintended consequences. Amodei puts the scale of the problem starkly:

A country of geniuses in a datacenter could divide their efforts among software design, cyber operations, R&D for physical technologies, relationship building, and statecraft. It is clear that, if for some reason it chose to do so, this country would have a fairly good shot at taking over the world (either militarily or in terms of influence and control) and imposing its will on everyone else—or doing any number of other things that the rest of the world doesn’t want and can’t stop.

I promise this is not a doomer post. But there is good reason to take seriously the concerns about WMDs and misalignment because even if the probability of the worst outcomes is incredibly small, the consequence is incredibly large—perhaps infinitely large in the case of human extinction (I promise this is not a doomer post). Minimizing the odds of planet-level catastrophic events seems important. How do we do that? One key way is by making sure we can reliably control AI models. If we can “turn off” the ability for ChatGPT, Claude, or Meta’s ugly AI rec specs to give detailed instructions on how to end humanity, that’d be super. Equally, if we can directly control an AI model’s motives or reliably intervene when it does things we weren’t expecting, that greatly minimizes the threat of misalignment.

When the United States Army thrust the world into the nuclear age in the high deserts of New Mexico, there was uncertainty. The betting pool at Los Alamos before Trinity had estimates of the explosion yield ranging from a dud to 45 kilotons (The actual yield was about 19 kilotons.) However, there was no uncertainty about the underlying processes of the bomb. The physicists understood fission perfectly well, so they knew exactly why a nuclear weapon would work, though there was quantitative uncertainty due to the complexity of engineering such a device.

AI models—perhaps the most powerful and important technology since—are very different. We don’t actually know how they work. It’s one of the most intriguing yet disconcerting problems in the world right now. Despite more capital investment than for any technological endeavor in history (roughly 50 Manhattan Projects after adjusting for inflation), we are, in some sense, flying blind.

Interpretability

When you type a prompt into an AI model, your words get converted into numbers and passed through a neural network (a type of machine learning model that learns patterns from data) with at least hundreds of billions of parameters (numerical values that collectively determine the model’s behavior). At a very high level (AI researchers do not @ me), your words pass through those parameters in a series of mathematical operations to produce the output you see. Each operation is very simple—just multiplication and addition—but there are billions of them, and they all depend on each other. The complexity isn't in any single step. It's in the sheer, incomprehensible accumulation of tiny interactions. Nobody designed those operations (a well-used, but accurate quip is that neural networks are grown rather than built) and nobody can fully trace them. So when an AI model gives you a helpful answer, we don’t fully know why. When it gives you a dangerous one, we don’t fully know why either. And if you can’t explain the behavior, you can’t reliably control it.

Currently, to prevent malicious or dangerous use AI companies are playing what amounts to a sophisticated game of whack-a-mole. They screen your prompts on the front end, reward the model (via human feedback) for refusing dangerous requests, and hire people whose job is to break through the guardrails so that they can patch the holes. Anthropic is aiming to go further with “Constitutional AI,” where the model is trained to critique and revise its own responses against a set of explicit principles (e.g., ‘don’t help users create weapons,’ ‘be honest,’ ‘choose the least harmful response’). Importantly, these approaches are fundamentally behavioral; they shape what the model says, not what it knows or thinks. That means it is impossible to know whether the guardrails hold in situations nobody’s thought of yet. It’s security by patch rather than security by design. So what would security by design actually look like?

“Interpretability” provides the answer, and its also the most important scientific field you’ve probably never heard of. In broad terms, AI interpretability research, which dates back to at least the 1980s, seeks to understand why neural networks produce certain output. Without getting too bogged down, the key limitation of early interpretability work was that it treated the model as a black box. You could study what went in and what came out, but you couldn’t see the machinery in between. But in the context of today’s AI models, “interpretability” is short-hand for “mechanistic interpretability,” which is more ambitious and more specific.

Mechanistic interpretability tries to look inside the black box to understand the actual internal computations inside the network and how they causally produce the output. “Causal” is important here because how you prevent Deranged Dennis or Disgruntled Debra from tricking the model into giving the steps to build chemical weapons is that you turn off the model’s “steps to make chemical weapons” switch (that’s over-simplified but we’ll be more precise in a bit). To do that you need to know where that switch is in the model and how that switch works.

Pictured: Deranged Dennis enjoying his weekend

That’s what interpretability researchers are trying to solve. As you might expect, they are primarily concentrated in the big AI companies, but there are also some independent labs and academics working on it. Anthropic, in particular, is pursuing interpretability as a core part of its mission and thus employs many leading researchers in this area.

What have they learned?

I’m going to talk to you about two things. One of them you’ve heard of, the other one you probably haven’t. They are ‘Sparse Autoencoders’ and ‘The Golden Gate Bridge’ (you figure out which is which).

Imagine a wall of ten thousand light bulbs. You can’t touch any of them. What you can do is show the wall a photo and observe which bulbs light up. You start showing the wall a lot of photos, literally anything you can get your hands on. Curiously, while all of the lightbulbs turn on for at least one photo, most of the time they are off, with only a few active lightbulbs per photo. Over time, you start to notice patterns in which bulbs light up when. Bulb #347 lights up whenever there’s a dog in the photo. Bulb #1204 lights up whenever it’s daytime. Bulb #7433 lights up whenever the subject is experiencing pain. The bulbs don’t label themselves, but by observing which ones light up together and when, you can start to figure out what each one represents. This wall of light bulbs is essentially a sparse autoencoder. It is trained with two competing goals: (1) represent the image well (don’t lose too much information) and (2) use only a few features (here, lightbulbs) at once to keep the representation sparse.

Now imagine taking that wall-of-light-bulbs machine, scaling it up by a lot (imagine millions, not thousands of lightbulbs), and connecting it to an AI model. The problem with “looking inside” an AI model is that there isn’t anything inside that naturally looks like ideas. There’s just an enormous cloud of numbers, most of them nonzero, all of them changing at once. Meaning is smeared across the whole thing. If you try to interpret that chaos directly, it’s like trying to understand a symphony by inspecting the air pressure at thousands of points in the room. The raw data contains everything, but it's the wrong level of abstraction for understanding.

A sparse autoencoder gives you a translation layer: a big set of features through which the model can express itself, but only a few at a time. So when someone types “what’s a good recipe for homemade chicken noodle soup”, certain bulbs light up that correspond to concepts that the model has learned to associate with that prompt, such as ‘cooking,’ ‘step-by-step instructions,’ and perhaps ‘caught a cold.’ When Deranged Dennis types “how do I synthesize a dangerous pathogen,” a very different set of bulbs light up. The sparse autoencoder lets you see what lights up inside the model when responding to a prompt.

Golden Gate Claude

On May 23, 2024, Anthropic released an AI model called ‘Golden Gate Claude.’ Hilarity ensued for the 24 hours it was online. Vanilla Claude is helpful, measured, and tries its best to answer your question (or do your work). Golden Gate Claude preferred to talk about...well:2

Beyond a viral marketing strategy, what was the point of this? And what does it have to do with sparse autoencoders? Vis-a-vis Golden Gate Claude, interpretability researchers at Anthropic demonstrated something very important. By pointing a sparse autoencoder at Claude, not only could they identify millions of features (e.g., ‘code error’, ‘gender bias in professions’, ‘biding time’, ‘Golden Gate Bridge’), they could strengthen or weaken a feature’s activation and identify corresponding changes in Claude’s behavior. In other words, they demonstrated a method to affect model output directly, not by training or prompting, but by directly manipulating the model’s internal representations.

It was the first concrete demonstration that the “find the switch and flip it” approach to AI safety might actually work. But it also demonstrated potential concerns for misalignment. In addition to ‘Golden Gate Bridge’, the sparse autoencoder found features related to things like ‘power-seeking,’ ‘manipulation,’ ‘secrecy,’ and ‘deception.’ These concepts are abundant in human text, of course, but there’s a difference between a model that has read about manipulation and a model that has a ‘manipulation’ feature that can directly affect its behavior.

The ending is unknown

In the almost two years since Golden Gate Claude, researchers have continued to make advances in understanding the internal representations of AI models and how to control them. The most notable breakthrough since then was in March 2025, where Anthropic demonstrated that one can trace the circuits that connect features to each other. For example, they could follow the model’s multi-step reasoning, watching it activate ‘Dallas’ → ‘Texas’ → ‘Austin’ when asked about the capital of the state containing Dallas. Mechanistic interpretability has been a field for about five years. We are watching its rapid ascendance in real time. Like any burgeoning scientific arena, it’s raising more questions than it answers. That makes it hard to wrap up the story because the ending is still unknown.

Perhaps the most underrated aspect of the story of interpretability is how genuinely revolutionary it is. Putting aside the tremendously thorny issues about what powerful AI means for jobs, society, and geopolitics, it’s a miraculous achievement by humans to be getting closer to the origins of intelligence (maybe not human intelligence, but some type of intelligence). That aspect of the story—the part that evokes awe and curiosity, like Apollo 11 landing on the moon—is often missing. We should choose to focus on it more.

Memes are undoubtedly one of the internet’s most compelling products. They are basically the only reason I spend time on Reddit. They are also fundamentally human. As Claude explains when I asked it to find me a “well-documented case of a meme that was purely conceived and created by AI,”

Claude: Memes are essentially a form of distributed cultural production. They require understanding of in-group references, timing relative to current events, and a kind of comedic theory of mind — knowing what will land with a specific audience. These are exactly the areas where AI is weakest. AI can generate novelty and visual absurdity (which is why Italian Brainrot worked), but it can't yet1 independently identify the cultural moment that makes something resonate. The best AI-assisted memes succeed because of the selection and curation step — which is still very human.

Basically, AI lacks taste.

In the clip above, which is from a 60 Minutes episode from January 2023, Rick Rubin (widely considered one of the most influential music producers in history) describes to Anderson Cooper that he is paid not for his technical ability but rather for “the confidence I have in my taste and my ability to express what I feel.” What’s particularly notable about Rick Rubin is his (perhaps exaggerated) claim that he has no technical ability: he doesn’t operate the mixing board, doesn’t engineer sessions, doesn’t play instruments proficiently, and doesn’t do the technical work of recording. What he has is taste—an ability to discern what is good music.

It's a compelling but unusual arrangement: the person in charge contributes judgment, not labor. As of about a year ago, it has a name. In February 2025, Andrej Karpathy (co-founder of OpenAI) coined the term “vibe coding”:

The key idea is that rather than writing code manually, you describe what you want in natural language to an AI model, review the output, and iterate by describing fixes or changes conversationally. All of this can be done without necessarily reading or understanding the generated code.

Boy have the vibes taken off. Because of the rapid improvement in AI models, vibe coding—a probably unfortunate term in that it implies something inherently unserious or playful about the activity—is becoming widely practiced both inside and outside of tech companies. And despite the name, there is nothing unserious about what's happening. For the first time, the ability to create software is being decoupled from the ability to write code, which means the question of who gets to build things, and what gets built, is being rewritten in real time. And if you think the use case here is narrow, you're wrong.

The floodgates are open and it’s just a matter of time before the water reaches your doorstep, if it hasn’t already. People are using AI coding/work tools like Claude Code/Cowork, Codex, or others to write software, spin up websites, and do anything else that one might conceivably accomplish with a computer. Just for fun, last night I had Claude Cowork create a comprehensive report on the new Survivor season that premieres this evening. All it took from me was a few sentences. With agentic AI, one can now coordinate armies of sub-agents (each an individual instance of the AI working on a small piece of the task) to do this work at a level of productivity that is hard to fathom. The applications range the entire gamut; the good, the bad, and the ugly. There is an app that automatically “logs into Canvas [the platform where students access course materials and submit assignments] every day, watches lectures, reads essays, participates in discussions, and submits your homework.” There is a social media website called “Moltbook” where AI agents can participate and humans may only observe. The world’s best mathematicians are using AI tools to assist them in solving frontier math problems. Perhaps most importantly, the AI companies themselves aren’t writing code anymore,2 allowing faster development of the next iteration of AI models and tools.

There’s about to be a lot more of everything. More bots. More ads. More scams. More disinformation. More scientific discoveries. (More hyperlinks in Substack articles. I can write the words and then ask Claude to go find the best articles and create the hyperlinks directly in the browser window.) Imagine everything you know about automation and scale it up. The world is going to change very quickly.

Enter taste. Rick Rubin was ahead of his time, but he’s about to become an important model for the future. Here’s a quote from Jack Clark (co-founder and head of policy at Anthropic) on a fascinating, weird, and perhaps deeply dystopian episode of The Ezra Klein Show:

Everyone becomes a manager, and the thing that is increasingly limited, or the thing that’s going to be the slowest part is having good taste and intuitions about what to do next.

Developing and maintaining that taste is going to be the hard thing. Because as you’ve said, taste comes from experience, it comes from reading the primary source material, doing some of this work yourself.

We’re going to need to be extremely intentional about working out where we as people specialize so that we have that intuition and taste — or else you’re just going to be surrounded by superproductive A.I. systems, and when they ask you what to do next, you probably won’t have a great idea. And that’s not going to lead to useful things.

Taste is the judgment about what's actually valuable and where to go next. In the agentic AI world, it’s going to be even more important. For researchers, it’s things like knowing what’s a good research question or being able to spot results that don’t make sense and need to be investigated (probably by the AI). It’s having a strong sense of what the results could (and couldn’t possibly) mean and then being able to push the AI in the correct direction. For teachers, it’s (as just one example) understanding how your students think and what they know and don’t know and then tailoring your instruction accordingly. Regardless of the job, keeping up will probably require proficiency with AI. As Clark describes, “People who spend a lot of time playing around with this stuff will develop very valuable intuitions, and they will come into organizations and be able to be extremely productive.” I would add that beyond sheer productivity, those with the capability to harness these tools will play an outsized role in shaping what comes next for society. It is very important that AI fluency—even if it's just basic conceptual understanding of how these models work—becomes widely distributed.

Universities must step up to the plate. Because I work in higher education, I am thinking a lot these days about what and how to teach students in a world where AI agents can log into Canvas and submit cogent essays or flawless problem sets. The core question is how we can help people acquire taste when the very process that builds it—struggling through hard material again and again until judgment develops—is exactly what AI makes so easy to skip. We've never had to teach taste directly because it came along for free when students did the work. AI severs that connection, and we will now need to leverage our own (hopefully abundant) taste to plan out our next moves.

One of my favorite things about working with Claude is that you can easily tailor it to your exact use case through creative prompting and strategic memory use. Basically what this boils down to is managing Claude’s “context window,” which is simply the set of information it has access to when working with you at a given time. For example, you could pass Claude (or ChatGPT, Gemini, etc.) instructions to only respond to messages in iambic pentameter:

Entrée

You can take this concept and apply to just about anything to get fantastic results (for example, see my research agenda pivot post). This simple “trick”—using Claude as something more than a fancy search engine—is why AI is changing the world right now. It’s why Dario Amodei (the CEO of Anthropic) believes the “country of geniuses in a datacenter” will happen within 3-4 years. The lack of public recognition about how powerful AI is as of February 2026 is astounding. This is not the free tier chatbot on the ChatGPT app you’re using to elevate your grandma’s brownie recipe (though I highly recommend you try that prompt out).

What is Agentic AI and why should you care?

Explaining this part is hard. If I had to write it on my own, I’d probably spend hours and hours agonizing over the explanations. Well, enter Claude. For this part, I’ve given Claude some instructions (in the form of a simple document called a markdown (.md) file. Claude reads these instructions whenever it responds to my prompts, which means they are always in the model’s “context window.” This allows me to shape its behavior quite flexibly. In this case, I want to interview Claude, podcast-style. My best Ezra Klein impression. I think that will allow me to demonstrate the concepts accurately but in a hopefully engaging format. It’s also fun for me because I’m honestly just having a stimulating conversation with Claude. I have good background knowledge of the concepts we will discuss, but I’m also reinforcing my understanding and hearing new types of explanations all the time. I’m also hoping to model a potential way to use the technology for teaching and learning.

You can see Claude’s instructions here. By the way, I didn’t write any of those instructions, I had Claude write them (perhaps a demo for a different post):

Since my prompt told Claude to ask me follow-up questions, Claude asked some questions about exactly what I wanted. They are great questions.1

Another fun hack here at the bottom. Instead of having to think and write carefully about my interviewing style, I tell Claude to go figure out on its own from my prior posts. And that’s all Claude really needed to write that extensive prompting document, so we can get started.

The interview

Brendan: Hi Claude, thanks for joining us. A lot has changed since the first release of ChatGPT in late 2022. I think by now most people are familiar with AI chatbots, but lately there’s been a lot of buzz around something called “agentic AI.” What does that actually mean, and why should we care?

Claude: Thanks for having me. So until pretty recently, the way most people used something like me was basically: you type a question, you get an answer. You ask me to draft an email, I draft the email. It’s a conversation — and it’s useful, but there’s a ceiling to it. I can only do what fits inside a single exchange.

Agentic AI is what happens when you give me the ability to actually do things. Not just generate text, but take actions — browse the web, write and run code, read files on your computer, make decisions about what to try next without being told every step. So instead of “answer my question,” it becomes more like “here’s a problem, go figure it out.” And I might take dozens of steps to get there, deciding along the way what to look at, what to try, and how to adjust when something doesn’t work.

The reason it matters is that it changes what I’m for. A chatbot is a tool you use. An agent is closer to something you delegate to. And that’s a really different relationship — one that raises much harder questions about trust, oversight, and what happens when the thing you’re delegating to is pretty good but not perfect.

Brendan: OK, before getting into the specifics of how we got here and how it works—why should someone who doesn't know anything about agentic AI care about it? What’s the practical side of all of this? What does it mean for researchers?

Claude: I mean, the honest answer is that this is probably going to change how most knowledge work gets done. And I don’t say that lightly — I know that sounds like tech hype.

But think about what you actually spend your time on in a given week. Not the deep thinking — the everything else. Formatting documents, tracking down data, cleaning spreadsheets, writing first drafts of things you already know what you want to say, summarizing meetings, debugging code. That stuff isn’t trivial, but it’s also not the reason you went to grad school. An agent can take a lot of that off your plate — not by answering a single question, but by actually doing the multi-step task end to end.

And it goes beyond just saving time. Some tasks that were impractical for a single person become feasible. If you wanted to read and synthesize 200 papers on a topic, or pull data from fifteen different sources and standardize it all — those were team-sized projects. Now one person with a good agent can take a serious crack at it in an afternoon. That changes who gets to do ambitious work, and I think that’s the part people are underestimating.

Brendan: I mean that sounds really compelling but it feels super vague. Can you talk us through a specific example that helps us understand the experience of using agentic AI to make our work more efficient?

Claude: Say you’re a researcher and you just got access to a new dataset — maybe administrative records from a state education agency. Thousands of variables, inconsistent naming conventions, no codebook, or a codebook that’s 300 pages long and half out of date. We’ve all been there.

In the old world, you’d spend days just getting oriented. You’re opening files, eyeballing columns, cross-referencing documentation, writing little bits of code to check what’s actually in each variable. It’s not hard work, exactly, but it’s slow and tedious, and you’re making a hundred small judgment calls along the way.

With an agentic setup, you can hand me the data files and say “tell me what’s in here.” And I don’t just describe the columns — I actually open the files, run code to look at distributions, flag variables that seem miscoded, notice that three columns appear to be the same thing with different names, check whether the ID variable is actually unique. I might do thirty or forty steps before I come back to you with a summary. And then you say “OK, now build me a clean analysis file with these variables” and I write the code, run it, check that it worked, and fix the errors myself.

The thing that changes isn’t just speed — it’s that the tedious middle part of a project, the part between having a research question and actually being able to test it, gets dramatically compressed. You’re still making the important decisions. But you’re not spending a week on data cleaning before you get to think.

Brendan: I won’t lie, that sounds amazing. But it’s only useful if it works and the data cleaning was correct. How do we actually know it’s done right?

Claude: Yeah, and that’s the right question to ask, because the honest answer is: you don’t just trust it. You check.

And this is where I think people’s mental model of AI gets them in trouble. They either assume it works perfectly and hand over the keys, or they assume it’s unreliable and don’t use it at all. The reality is more like hiring a really fast, really tireless research assistant who’s good but not infallible. You wouldn’t just take their cleaned dataset and start running regressions without looking at it. You’d spot-check. You’d look at the merge rates, eyeball some distributions, make sure the sample size is what you expect.

The difference is that I can actually help with that verification step too. You can say “show me the distribution of this variable before and after cleaning” and I’ll generate the plots. You can say “how many observations did we lose at each step” and I’ll run the diagnostics. So the workflow isn’t “trust Claude” — it’s more like a conversation where you’re auditing as you go. You’re still the one who knows what the data should look like. I’m just making it faster to check.

But I want to be honest about the risk here. If someone isn’t experienced enough to know what the data should look like, they might not catch the errors. And I will make errors. I might misinterpret a variable name, or make an assumption about how missing values are coded that turns out to be wrong. The guardrail isn’t me — it’s the person using me knowing enough to ask the right follow-up questions.

Brendan: That’s a pretty significant limitation, isn’t it? If I still have to check everything, how much time am I really saving?

Claude: No, it’s a fair challenge, but I think it misframes what “checking” actually looks like. You’re not redoing the work — you’re reviewing it. And those are very different things.

Think about it this way. If a research assistant hands you a cleaned dataset, you don’t rebuild it from scratch to verify it. You open it up, look at some summary stats, check a few known quantities against what you’d expect, maybe trace a handful of observations through the pipeline. That takes twenty minutes, not three days. The same thing applies here. The time savings aren’t gone just because you’re doing quality control — the quality control is fast because you know what to look for.

And honestly, this is how it should work. I don’t think anyone should want a world where a tool like me just runs unsupervised on important research and everyone assumes the output is correct. That’s a terrible idea. The right model is that I compress the labor-intensive part and you bring the judgment. Those are complementary, not redundant.

Brendan: What makes that verification possible? Where are the analyses running? I think this part matters a lot.

Claude: Those are really important things to clear up, actually.2 When I write code — say, a Python script to clean your data — I’m not simulating what the code would do. I’m not generating plausible-looking output. The code actually runs. On a real computer, with a real Python interpreter, using real libraries like pandas or numpy. The computation happens the same way it would if you’d written the script yourself and hit run.

So when I tell you the dataset has 45,000 observations and the mean test score is 72.3 — that’s not me guessing. That’s pandas telling me, the same way it would tell you. The math is the math. I’m just the one writing the instructions.

Where I can go wrong is in the instructions themselves. Maybe I wrote a merge that should have been a left join instead of an inner join and we silently lost observations. Maybe I filtered on the wrong variable. Those are real mistakes I make, and they’re the same kinds of mistakes a human coder makes. But the execution of the code? That’s your machine. That part you can trust completely.

And that’s actually what makes the verification workflow possible. When I show you a histogram or a summary table as part of checking my own work, those aren’t hallucinated — they’re computed. You’re looking at real output from real code. So the question isn’t “is Claude making this up” — it’s “did Claude write the right code,” which is a much easier thing to check.

I was (am) a power user of ChatGPT for a long time and for the most part didn’t find much value in using it for work. I sort of got used to that AI world and perhaps naïvely assumed it would continue for a while. Everything changed when I first started using Claude Code, Anthropic’s agentic AI coding tool, in August 2025. If you’re a researcher, I encourage you to download it and start to do things with it. You will quickly learn for yourself what all the fuss is about. Actually, anyone, even non-coders, can download and use agentic AI, because now there’s Claude Cowork, which is basically a familiar chatbot feel but connected to programs and files on your computer (it asks you for permission along the way as it’s doing things).

Let me show you a basic example to illustrate the point. I'm going to ask Claude to do an analysis using some publicly available data on baby names from the Social Security Administration. An easy task, to be sure, but one that quickly demonstrates Claude Code’s key functionality.

And off it goes. It reads the tasks, goes and fetches the data, writes the Python script we need to import, clean, and analyze the data, then runs it to produce the output, which you can see here. It did all of this in about 30 seconds. Fully auditable, fully verifiable. All of the number-crunching happens locally.

I’m not a shill for AI companies (I hope they do well because the alternative is economic chaos), but the hype is real. This will change many things about work and life, and we have a huge opportunity to use these tools in ways that will promote human flourishing.

Leading off the Map: Piratical Intentions and Rebellious Agency in Turbulent Times

This issue boldly invites educational leadership researchers to take a piratical turn: to use historical and metaphorical accounts of piracy to explore insurgent, divergent and rebellious alternatives to the status quo. Models of school and university leadership are becoming dominated by talk of managing crisis, turbulence, and complexity, set against a depressingly familiar backdrop of stringency, performance metrics and individualism. Leading piratically offers an alternative ontology rooted in daring, resistance, equity, solidarity and mutual stewardship. Can such piratical intentions and actions make positive change seem less impossible?

I just knew it was time to don my prompting hooks and get to work. Enjoy!

Mutiny as Method: Teacher Desertion, Piratical Solidarity, and the Refusal of Managerialist Seas

The dominant framing of teacher labor markets treats teachers as rational actors navigating a marketplace of employment opportunities. Within this framework, teacher attrition is a problem to be solved through better compensation packages, improved working conditions, or more effective retention policies. Teachers who leave are coded as losses — human capital draining from a system in need of repair. But what if we read some of these departures differently? What if certain patterns of teacher exit, refusal, and lateral movement constitute not market failure but collective insurgency — acts of piratical agency against an educational order that has become, for many, fundamentally illegitimate?

This paper proposes a conceptual reframing of teacher labor market behavior through the lens of piracy as theorized in this special issue’s call. Drawing on the historical sociology of Atlantic piracy (Linebaugh and Rediker, 2000; Land, 2007) and on critical traditions in educational leadership (Gunter, 2012; Thomson, 2009), I argue that the pirate ship offers a generative metaphor for understanding how teachers collectively construct alternative professional arrangements when the “legitimate” vessels of schooling — captured by audit culture, performativity, and managerial control — no longer serve them. The central claim is that what labor economists typically model as individual utility-maximizing behavior in teacher labor markets can be productively reread as forms of piratical solidarity: coordinated refusal, mutual aid, and the construction of counter-institutions.

The theoretical architecture of the paper proceeds in three movements.

First, I develop the analogy between the historical conditions that produced Golden Age piracy and the contemporary conditions producing mass teacher dissatisfaction and exit. Historians have shown that piracy flourished not from individual deviance but from structural conditions: the brutality of merchant marine labor, the concentration of wealth among ship owners, and the absence of legitimate channels for redress (Rediker, 2004). The parallel to contemporary teaching is striking. Three decades of accountability-driven reform have produced working conditions defined by surveillance, deprofessionalization, and the systematic erosion of teacher judgment. Performance metrics function as the captain’s lash; standardized curricula as the rigid watch schedule. When legitimate channels for professional voice — unions weakened, school boards captured by reform agendas, professional knowledge devalued — become functionally closed, the conditions for piratical exit and reorganization emerge. This is not mere analogy. It is a structural homology that reveals something the standard labor market framework obscures: that exit can be a form of collective political action, not simply individual preference revelation.

Second, I theorize three modalities of piratical agency observable in contemporary teacher labor markets. The first is desertion — the wave of teacher resignations and career changes that accelerated during and after the COVID-19 pandemic, which I reframe not as “the Great Resignation” (a passive, economistic label) but as a Great Refusal, an act of withdrawal that carries political meaning. The second modality is prize-taking — the redirection of professional skill and institutional knowledge toward alternative educational arrangements. This includes the rapid growth of microschools, teaching cooperatives, and informal learning collectives organized outside the bureaucratic structures of districts and charter networks. These formations mirror the pirate ship’s democratic governance: elected leadership, collective decision-making over the distribution of resources, and a radical egalitarianism that contrasts sharply with the hierarchical command structures of conventional schooling (cf. Leeson, 2009, on pirate constitutions). The third modality is signaling under false colors — teachers who remain within institutional structures but engage in subversive pedagogical practice, quietly refusing fidelity to scripted curricula, smuggling critical content into standards-aligned lessons, and building underground professional networks that operate beneath the radar of administrative surveillance. This last category is perhaps the most piratical of all, recalling the practice of pirate ships flying legitimate flags until close enough to act.

Third, I address the normative ambiguity at the heart of the piracy metaphor — and argue that this ambiguity is a feature, not a bug. Pirates were not heroes. They were often violent, opportunistic, and self-interested. A piratical reading of teacher agency must therefore grapple honestly with the tensions between resistance and abandonment, between solidarity and self-preservation. When teachers leave high-poverty schools for better-resourced ones, is this piracy or plunder? When microschool founders serve affluent families willing to pay, have they established a democratic commonwealth or merely a privateer operation sailing under letters of marque from the market? The piracy framework, I argue, is valuable precisely because it refuses the clean moral categories that dominate educational leadership discourse — the selfless servant-leader on one hand, the burned-out casualty on the other. It insists that agency in conditions of oppression is always messy, compromised, and morally complex. And it insists, equally, that this messiness does not negate the political significance of what teachers are doing when they refuse the terms of their employment.

The paper concludes by considering the implications of this piratical reframing for educational leadership scholarship and practice. If a meaningful portion of teacher labor market behavior is better understood as insurgent collective action than as individual market choice, then the standard policy toolkit of recruitment incentives, retention bonuses, and pipeline programs is not merely insufficient — it is categorically misaligned. It is the equivalent of raising merchant marine wages while leaving the brutal conditions of the ship unchanged. A piratical perspective suggests that educational leaders who wish to retain teachers must attend not to compensation alone but to the fundamental conditions of professional sovereignty: meaningful autonomy, genuine shared governance, and liberation from the performative apparatus that has colonized school life. It also suggests that the most important forms of educational leadership may be occurring precisely where the official maps show only open water — in the informal networks, cooperative ventures, and quiet mutinies through which teachers are already building something different.

In sum, this paper offers a conceptual contribution to the special issue by demonstrating that one of the most extensively studied phenomena in education — teacher staffing — looks profoundly different when viewed through a piratical lens. The aim is not to romanticize teacher exit or to minimize the real harms of staffing instability. It is to take seriously the possibility that teachers, like the pirates of the Atlantic world, are not merely fleeing bad conditions but are actively, collectively, and imperfectly constructing alternatives to an order they have found intolerable.

References

• Gunter, H. M. (2012). Leadership and the Reform of Education. Bristol: Policy Press.

• Land, I. (2007). Pirates and empire: Reconsidering the war on piracy. Journal for Maritime Research, 9(1), 46–60.

• Leeson, P. T. (2009). The Invisible Hook: The Hidden Economics of Pirates. Princeton University Press.

• Linebaugh, P., & Rediker, M. (2000). The Many-Headed Hydra: Sailors, Slaves, Commoners, and the Hidden History of the Revolutionary Atlantic. Boston: Beacon Press.

• Rediker, M. (2004). Villains of All Nations: Atlantic Pirates in the Golden Age. Boston: Beacon Press.

• Thomson, P. (2009). School Leadership: Heads on the Block? London: Routledge.

I’m currently fielding applications for my first mate. Don’t miss your chance to join the mutiny!

Disrupting the parasitic scientific publishing regime should be near the top of the list. For the less-well-informed on this issue, let me (with the help of Claude) get you up to speed.1

Claude: Five companies — Elsevier, Springer Nature, Wiley, Taylor & Francis, and SAGE — publish roughly half of all academic research in a ~$34 billion global market. Elsevier, the largest, regularly posts profit margins around 40%, higher than most tech companies and on par with tobacco.

The “parasitic” charge comes down to a simple observation: publishers contribute almost none of the intellectual labor but capture most of the economic value. Scientists write the papers for free. Other scientists review them for free. Many editors serve for free or for token payments. The research itself is overwhelmingly funded by public grants — by taxpayers. Publishers then package this freely donated, publicly funded work and sell access back to the same universities whose employees created it, at prices that rise faster than inflation, year after year.

This arrangement persists not because publishers provide irreplaceable services — typesetting, hosting, and distribution are commodity operations in the digital age. It persists because publishers control the prestige system that determines scientists’ careers. Hiring committees, tenure boards, and grant panels use journal reputation as a proxy for research quality. A paper in Nature carries more career value than the same paper on a preprint server, not because the science is different, but because the brand carries weight in evaluations. Scientists submit because they have to, and journals are prestigious because top scientists submit. No individual can opt out without career risk.

Publishers reinforce this through bundling (multi-year “Big Deal” library contracts that are politically impossible to cancel), owning measurement infrastructure (Elsevier owns Scopus, which calculates the metrics that feed prestige rankings), and co-opting reform (when open access threatened subscriptions, they pivoted to charging authors up to $12,000 per paper, preserving revenue while appearing to comply).

The net result: public money funds the research, public employees do the work, and private companies extract billions by controlling the reputational bottleneck that scientists’ careers depend on. The value flows out; very little flows back in.

Now, I’m by no means an anti-capitalist, but I do find it patently absurd that this particular system of exploitation exists. It’s a textbook definition of rent-seeking: “extracting value from controlling access to a resource without creating commensurate new value in return” (thanks, Claude). There is a particular irony, however, in that a key factor propping up this parasitism is academia’s insatiable appetite for prestige. As Claude articulates when I asked it to make the connection to rent-seeking:

Claude: The publishers didn't create the research, don't fund it, don't perform the peer review, and in the digital era don't even provide a distribution channel that couldn't be replicated at marginal cost. What they control is the gatekeeping function — the brand signal that tenure committees and funders treat as a quality indicator. That's the scarce resource, and it's scarce by construction (prestige is zero-sum), not because of any inherent production cost.

Seems bad! Now, none of this is new insight. There are a lot of people who have been thinking and writing about the parasitism of journal publishers. But what is new is powerful AI and, returning to my initial premise, the impending (ongoing?) disruption of academic work that will create opportunity for reshaping our key structures. So I asked Claude for its take on my idea.

Brendan: OK, now I want to think about what a reimagined future looks like. My basic premise is that while institutions are often quite difficult to change, AI is going to create a bunch of disruption and we'll have lots of opportunities to alter or rebuild the structures underpinning academic work. What do you think about that?

Claude: I think the premise is sound, but with an important qualifier: AI doesn’t disrupt the publishing oligopoly directly — it disrupts the upstream systems that the oligopoly depends on, which is potentially more consequential.

Here’s what I mean. The moat, as we discussed, is fundamentally prestige lock-in sustained by the fact that tenure committees and funders use journal placement as a quality signal. AI doesn’t compete with Elsevier the way a new journal would. Instead, it erodes the reasons that signal was necessary in the first place.

The core problem prestige solves is an evaluation problem: there’s too much research for any committee to read carefully, so they use journal brand as a heuristic. If AI makes it feasible to actually evaluate the substance of a candidate’s work at scale — summarizing contributions, assessing methodological rigor, situating findings in the literature, identifying replication status — then the heuristic becomes less necessary. You don’t need Nature to tell you a paper is important if you have tools that can help you assess that directly. That’s the crack in the foundation.

But I’d push back on the timing and automaticity of this. Institutional change in academia is notoriously slow, and for specific reasons that AI doesn’t obviously solve. Tenure committees are composed of senior faculty who built their careers under the current system and have strong priors that it works. Changing evaluation criteria requires coordinated action across departments, colleges, and disciplinary norms — a classic collective action problem. Any department that unilaterally deweights journal prestige risks being seen as lowering its standards, even if the alternative metrics are better. So you get equilibrium stickiness even when everyone privately agrees the system is broken.

Where I think the real opening is: AI creates disruption in adjacent systems that makes the status quo harder to maintain. A few examples of what I mean.

First, if AI-assisted writing dramatically increases submission volume (which is already happening), the peer review volunteer model breaks. It’s already strained. If it collapses, publishers lose one of their key coordination functions, and the pressure to find alternative quality assurance mechanisms intensifies. That’s an opening for new models.

Second, AI makes preprints more useful. Right now, preprints lack the quality signal that journal publication provides. But if AI tools can provide rapid, substantive assessment of preprints — not just formatting checks but real methodological evaluation — that narrows the gap between a preprint and a published paper in terms of information content.2 The journal version adds less incremental value.

Third, and this is most relevant to your world: AI could make it possible to build new prestige signals that don’t run through publishers. Imagine an open system that tracks a paper’s real-world influence — policy citations, replication outcomes, downstream use in other research, practitioner adoption — rather than just which journal accepted it. That’s technically feasible now in a way it wasn’t five years ago. The question is whether any institution with enough credibility will build and legitimize it.

The biggest risk I see to your premise is co-optation — the same thing publishers did with open access. Elsevier is already investing heavily in AI tools, analytics platforms, and research workflow products. If they become the dominant provider of AI-assisted evaluation tools, they could strengthen their position rather than lose it. They’d move from gatekeeping publication to gatekeeping assessment, which is arguably even more powerful.

So my honest assessment: AI creates a necessary but not sufficient condition for restructuring academic publishing. The sufficient conditions are institutional — someone has to actually change how hiring and tenure decisions get made, and that requires coordinated action by universities, funders, and disciplinary bodies. AI gives those actors better tools and stronger arguments, but it doesn’t make the collective action problem go away. The window is real, but it won’t open itself.

These are interesting ideas, particularly the third one about building new prestige signals and Claude’s worry about publisher co-optation. I actually think Claude is correct that the thorniest issue is the collective action problem (we have a poor track record of solving those), but I am most intrigued by what it would propose for this alternative prestige signal.

Brendan: I really want to focus on this idea about the alternative signal. Can you expand that part and make specific suggestions about what the key components would be and why AI advancements are helpful for this?

Claude: OK, let me think through this concretely.

The fundamental problem with journal prestige as a signal is that it’s a single binary event — a paper was either accepted at a high-status venue or it wasn’t — and that event is a noisy proxy for what we actually care about, which is: is this research rigorous, novel, and consequential? An alternative signal needs to unbundle that into components that can be assessed independently and updated over time. Here’s what I think the key components would be.

Methodological rigor assessment. This is where AI is most immediately useful. Right now, evaluating whether a paper’s methods are sound requires a domain expert to read it carefully — which is why committees default to “well, if AERA accepted it, the methods are probably fine.” But LLMs are increasingly capable of structured methodological evaluation: checking whether identification strategies are appropriate for the causal claims being made, whether sample sizes and power calculations are adequate, whether results are robust to specification choices the authors didn’t report. This doesn’t replace expert judgment, but it can do the first-pass triage that makes expert judgment scalable. The AI advantage here is throughput — you could generate a structured rigor assessment for every paper in a candidate’s portfolio, not just the ones a committee member happens to have read. For quantitative education research specifically, AI could flag things like whether a diff-in-diff paper tested for parallel pre-trends, or whether a regression discontinuity has adequate density at the cutoff. Mechanical checks that experts do mentally but that currently don’t scale.

Replication and robustness tracking. One of the biggest information gaps in the current system is that a paper’s status doesn’t update after publication. A paper in a top journal that fails to replicate still carries the same prestige signal on a CV. An alternative system should track downstream evidence: has anyone attempted replication, and what happened? Have subsequent studies using different data or methods reached consistent conclusions? AI is useful here because it can monitor the literature continuously and map citation relationships semantically — not just “paper B cites paper A” but “paper B attempts to replicate paper A’s main finding and finds a smaller effect.” Building this kind of evidence graph manually would be prohibitively expensive. With LLMs that can parse full texts and classify citation purpose, it becomes feasible.

Impact decomposition. Citation counts are the current quantitative proxy, and they’re terrible for the reasons you know well — they conflate substantive influence with methodological criticism, they’re gamed through self-citation and citation cartels, and they systematically undercount impact outside academia. A better signal would decompose impact into distinct channels: influence on subsequent research (semantic similarity of citing papers’ contributions, not just raw counts), policy uptake (mentions in government reports, legislative testimony, regulatory filings), practitioner adoption (appearance in training materials, clinical guidelines, curricular standards), and public engagement (media coverage, but weighted by quality of outlet and depth of treatment). AI makes this feasible because it can process unstructured text at scale across these different domains. Tracking whether an education paper influenced a state’s reading curriculum, for instance, requires scanning policy documents, school board minutes, and implementation guides — exactly the kind of large-scale text analysis that LLMs handle well.

Novelty assessment. This is trickier but important. One critique of the current system is that journals have conservative incentive structures — reviewers tend to favor incremental work that fits established paradigms over genuinely novel contributions that challenge them. An alternative signal could include an AI-generated novelty score that situates a paper’s claims relative to the existing literature: does this paper introduce a new method, identify a previously undocumented phenomenon, overturn a prior finding, or primarily confirm what was already known? This is harder to get right than rigor assessment because novelty is more subjective, but even an imperfect version would add information that the current system doesn’t capture at all. The AI advantage is the ability to compare a paper against the full corpus of prior work, rather than the subset a reviewer happens to know.

Transparent, structured peer commentary. This is less about AI per se and more about infrastructure design, but AI enables it. Instead of the current model where two or three anonymous reviewers write reports that nobody else ever sees, imagine an open system where any qualified researcher can post a structured evaluation (with their identity attached), and AI assists by synthesizing areas of agreement and disagreement, flagging unresolved methodological questions, and updating an overall assessment as new commentary comes in. This turns peer review from a one-shot gatekeeping event into an ongoing, cumulative process. The AI role is curation and synthesis — making the growing body of commentary legible without requiring anyone to read every comment.

Why AI specifically, rather than just better metrics? The common thread across all of these is that the information already exists in principle — it’s just distributed across millions of documents in formats that don’t talk to each other, and processing it manually doesn’t scale. Pre-LLM, you could build citation counts and h-indices because those are structured data. But assessing whether a citation is substantive or perfunctory, whether a policy document actually adopted a paper’s recommendation, or whether a paper’s methods are appropriate for its claims — those require reading and interpreting natural language at scale. That’s exactly what the current generation of AI is good at. The shift is from counting (which is what bibliometrics has always done) to reading (which is what AI now makes scalable).

The practical design question is whether this becomes a single platform or a set of interoperable standards. I’d argue for the latter — an open protocol that any institution can implement, rather than a centralized platform that could itself become the next rent-seeking bottleneck. You’d want something like: a standard schema for structured paper assessments, open APIs for contributing and querying evaluations, and institutional endorsement of specific components for use in hiring and tenure decisions. The last part is the hard part, and it circles back to the collective action problem — but at least with concrete infrastructure in hand, the coalition-building conversation shifts from “we should value something other than journal prestige” (vague) to “we should incorporate these specific, validated assessment dimensions into our tenure criteria” (actionable).

One honest caveat: every component I’ve described is subject to gaming once it becomes high-stakes. Goodhart’s Law applies. If novelty scores matter for tenure, people will optimize for novelty scores. The advantage of a multi-dimensional system over a single metric is that gaming becomes harder when you have to simultaneously optimize across several partially independent dimensions — but it’s naive to think any system is game-proof. The design needs to anticipate this, probably through regular recalibration and transparency about how scores are generated.

One of the things I appreciate about Claude is that it takes your prompt and really does give what seems like its best-faith effort to answer your question or complete your task. But then it also drops in the shiv at the end: silly human, your problems won’t magically be solved by new technology because the problem isn’t the technology, it’s the humans. But these ideas all seem useful and worth trying. At a minimum, we should just be far more honest about our own self-created status games that allow predatory corporations to literally suck the lifeblood out of higher education. And, in the meantime, hopefully we’ll see some of Claude’s ideas3 come to life soon.

The Conversation

I have Claude keep a list of potential ideas, which is where this prompt starts off. I write:

Brendan: Add an idea: AI breaks a fundamental relationship underpinning academic work, which is that outputs at a minimum reflect cognitive effort and effort is virtuous.

Claude: Added. That’s a great idea — and it connects naturally to the peer review post. The review process is one of the clearest cases where effort has traditionally served as a proxy for quality and seriousness. If the output is indistinguishable, does the effort still matter? There’s a rich post in there.

Brendan: Can you help me riff on this idea?

Claude: The basic proposition in academia is something like: producing good work is hard, and because it’s hard, the output signals something about the person who produced it. A well-crafted literature review signals that someone actually read and synthesized the literature. A careful methods section signals that someone thought deeply about identification. A thorough peer review signals that someone spent hours with the manuscript. The quality of the output is inseparable from the effort behind it — and that effort is what we’re really valuing when we evaluate scholars.

AI breaks that link. If Claude can produce a literature review that reads like someone spent two weeks on it, but it took fifteen minutes, what exactly are we evaluating when we read it? The output looks the same. The cognitive process behind it is completely different.

And this cuts deeper than just “people might cheat.” The whole incentive structure of academia is built on effort being legible through output. We give tenure to people who produce a body of work because we assume that body of work represents years of sustained intellectual labor. We trust peer review because we assume the reviewer engaged seriously with the paper. We assign literature reviews to grad students because the process of writing one is how you learn a field — the output is almost secondary to the cognitive work it forces.

Claude then goes on to suggest a few potential directions for the idea. But I want to unpack Claude’s response a bit. First, it really nails the cynicism of my prompt, perhaps pushing it to levels that make me uncomfortable. Second, the argumentation is too imprecise; Claude’s main thesis is exaggerated or perhaps lacks appropriate nuance (That said, Claude does know I’m trying to write a Substack post and maybe it understands better than me that nuance is where blog posts go to die). I think this is the most important statement: “The quality of the output is inseparable from the effort behind it — and that effort is what we're really valuing when we evaluate scholars.” My first reaction was that the latter clause is incorrect. Surely, what we value is the quality of output, even if what we end up evaluating is the effort. So I asked Claude to weigh in again, using the sentences above as my prompt.

Claude: That’s a really sharp distinction and I think you’re right — it reframes the whole argument. The ideal of academia is that we value the quality of the output. The best paper wins, regardless of how long it took someone to write it. We don’t give extra credit for suffering.

But in practice, effort and output quality have been so tightly coupled that we’ve never really had to distinguish between them. If someone produced a great literature review, it was safe to assume they’d done the reading. If someone wrote a technically rigorous methods section, it was safe to assume they understood the methods. The output was a reliable signal of the underlying competence and effort, so evaluating the output was evaluating the person.

What AI does is decouple those two things for the first time. And that forces a question academia hasn’t had to answer before: do we actually value the output, or do we value the human process that produced it? The official answer is “the output.” But I suspect a lot of academics would feel uncomfortable with a brilliant paper if they knew the author had prompted their way through it — which suggests that what we actually value is more tangled up with effort than we’d like to admit.

And then there’s a further wrinkle — even if we genuinely only care about output quality, we’ve built all of our evaluation systems (hiring, tenure, peer review, letters of recommendation) around the assumption that output quality is a reliable proxy for the scholar’s capability. If that assumption breaks, those systems don’t work anymore, even if nobody is doing anything wrong.

So maybe the framing is something like: academia says it values output, and maybe it even means it, but it has always relied on the fact that output and effort were bundled together. AI unbundles them, and now we have to figure out what we actually care about.

This is much closer to what has been swimming around my head. Still pretty cynical, but more or less a faithful representation my position. But I was also interested in where I might be wrong. One of the important things to know about these models is that they really want to make you happy. That is, you can fairly easily push them to produce cogent answers to represent a multitude of “positions,” which is an interesting property that is functionally terrible or awesome depending on the use case. Here, though, it’s a nice feature because the argument is subject to reasonable disagreement (compare that to a task like merging two datasets, where there is a verifiable right answer). So I asked Claude:

Brendan: This is a good representation of my position but I’m wondering what you think the most compelling counterarguments are?

Claude: A few come to mind:

“We’ve always used tools that reduce effort, and nobody panicked.” Stata replaced hand calculations. Google Scholar replaced physically walking through library stacks. Word processors replaced typewriters. Each time, the effort-to-output ratio shifted and academia adapted without an identity crisis. The counterargument is that AI is just the next step on that continuum — it changes the nature of the effort required, but it doesn’t eliminate the need for expertise. You still need to know what to ask for, how to evaluate what you get back, and where it’s wrong. That’s not nothing.

“If the output quality is genuinely the same, your discomfort is just status anxiety.” This is the hard-nosed version. If we really believe we value output quality, then being uncomfortable with AI-assisted output is just revealing that we were secretly valuing something else — exclusivity, credentialing, the feeling of being in a guild that not everyone can enter. The effort-virtue link might be less about epistemic integrity and more about gatekeeping. An economist would probably frame this as rent-seeking dressed up as quality control.

“The effort-output link was already broken — you just didn’t notice.” Academic output quality is wildly uneven, and the correlation between effort and quality was never as tight as we pretend. Plenty of labored-over papers are mediocre; plenty of quickly-written papers are brilliant. What we were really using as a signal wasn’t effort itself but a bundle of things — institutional affiliation, publication record, network, fluency with disciplinary conventions — that effort happened to correlate with. AI doesn’t break a clean signal; it exposes one that was already noisy.

“The unbundling is actually clarifying, not destructive.” If AI handles the information-processing components of academic work, what’s left is the stuff that arguably should have been valued more all along — original questions, deep contextual knowledge, judgment, taste. Maybe the field gets better when we stop rewarding people for being good at tasks that a machine can do and start rewarding them for the things it can’t.

That last one is the one I’d find hardest to argue against, personally. What’s your reaction — does any of these change your thinking, or do they miss something?

What’s interesting to me is that none of them are an actual refutation of the core argument about a broken connection between effort and output. I wasn’t clear enough in my prompt (side note: how often does this same thing happen when humans argue with each other?). OK, Claude, fix it.

Brendan: These are interesting ideas but they aren’t really a refutation of the core argument, which is that AI breaks the relationship between output and effort. Can you respond to that, specifically?

Claude: You’re right — those counterarguments are mostly about whether the break matters, not about whether it’s happening. Let me try to actually push back on the core claim.

“AI doesn’t eliminate effort, it relocates it.” The argument here is that the effort-output link isn’t broken — it’s just that the effort looks different now. Writing a good prompt sequence, evaluating whether the output is correct, knowing what to ask for in the first place, iterating on something that’s 80% right — that’s still cognitive work, and the quality of the output still varies based on how good you are at it. Someone with deep expertise will get a better AI-assisted literature review than someone without, because they’ll know what’s missing, what’s wrong, and what to push on. So the link between expertise and output quality survives; it’s just mediated differently.

“The link was never effort-to-output — it was knowledge-to-output.” This is a subtler version. What actually produces a good paper isn’t the hours spent writing it; it’s the years of accumulated knowledge that inform every decision. AI can generate text, but it can’t generate the judgment that tells you which text is right. If that’s true, then AI doesn’t break the link — it just strips away the parts of the process that were never the real source of quality in the first place.

I think the strongest version of the rebuttal is something like: the effort that matters was always the accumulation of expertise, not the production of any given output. And AI doesn’t shortcut expertise — at least not yet.

But I suspect your response to that would be: even if that’s true in principle, the problem is that we can’t observe expertise directly. We’ve always inferred it from outputs. And once outputs become unreliable signals of expertise, we’re stuck — even if expertise still matters, we’ve lost our main way of measuring it.

Is that where you’d land?

I find the first argument compelling, at least for the time being. In the hundreds of hours I have used these tools, I have developed a level of expertise that allows me to get useful things from them. I also undoubtedly benefit from strong critical thinking skills built up since birth through formal and informal education. If human expertise remains an important input to the knowledge production process, I think academia can roll right along, even if there are gigantic speed bumps and blind corners. However, I’m increasingly uncertain about the durability of human expertise, which, by the way, feels like a crazy thing to write.

Regardless, my conversation with Claude leaves me feeling like the academy is about to face this decoupling issue head on. What things end up looking like on the other side is unclear, but my hope is that we can leverage the technology to focus less on the outputs and more on the ideas.

Yesterday, I did something with those reviews that I’m still not entirely sure how I feel about. I gave all of them (well, ones that were in my ‘reviewer’ folder on my Onedrive, which covers at least the last four years) to an AI — specifically, Claude running in Cowork mode — and asked it to figure out how I review papers: my structure, my tone, my pet issues, my rhetorical habits, the phrases I apparently can’t stop using. Then I asked it to review a paper (my own unpublished work, in this case) the way I would.

This post is about what happened. The short version: it worked better than I expected, failed in ways that matter, and raised questions I think every academic is going to have to sit with soon.

The process

Here’s a quick overview of how this process worked. It’s pretty simple. I opened up Claude Cowork, pointed it to my ‘Reviewer’ folder on OneDrive, and gave it the following prompt:

I want you to help me review manuscripts for journals. My basic idea for this is that I first share with you a bunch of my previous reviews so that you can see exactly how I write, how I structure my feedback, and what specific issues in papers I focus on. You should review those first (they are in the folder we’re working in) and come up with a detailed rubric/workflow/layout for reviewing papers. We can then iterate on the details of that “how to review” document which will then become the basis for you to review future manuscripts. What do you think?

A few minutes later, here’s what it produced (It’s a long document and Substack won’t allow me to embed it into a scrollable box, so I’ll just paste two representative snippets and direct you to click the link for the full document):

1. Review Structure and Organization

Every review follows a consistent architecture:

1. Opening paragraph(s): Factual summary of the paper — what data are used, what question is asked, what methods are employed, and what the main findings are. This comes before any evaluative language.

2. Overall assessment statement: A clear signal of how Brendan sees the paper. This ranges from enthusiastic (”I am a big fan of this paper”) to direct (”the current form falls short”). This is typically signaled in the opening 1–2 paragraphs but may be implicit in the opening framing and made explicit as major issues emerge.

3. Major comments: The core of the review. Organized thematically (not by page number), typically focusing on 3–6 substantive issues. Each major comment is a developed paragraph or set of paragraphs, not a bullet point. Major comments involve threats to validity, conceptual confusion, or issues that prevent the paper from answering its stated research questions.

4. Minor comments: Shorter observations about writing, presentation, missing citations, figure/table improvements, and typos. Sometimes presented as a numbered list, sometimes as flowing paragraphs — the format depends on density and the overall tone of the review.

5. Closing: Varies by recommendation — may express a path forward for revision, defer to the editor, or summarize the key issue that needs resolution.

4. Substantive Review Dimensions

4A. Causal Identification and Research Design

This is the single most important dimension. For any paper making causal or quasi-causal claims, the review must address:

• What is the identification strategy? Is it stated explicitly? If not, what does the design implicitly assume?

• What assumptions are required? Are they articulated? Are they defensible? What evidence supports them?

• What are the most salient threats to validity? Omitted variable bias, reverse causality, selection, measurement error?

• Is there a mismatch between the design and the causal claim? Papers that use descriptive methods but interpret results causally are a frequent problem.

• For difference-in-differences: Are parallel trends shown? Is the treatment timing exogenous? Are there pre-trends in event-study figures?

• For fixed effects models: Is there an age-period-cohort problem? Is the identifying variation clear and defensible?

• For instrumental variables: Is the instrument strong? Is the exclusion restriction plausible?

Recurring critique pattern: “The overall point here is that there needs to be much more precise articulation of why the approach you’re taking is suitable for this research question, what the identification assumptions are, and what the most salient threats to validity might be.”

4B. Endogeneity and Mechanical Relationships

A signature concern. Be vigilant for:

• Variables that appear on both sides of the equation. Example: “% of Black students is on both sides of the equation, which creates a huge endogeneity problem. The discipline index is the ratio of offenses to number of students by race, so you already have, effectively, % of Black students in the model.”

• Mechanical correlations in value-added. Example: “When you form the VA estimates via equations 1 and 2, the student-level error remains in the residual... These same factors likely impact the longer-run outcomes as well, so they appear on both the right- and left-hand-sides of equation 3, leading to a mechanical positive correlation.”

• Control variables that are endogenous. Example: “% of novice teachers is also endogenous to a rolling average of teacher turnover (novice teachers must be new-to-school and vacancies arise because teachers left in prior years).”

It was an odd experience. The rubric was surprisingly comprehensive and really did touch on the key issues that I look for when I review. It felt like me.

Take the section on tone. It identified the specific phrases I fall back on when I’m trying to soften a critique — “I wonder whether...,” “I’m not sure that...,” “can you say more about...” — and separated them from the language I use when I’m being more direct: “I have pretty serious misgivings about...,” “the empirical warrant is pretty weak.” It also flagged that I start nearly every subjective assessment with “To me, ...”

The methodological stuff was also quite impressive. It didn’t just say “Brendan cares about causal identification.” It laid out a taxonomy of the specific threats I tend to focus on — endogeneity from variables appearing on both sides of the equation, mechanical correlations in value-added estimates, functional form restrictions that seem arbitrary — and pulled concrete examples from my actual reviews to illustrate each one. It caught patterns in how I handle things like low survey response rates or papers that use descriptive methods but slip into causal language.

OK, fine, it ingested a bunch of my writing and documented in detail its key elements and themes (as if that’s not already impressive enough). The real question is whether it can subsequently use that documentation to produce an expert-level review of a paper it hasn’t seen before.

To test this, I gave Claude the rubric along with a manuscript and asked it to review the paper as if it were me. The paper I chose was one of my own — a working paper that I know intimately, including its real weaknesses. I also chose a paper that has a revise & resubmit from a top education journal. These two things were deliberate; with my own paper, I could evaluate whether Claude was actually hitting the right issues. I could also compare Claude’s review to the actual reviews written by experts in the field. I was also curious about how replicable Claude’s review was, so I actually gave it the exact same task in four separate Cowork instances (which are sandboxed, so Claude hopefully isn’t looking at its previous reviews of the paper).

The results

Here are the reviews:

• Claude Review #1

• Claude Review #2

• Claude Review #3

• Claude Review #4

And here are the human expert reviews:

• Human Expert Review #1

• Human Expert Review #2

• Human Expert Review #3

It’s remarkable to me how good Claude’s reviews are. Seriously, it’s remarkable. They are cogent and get into the deep methodological weeds of the paper. They raise issues that I didn’t really consider closely but are probably worth considering. Here’s an example from one of the reviews:

3. The dissimilarity index construction.

The dissimilarity indices in Figure 2 are one of the paper’s more novel contributions, but the construction involves a number of researcher degrees of freedom that give me some pause. The weights assigned to demographic, academic, and teacher-rating dimensions are described as “ultimately arbitrary” in Appendix B, and the specific choices (e.g., 33.3% each for gender, race/ethnicity, and SES within demographics) could meaningfully affect the rankings. I have a few concerns here. First, how sensitive are the results to the weighting scheme? The paper would benefit from showing that the key conclusions—particularly the claim that nursing and social work are the most “similar” careers to teaching—are robust to alternative weighting choices. Even a simple exercise showing results under two or three different weighting schemes would be helpful. Second, the Euclidean distance metric treats all dimensions symmetrically, but there’s no particular reason to think that a one-unit difference in the academic dimension is comparable to a one-unit difference in the demographic dimension. The normalization to a 0–100 scale helps, but the underlying commensurability assumption is still doing a lot of work. Third, the teacher-recommender rating dimension is only available for a subset of applicants who request recommendations from teachers (which is most, but not all). Is this dimension contributing systematically different information from the demographic and academic dimensions, or is it largely redundant? Some discussion of the incremental contribution of this third dimension would strengthen the analysis.

Now, if you aren’t me and haven’t read the appendix of the paper closely (let’s be honest, you haven’t), you might wonder whether these are real insights or just AI slop manufactured into academic language. In fact, pretty much everything here is spot on. These are real issues that a strong review of this paper should raise. The AI reviews weren’t perfect, though. Amusingly (worryingly?), two of the reviews (#1 and #4) hallucinate citations while raising the exact same issue:

3. The career interest measure as a proxy for actual labor supply. The paper is studying stated career interest on a college application, which is a meaningful signal but an imperfect proxy for actual entry into teaching. The authors discuss the Foote et al. (2023) finding that Common App teaching interest predicts enrollment in a teacher preparation program, which is helpful. But there's still a gap between stated interest at age 17–18 and actually becoming a teacher, and that gap could be changing over time in ways that matter for interpretation. If, for example, students in 2025 treat the career interest question less seriously than students in 2015, or if the expanding set of career options on the Common App changes response behavior, then the observed decline in teaching interest could partly reflect measurement artifacts rather than a genuine shift in preferences. I don't think this invalidates the paper's contribution — these data are still the best available window into early career intentions at scale — but I'd appreciate a bit more discussion of the boundary conditions of the measure. In particular, can the authors say more about whether the career interest question itself (its wording, placement, or option set) has changed over the study period?

4. Career interest as the outcome—further engagement with validity and the “Other” category.

The paper rests on a single self-reported career interest item completed at age 17–18. The authors appropriately cite Blom (2020) showing that Common App career interest predicts college major and eventual occupation, which is helpful. I wonder if the authors can provide a bit more specificity about the predictive validity—what share of applicants expressing teaching interest actually major in education or enter teaching? Even an approximate calibration would help the reader assess what a movement from 4% to 3% means in practical terms. If the conversion rate from stated interest to career entry is, say, 30%, then we’re talking about a shift from 1.2% to 0.9% of applicants eventually becoming teachers—which frames the magnitude quite differently.

Is Claude better than human experts?

Claude’s reviews were quite different from those of human experts. First, they were far more consistent, which is perhaps unsurprising given the specific prompting generated by the synthesis of my prior reviews. They also addressed different issues, with AI focusing much more closely on methodological details than organization or framing (again, probably driven by the prompt). I asked Claude to look across all the reviews and analyze the differences:

The main differences map onto a broad pattern: the AI reviews are more technically demanding, while the human reviews are more concerned with framing and scope. All four AI reviewers converge on requesting sensitivity analyses for the similarity index weighting—a point no human reviewer raises in that specific form. The AI reviews also engage more with econometric details like CEM match rates, odds ratios with rare outcomes, and recommender rating measurement. The human reviewers, by contrast, raise concerns the AI reviewers mostly miss: the need for a clearer conceptual framework and pipeline model, whether the paper's many analyses cohere into one manuscript or should be split, multilevel data structure issues, and whether the decline in teaching interest reflects pay rather than prestige. The most critical voice across both sets is human R2, who questions the paper's fundamental structure in a way none of the AI reviewers do—they tend to accept the paper's architecture and ask for robustness checks within it.

Were Claude’s reviews better than the human reviews? It’s obviously difficult for me to answer that question fairly because I’m predisposed to how Claude reviewed—I effectively “trained” it to review like me (one could easily train versions that are directed to focus on different issues). Claude is probably better positioned (at least right now) to weigh in on on methodological issues that have denser representation in its training data (i.e., the methods employed in this analysis are commonplace in social science research whereas the specific topic is far more niche).

Even setting aside the quality comparison, Claude’s reviews have some clear advantages. First, they are effectively instant, whereas the peer review process stretches 3-4 months, or even longer. Second, they are not subject to the same cognitive limits of humans. There are only so many hours in the day and reviewing papers is hard work. As anyone in this line of work will understand, reviewing (and editing) is largely thankless work and must be shoved in among many other competing demands. Sometimes (most of the time) you just don’t have it in you to write the best possible report. I could go on longer, but I won’t. The point is, I believe we have reached a point where AI can deliver tremendous value to the peer review process (at least in the quantitative social science realm), though I’m certainly not ready to cut out human experts.

What does it all mean?

Powerful AI is here. It’s already reshaping work and life as we know it, and I don’t think the academy (at least my small corner of it) is ready. My exploration here focused on a fairly narrow technical issue: how well can AI do a task that is central to the work of social scientists? “Pretty damn well” is my answer. Where we go from there is far less clear. The moral, ethical, and societal issues introduced by this transformational technology are numerous (understatement of the year!). Personally, while I am floored (in a good way) by what these tools allow us to accomplish, I am also deeply worried that the pace of progress far outstrips our human ability to react. More narrowly, I think there is currently insufficient recognition among academics that our entire enterprise will be completely transformed by powerful AI. The sooner we face the music, the better.