The AI and History Conference is happening October 15-16 at Johns Hopkins, and registration is open.

This isn’t a conference about whether historians should use AI. It’s a conference for people, like you, who are already doing the work or want to know how to do the work: building OCR pipelines that beat commercial services, training human-in-the-loop machine learning, linking 2.2 million historical places across languages so historical GIS stops being reinvented from scratch every project.

Projects span the globe as well as the centuries. Also we have a cool video game panel!

Some of your favorite substackers are on the program:

Jim Clifford is speaking on cleaning and structuring historical data. Jo Guldi is showing how data methods underpin her upcoming book. Christopher Phillips will discuss what graduate training in computational history should look like. Loren Moulds is live-demoing a multi-pass OCR pipeline. I will, of course, be making sure there is lunch.

Faculty/professional is $100, and it is free for anyone for whom the fee would be a barrier.

https://proflouishyman.github.io/ai_conference_2026

Capacity is filling up quickly. Register today!

Share

There are many tools out there to help manage this situation. Some historians process each document as they go in the archive—though many of us don’t have the luxury of the time this requires. Others use Tropy, an open-source app designed explicitly for organizing research photos. But Tropy lacks built-in OCR, and still requires you to manually group photos of multi-page documents together and add essential metadata, like authorship and year. I started my dissertation using a modified version of Elena Razlogova’s Zotero workflow for the archive, which allows me to add OCR-ed scans of typed archival documents that can easily be tagged, organized, and cited alongside my library of secondary sources. But doing this one document at a time, whether at home or in the archive as I go, is time-consuming. I wish I was a tenured Ivy League professor with a research assistant who could do all this grunt work: grouping photos by document, and saving each document in a database with its title, author, date, type, and citation information, so I could jump into the fun part: actually reading the documents and understanding what they add to my project.

Well, I’m still a lowly PhD candidate, but I have found a research assistant who can do all this work for me, at a shockingly low price of $20/month. His name is Claude.

Share

My AI-enhanced archive workflow

As this Substack has shown, LLMs are not a good substitute for the historian’s essential functions of close reading, critical analysis, and writing. But for extracting structured data from images, they have become quite excellent, and very cost-effective. My archive workflow, diagramed below, takes advantage of this division of labor:

I’ll go through this workflow, step-by-step, but first, a few caveats: This workflow is tailored to me, and you will likely want to modify it for your needs (especially if, for example, the archival material you work with is mainly images or handwritten documents). You also don’t need to use the same apps I use, and you can even have AI just take raw photos from your phone directly and import them into Zotero. And if you want to see how I set up Claude to do just the AI portion of this, skip to Step 5.

Steps 1 and 2: Scan documents + OCR

I’m at the New York State Archives, looking at the Comptroller’s Subject File series. I’ve opened box 64 and pulled out folder 1. There’s a bunch of useful material for my chapter on pension funds and private equity. Time to get scanning.

I scan documents with the iPhone/Android app vFlat: for $4/month it offers automatic adjustment for color, shadows, and even page curvature (essential when I’m working with bound magazine volumes), can do 1 or 2 page mode, and has built-in OCR. But you could just as easily use Adobe Scan or the Camera app. OCR’ing at this stage is also optional–you could do it on your laptop, in Zotero with an extension, or even have the AI do it in Step 5.

After scanning the whole folder, I select all the images in vFlat and choose the “Export to PDF” option. I Airdrop the file to my laptop. Again, you don’t even have to make your images a PDF at this stage. You could just copy the JPEGs to a folder on your computer.

Step 3: Create Zotero item for folder scan

In my Zotero library (I have a different folder, called a “subcollection,” for each day at each archive), I create a manuscript-type item (an item is an entry in the Zotero database, which can have a PDF and notes attached to it) and give it the name of the folder and box number, in brackets (the brackets make it easy for the AI to identify later). “Manuscript” is basically Zotero’s default item type for an archival document. Then I add the folder and box number to the “Loc. in Archive” field and the abbreviation for the archive (in this case, it stands for “New York State Comptroller Subject Files, New York State Archives). This is important: it’s how I’m able to later cite all these documents! You don’t need to add the folder to Zotero now, but I like to just as an extra step to make sure nothing gets lost in the AI parsing process. Just make sure you label the PDF or folder with your scanned folder images with the folder, box, and archive citation information.

Finally, I like to use tags to organize items in Zotero. When I was scanning this folder, I saw that everything in it pertained to the NYS pension funds and to corporate takeovers, so I added those tags to the folder item. When AI extracts the individual documents, it will automatically copy these tags to each of them. This is the core reason to use a database like Zotero: a year from now, if I want to revise my chapter draft and add more on New York, I can instantly pull up every archival document, journal article, newspaper article from Proquest etc with the “ny pension” tag and then sort by date, author, other tags, etc. I can forget about a source, but Zotero remembers.

Anyway, this is what it looks like when I’ve finished adding folder 1, box 64 to Zotero:

Step 4

Repeat this for every folder of every box that I look at which is useful for me. Then when I get home…

Step 5: Run my “Zotero-split-scans” AI skill

This is where the magic happens. Using Claude Code, I created a “skill”—basically, a package of ready-made prompts and scripts—that allows Claude to take this PDF of a whole folder worth of scans and turn it into discrete PDFs and Zotero items for each document. All it took was a $20/month Claude Pro subscription and some back and forth in Claude Code telling it what I wanted the skill to do. Here’s how it works:

1. Download my zotero-split-scans skill from Github and follow the instructions on how to install it in Claude Code or ChatGPT Codex). You will need to have the Python coding language installed on your computer (you won’t need to actually write any code, though), and you will need to generate an API key for Zotero, which is basically a password that allows Claude to do things in your Zotero library. You can also use an AI coding agent to adapt this skill to your own workflow so it can talk to, say, Obsidian or Tropy or DevonTHINK. The skill is just some Python scripts and instructions to Claude on how to use them.

2. Fire up Claude Code and tell it to run the zotero-split-scans skill on the Zotero item created in Step 3:

3. Claude gets to work. First, it finds the “Folder 1, Box 64” entry in Zotero (“NYSA: 2026-06-30” is the subcollection it’s in). Then it extracts the images from the PDF and makes judgements about which pages are part of the same document.

4. When it’s done, Claude creates a summary of how it plans to segment the folder

The skill is designed to categorize documents as either newspaper articles, magazine articles, letters (the Zotero item type for any form of correspondence), or manuscript (everything else). That’s just based on the types of documents that are normally in my archives.

I also set up this skill so Claude would detect handwriting and add, as an attached note in Zotero, its own transcription. OCR is basically useless with handwriting, and while LLMs are not full-proof, it’s really helpful to have their best effort transcript on hand when reviewing handwriting.

5. Claude creates the new PDF files and items in Zotero, adding the relevant metadata (including copying the archive tags I manually added to the folder scan in Step 3). It does a self-check to verify no pages got dropped, and creates a file where I can verify how it segmented the folder. Back in my Zotero, the collection now looks like this:

The original folder scan is still here, but marked as DONE,. And now all the documents I scanned have been organized with metadata that I can search for in Zotero, whenever I want.

Here’s Claude’s attempt at transcribing some gnarly handwriting. Not that helpful, but better than nothing!

I haven’t caught any mistakes with this system yet, but even if I do, it’s easy to update the metadata. What’s important is that I get everything somewhat organized from the get-go. Even if a document isn’t useful for me right now, it’ll be easy to find later if I want to come back to it. For example, If I want to write a standalone article on Henry Kaufman in a few years, I’ll be able to instantly summon all the relevant sources, without needing to wade through disorganized notes or photos.

And because all the documents have been OCRed, I can even use Zotero to search the full text of everything in my library for a particular phrase, even if it’s not in metadata or tagged:

Step 6: Be a Historian

Now that the busywork is out of the way, I have a dissertation to write!

Aaron Freedman is a PhD Candidate in History at Columbia University.

Subscribe now

Yet as we learn to use LLMs, we need to embrace the experimental method. Getting the right combination of model, hardware, and configuration for your particular corpus requires playing around. There is no universal solution.

I want to convey in this long-winded post, if not the glamour and drama of my recent experience, at least the experience itself. I used experiments to figure out how to improve my throughput efficiency orders of magnitude, that is the tokens generated per second by the LLM, while, at the same time, increasing the quality of the results.

I did this by playing around.

Ollama, oMLX, and vLLM

A few posts back, I wrote about the virtues of local models and Ollama. I use it for OCR. The pipeline underneath all of this turns scanned B&O Railroad employment cards, roughly 535,000 of them, mostly from the first half of the twentieth century, into structured data. The first stage (Pass A) reads the image and produces raw text. The second stage (Pass B) takes that raw text and turns it into a proper record: a summary, a set of linked personal and employment fields, a flat table of rows. I’ve found that breaking an OCR task into steps increases quality and speed.

What I didn’t know yet is that Ollama has a real speed limit built into how it works, and getting past it pushed me into using two different solutions: oMLX on my Mac and vLLM on my cluster.

On my Mac, oMLX took throughput from 31 tokens per second (tok/sec) to 182, a nearly 6x increase. On the cluster, a single request through Ollama topped out at around 129 tok/sec. Once I switched to vLLM, those same L40S GPUs pushed aggregate throughput to 345 tok/sec by working on 88 requests at once.

Before I get into what actually happened, in the order it actually happened, it’s worth explaining the mechanism underneath all of it, because it is about software, not hardware.

Beyond Ollama: two ways to build a car

For most of the early history of the automobile, a car was built in place. A team of workers stayed with one chassis from start to finish. They fetched the parts, did every step in sequence, and only once that car was completely finished did they move on and start the next one. Assembling a car this way took about twelve and a half hours. If you wanted to build more cars at once, you needed more teams and more floor space, one team per car, each team locked to that one car until it was done, whether they were actively working on it or waiting on a part.

Henry Ford‘s assembly line did something different. Instead of a stationary car and a team that stayed with it, the car itself moved, continuously, past a sequence of workers who each did one specific task on whatever car happened to be in front of them at that moment. A worker was never idle waiting for an entire car to finish, because a new one was always arriving. Cars entered the line continuously and rolled off the other end continuously. For me, the astonishing thing was the speedup from just reorganization, no technical change: twelve and a half hours down to five hours and fifty minutes, with no new machinery at all, just a different arrangement of the same workers doing the same tasks. In a few years that time would get down to 30 minutes. How you organize your tasks matters.

Ollama builds in place. It hands a request its own fixed slot, and that slot works on nothing else, no matter how long the request takes, until the request is completely done. If you want to handle more requests at once, you need more slots, and a slot’s idle time, waiting on the next token the same way a team might wait on a part, isn’t shared with anything else. This is a completely reasonable way to build a server, and it’s a real part of why Ollama is so easy to get running. It just doesn’t scale any better than a factory floor with one team locked to one car scales.

vLLM and oMLX are both assembly lines. vLLM is the version built for a cluster’s NVIDIA GPUs. oMLX, through a piece of it called mlx-lm, is the version built for a Mac’s Apple Silicon. Neither one waits for a request to finish before starting on another. Both treat the GPU’s compute as a single continuous line that requests move through: a token of this record, a token of that one, a new request joining as soon as there’s room, a finished one rolling off the far end. The industry calls this continuous batching, and it is, functionally, the same idea Highland Park worked out for cars a century earlier.

oMLX saves your cache

Take the exact same 20-billion-parameter reasoning model this pipeline was using in production at the time, gpt-oss-20b, and run it once on a Mac through oMLX, and once on the cluster through Ollama, with nothing else different.

The Mac won handily, which astonished me. The same model weights produced structured JSON at 95 tok/sec locally versus 35 tok/sec on the cluster’s own A100. A laptop beat a data-center GPU on raw generation speed for this task.

Part of that Mac speed came from reusing something called the KV cache. When you prompt a LLM to turn an image into text, it looks something like “turn this image that follows into text: [insert image here]”. In real life it is much longer because LLMs are astonishingly dumb: hundreds of words of “return exactly this schema, use null for missing fields, keep dates as written, don’t be stupid and make things up” that are identical on every single call. If you keep the computation that turns that prompt into AI speak, the KV cache , then you don’t need to recompute it. It makes a big difference in how fast you start getting to that first token of answer.

Across eight real documents, the first call to a fresh prompt took about 21 seconds on average before the first word of output appeared. A second, related call that could reuse a cached prefix took about 12 seconds, a real 1.8x cut in wait time, while the actual words-per-second of generation barely moved either way.

KV caching is a different kind of speedup from continuous batching entirely, even though it is important in my experimental journey. Caching a prompt prefix is about not re-tooling the same station from scratch for every car that passes through it.

Continuous batching is about how many cars can be moving through the line at once.

Why I was even looking at throughput

Fast and correct are not the same thing. And in the world of LLMs, there isn’t necessarily a tradeoff.

The JSON restructuring stage had been failing on 20 to 25 percent of records. Most of that came from that same large reasoning model (gpt-oss:20b) being asked to do what is actually a filing task. On repetitive, tabular records, instead of filing anything, it would often just start copying the raw text back out to the JSON, and burn its entire budget doing what it thought was “thinking”.

The answer was, counterintuitively, a smaller, non-reasoning model, gemma3:4b, that scored better for this exact job. When I swapped to the smaller model, the failure rate dropped to a clean 7 to 9 percent. The speedup, despite the smaller model size, wasn’t that much. I started to wonder if there was a way to cram more models into memory and run them all at once.

What I am foregrounding right here is my ignorance. A nice term would be “beginners mind” but this is ignorance. Yet, because of the magic world of AI, where I can ask “is there a way to cram more models into memory and run them at once” I can find the answer. After some conversation, Claude explained this was called continuous batching.

That said, at no point (so frustrating!) did Claude suggest this as a speedup path when I was toiling over optimization earlier. It focused on answering the specific questions I asked, not getting me to ask the right questions.

Human curiosity wins again.

The Assembly Line and Continuous Batching

Ollama’s fixed slots create exactly the kind of bottleneck I described above with cars. A request occupies a slot and holds it for however long the actual work takes, and the next request in line has no way to start until that slot opens back up. Computer science has its own name for this: head-of-line blocking. It doesn’t matter how much idle capacity exists in other slots. If your slot is occupied, you wait.

Dialing Ollama’s own concurrency setting from 1 up to 24 requests at once, throughput climbed a little, flattened out around 100 to 110 tok/sec by the time you’d turned the knob to 4, and then just sat there. Ollama is not built for concurrency, and certainly not continuous batching.

Look at the two panels side by side. Throughput (left) climbs to about 112 tok/sec by four requests, wanders up to its real high point of 118 at sixteen (with latency already pinned near the timeout ceiling by then), and then collapses to 57 once nearly half the requests are timing out. Latency (right) climbs the whole time regardless, with no throughput to show for it.

None of this is just my one weird machine, either. An independent benchmark, run by Red Hat, tested Ollama and vLLM on an identical NVIDIA A100, same concurrency, both servers. Ollama came in around 41 tok/sec. vLLM came in around 793.

vLLM is built for exactly this situation, allowing a GPU to work on dozens or hundreds of requests at once instead of finishing one before starting the next. But you still need to experiment with different models and configurations (or “knobs”) to figure out the right balance for your particular corpus and GPU. Only through experiments will you discover what works and doesn’t work.

IBM’s granite models were made for reading tables, so I was optimistic, but everything needs testing. I tested many different models, but the ones that had the fastest times and the best results (according to automated testing) turned out to be terrible! Granite3.1:3b scored a perfect 10 out of 10, and was the fastest model in the entire screen by a wide margin. It also turned out to be broken. The sections meant to hold the actual extracted information were quietly empty. Similarly kinds of problems surfaced in real testing.

Put these three models side by side and the actual tradeoff jumps out most clearly.

A disqualified model, llama3-groq-tool-use:8b (orange) has the best-looking throughput number on the entire chart, by a wide margin, but it was just fast garbage. Next, granite4.1:8b (green) is the mirror image: perfect validity on everything that completed, everywhere it was tested, but its completion rate started slipping early, at just 68 requests deep, and it didn’t reach its own best throughput (258 tok/sec) until well after that slipping had already begun. Meanwhile gemma3:4b (blue) is the only one of the three that holds a respectable validity rate across the widest concurrency range tested and keeps climbing in throughput the whole way out to 384. And the quality trade-off wasn’t that much. That combination, not the biggest number in isolation, is what a production choice actually has to be judged on.

Pushed to 384 simultaneous requests stacked onto the same GPU at once, gemma3:4b throughput kept climbing past granite’s own ceiling, settling around 345 to 365 tok/sec.

This tradeoff shows up in the data. Look at how differently the two panels move. Throughput (left) is basically flat after the first jump. It goes from 308 tok/sec at 32 stacked requests to 365 at 384, a gain of under 20 percent across a twelvefold increase in load. Latency (right) does not behave like that at all. Stacking more work onto the card past a certain point buys you almost nothing in throughput and costs you enormously in how long any one record sits in the queue. That asymmetry is exactly why 88 requests was the number I used, not 384. These weird numbers were, obviously, found experimentally. At 88, throughput is already within 5 percent of the eventual ceiling and nine out of ten records clear within about six minutes. Push to 192 and you buy another 5 percent of throughput for more than double the wait, which is always a risk in many different ways. The knee of the curve, not the top of it, is where you want to stand.

Before flipping anything in production, I read six full outputs by hand at the lightest and heaviest stacking levels tested, to confirm the JSON wasn’t just parsing but actually correct. An AI agent independently reviewed the deployment code, checking specifically that the concurrency number I’d chosen actually reached the server rather than being silently capped somewhere downstream (the exact bug that review was built to catch, and did catch once already during construction). Only then did the switch go live: the same small model, now served through vLLM, with 88 requests stacked onto each GPU at once.

What happened once it actually ran

The production chain finished clearing the entire backlog. Across hundreds of thousands of real structuring attempts on the full corpus, the failure rate came in at 1.09 percent, lower than every earlier, smaller sample had suggested.

Put the whole project’s failure rate side by side, in order, and the shape of it is the real point: it never got worse as the sample size grew.

Of 535,278 total transcripts, 514,198 are now fully structured and zero are waiting on this stage. Some images are irretrievable (historical archives!) but most are now processed.

Ollama Still Has Value

Don’t delete Ollama just yet. It has value, just not at scale. Just as you wouldn’t build an assembly line factory to put together just one IKEA sofa in your living room, you don’t need vLLM to do one-off work.

I ran an experiment on the cluster’s L40S GPU to see how the speed vs concurrency trade-off worked for ollama and vLLM.

Ollama actually wins at very low concurrency, faster and with dramatically better latency. The crossover happens right around concurrency 8. From there the gap only widens: by 88 requests, vLLM is at 353 tok/sec and still climbing, while Ollama has flattened at ~234 tok/sec and stopped.

Ollama is fine, even preferable, for light work, and stops scaling entirely the moment real concurrent load shows up.

oMLX: The question I left open, answered

Once I realized what vLLM could do on the cluster, I returned to oMLX on my mac and realized that I hadn’t fully read the documentation. oMLX could also do continuous batching.

I used gemma3:4b this time instead of the original gpt-oss-20b. One request at a time, then two, then four, climbing toward as many as the machine could plausibly hold. I just kept adding and adding, only backing off after failure. The experiments produced these curves.

Up to 32 requests at once, continuous batching on the Mac did exactly what it was supposed to do. Throughput climbed, noisily but really, from 31 tok/sec at one request to 182 at 32, nearly sixfold. Pushed to 36, the entire server died. Tried again at higher numbers with an identical crash.

That’s a genuinely different failure shape from everything else in this post. Ollama’s ceiling was a plateau of stagnation. vLLM’s ceiling, even pushed to 384 requests on a dedicated cluster card, was soft: validity dipped a little at the very top of the range, latency grew a lot, but nothing crashed. oMLX’s ceiling was catastrophic.

So the honest final answer to the question I opened this post with: yes, the Mac has continuous batching, and yes, it delivers a real throughput gain when you actually turn it on, on the order of what Ollama’s own concurrency knob managed, nowhere near what vLLM got out of a dedicated card.

But the ceiling on a single Mac is real, and a great deal less forgiving than anything on the cluster side of this story.

One Mac was never going to out-throughput ten GPUs on the cluster. That is not surprising.

Yet, astonishingly, oMLX on the Mac is better than Ollama on the cluster.

Fordist AI

I opened this post with Henry Ford, and here’s the detail from this project that made it click for me.

Under oMLX on my own Mac, each individual request’s share of the machine fell from 31 tok/sec running alone, down to about 6 tok/sec once 32 requests were all in. And yet the system, taken as a whole, went from 31 tok/sec to 182. No individual worker got more productive. What changed was how much work was moving through the line at any one moment, and how little of that line was ever standing still.

Nobody’s hands moved faster at Highland Park either. The productivity was never in the person, it was in the choreography: never letting a station sit idle waiting for one car to finish when another was already waiting to roll into place. Stacking eighty-eight requests onto one GPU is the same idea wearing different clothes.

It took two different machines and two different servers, a Mac and a cluster, oMLX and vLLM, to learn that old lesson again.

Share

A few months ago, Loren Moulds wrote that you deserve the cluster. He noted that the quiet assumption in the humanities is that high-performance computing belongs to physicists or geneticists but not to us, even though we too have needs for powerful GPUs to do our work (like OCR!) If you’re affiliated with a research university, you almost certainly already have access to GPU-equipped machines that can turn a faded, ink-stained index card into structured, searchable data.

Just as a reminder, a cluster is a group of many networked computers that’s managed as one shared system, but from your seat it looks and feels like logging into a single normal machine — the “login node” — even though the actual work happens on separate “compute nodes” that you’re allocated for a job and never see or touch directly, each provisioned for a specific purpose (some with GPUs, some just for moving files, and so on).

The JHU cluster called Rockfish has nodes for different purposes.

That post left a question hanging, though. Fine, I have access. Now what? Actually using the cluster is more challenging than your Mac. To run meaningfully-sized jobs, you need to use the colorfully-named Slurm. Slurm is a job scheduler so that users can share those GPUs. Job schedulers have their own mystifying vocabulary: partitions, arrays, dependencies, QOS. You don’t want to spend a month learning all that. Learning to run a shared supercomputer really was, until recently, a specialized skill, even apart from learning to code—and there were not nearly as many easy-to-use YouTube videos online.

Like many other topics in the era of AI, the most important part is knowing something exists. You do need to know why Slurm is and what it does: a shared cluster has a finite number of GPUs, many people want them, and a scheduler is the thing that queues everyone’s jobs and hands out turns. What you don’t need is the operational fluency with the actual syntax, the flags, or, crucially, the troubleshooting skills when things inevitably go wrong. I have spent the last several months running a large OCR pipeline with ~500,000 bureaucratic documents. When I first started this project, I had to learn slurm. Just this past week I learned that, in fact, no one does.

Claude can manage the cluster for you, and in doing so, the cluster can become an easy extension of your laptop.

Share

The one thing you actually have to do yourself

The first thing you’d expect to be annoying—and it is—is just logging onto the cluster.

You need to get a special login and password. My university has a webpage for this, yours does too, and I am sure it is just as confusing.

I found it utterly baffling when I started reading it for the first time. The instructions are not at all clear for the novice.

So let me explain how it actually works.

The way you would normally connect to the cluster is through ssh. SSH (Secure Shell) is a network protocol that lets you securely log into and run commands on a remote computer over an encrypted connection. You run it from the command line by typing ssh yourusername@login.university_cluster.edu and then typing in your password. Then you are connected to the cluster at a new command line.

My login to the university cluster

Now if you look closely at the image above, you will see that I did not enter any passwords (sorry mr. hacker). My cluster is named rockfish, so I have my config file setup to allow me to just type ssh rockfish to login. Using the below, you would type ssh cluster

This works because I did two things, both of which sound scary and impossible but are actually quite easy to do with Claude: setup an SSH client configuration file and created a key-pair.

The configuration file allows you to have a shorter name, like cluster, for login, which is nice, but the real handy part is the “multiplexing”. It allows multiple connections from your computer to use the same channel without re-logging in. If Claude is doing a lot of little things, this part will make your life easier.

The information that will be added to your ssh config file

The prompt for Claude

To make this work you need a key-pair, which is actually a key on your computer and a lock (called a public key) on the cluster. If you have the key, then the cluster will know it is you. You generate a key-pair, put the public one on the cluster, and keep the private one on your computer. Universities usually have arcane instructions on how to do this. I would suggest asking Claude to do it for you.

You will still need to enter your password when it logs into the cluster, but only that once. Once you have the private key on your computer and the public key on the cluster, you never need to enter your password ever again.

And neither will Claude.

Claude on the Cluster

Claude remains on your computer, but it can run the cluster for you.

Using ssh, Claude can now run commands on the remote cluster through the same tool calls it would use on your own computer. Check the job queue, submit a job, read a log file: these are just commands that happen to execute somewhere else.

This matters more than it sounds like it should, because clusters are full of small institutional quirks that have nothing to do with your research and everything to do with how the system happens to be wired. Those transaction costs add up, especially when you might only use the cluster every month or two. I used to have a cheatsheet of all the weird configuration vocabulary that I needed to re-remember. A special command to find software. A special command to request a node. A special command to see what was happening with my slurm. It went on and on.

Now I say what I want done, and Claude issues the actual command, on the actual machine, with the actual details filled in.

Make sure Claude has read the cluster’s documentation so that you are not breaking any rules, and then it is off to the races.

Claude oversees your processes

Once you’re past login, the harder problem starts, because a shared cluster is a live, noisy, multi-tenant environment, and it misleads you in specific ways if you don’t know to check.

Just yesterday I was running ollama on my cluster on ten different GPUs. Each job had two GPUs. Ollama works by claiming a “port” — a numbered address on the machine that says, in effect, “send requests for this job here.” The problem is that a port can only belong to one process at a time, and nothing stops two separate jobs on the same machine from reaching for the same one. Clearly I want my code to use all the GPUs I had requested. But along the way, as is often the case, there was a mixup.

From a certain perspective my job looked fine. On a shared node, a naive check is misleading: the top-level view of a compute node can reflect someone else’s job as much as yours, so “this GPU looks idle” can simply mean “I’m looking at the wrong process.” Claude knew to ask the scheduler directly which GPUs belonged to which specific job, rather than trusting a general glance at the machine, and found the real story: two of my jobs, running on the same physical node, had both grabbed the same network port for their own local model server. One of them lost the collision, silently, and had been running its entire workload against the other job’s models for hours, leaving its own allocated hardware untouched. Claude traced the cause, rewrote the startup script so each job claims its own port safely even when several land on one machine at once (what is called a race condition), and redeployed it, and I watched the previously idle GPU come back to life on the next run.

That is not a story about AI being clever in the abstract. It’s a story about a specific, boring, real bug, of the kind that eats a research budget’s worth of GPU-hours if nobody catches it, caught because the diagnostic habit, ask the scheduler what’s actually yours instead of trusting appearances, is one Claude already knows to reach for.

The same habit shows up in ordinary code. A script that walked my archive’s directory tree to find unprocessed images had, for weeks, simply never finished, because it was checking one file at a time against a networked filesystem, and a networked filesystem is slow per request in a way a local disk is not. Rewritten to check many directories at once, it finished in under three minutes. I didn’t ask for that fix by name. I said the scan was too slow, and Claude found where the time was actually going.

The same wall showed up again later, wearing a different tool. Pulling hundreds of thousands of small files back down to a local drive, a plain file-copy command over the network stalled completely, doing nothing for minutes on end. It looked like a different problem, but the diagnosis was the same instinct as before: check how much actual work is happening, not just how much time has passed. The command was stuck for the identical reason the old scan was stuck, one file at a time against that same slow-per-request filesystem, just inside a different piece of software this time. The fix was the same idea too, list everything fast in one pass, then move many files at once instead of one, and a transfer that had made zero progress in hours finished in a few minutes.

The Claude that delivers wizardry locally, can now do so on the cluster.

Moreover, Claude can submit the work itself. It can write and submit all the slurm that you desire. And even create “slurm chains” so that when your process cuts off after 24 hours (as mine does), it automatically requests another job. A multi-day run now survives the cluster’s walltime limits, requesting the GPUs and the partition that fit the task, without me ever opening a job script.

Claude keeps watch

The last piece is the one that made me trust the whole arrangement, because a run that processes hundreds of thousands of pages takes days, and the ordinary failure mode of unattended computing is that you launch something, walk away, and find out much later, expensively, that it died on the first hour, or wasted resources, or something equally irksome.

Claude keeping watch

Instead, Claude keeps watch. On my Mac, I ask Claude to use caffeinate so it doesn’t turn off, and then I ask it to check on everything hourly. It parses the raw log lines for actual throughput. It works out how many images are left against how fast the whole array is moving together. It gives me an honest estimate of when it will finish. It spot-checks that the GPUs are truly working, the same check that caught the port collision above. It texts me a plain-language update over Telegram. When things are fine, it says so in two lines. When something regresses, it says that plainly too, instead of reassuring me that everything is probably okay.

Clusters are still a hassle

I want to be honest: things still go wrong. The slow directory scan, the port collision, the walltime limits, these were real problems with real causes, not smoothed-over inconveniences. What changed is not that the difficulty vanished. It’s that the difficulty moved to where I could actually reach it. Fixing a race condition on a shared cluster used to be a separate professional skill I had to learn before I could do actual history. Now that debugging happens inside the same conversation as everything else I do with Claude.

Which brings me back to where the last post left off. You deserve the cluster! The resources are closer than you think, and the learning curve to reach them is shorter than the one you already climbed to get into the archive in the first place.

Claude runs the cluster. You do the history.

Subscribe now

Share

A recent post on Substack announced a GitHub-based working papers site for computational history under the cheerful banner “Journals are dead, up with GitHub”. Jim Clifford and Jo Guldi’s diagnosis is familiar: peer review is slow, the field moves fast, and scholars need a venue that matches the tempo of their work. We welcome the experiment. But whether journals are dead or not is beside the point. New forms of publication are not a fix for what ails digital history and digital humanities; they are digital history and digital humanities.

Subscribe now

Share

Over the past three decades the field has kept inventing publication forms because its epistemic needs keep shifting; work with digitized materials demands one kind of container, digital methods another, and digital uncertainty something else again. But the causality runs both ways: because many DH practitioners live deep in the technology of scholarly communication itself, their publishing experiments do not merely accommodate the field’s needs but anticipate them, building new forms for scholarship. What follows here is another history of DH publication, shaped by our experiences in North America and Western Europe. It is incomplete; that imperfection underscores our point.

The project era: publishing beyond the article

Starting in the early 1990s, the “publication” was often the project itself. Edward Ayers began the Valley of the Shadow at the University of Virginia in 1993, an archive and argument about two Civil War communities initially published on the web and later on shelves. A year later Roy Rosenzweig founded the Center for History and New Media, which treated CD-ROMs, websites, and eventually software as legitimate outputs of historical scholarship; the American Historical Association’s first digital history award, established in 2009, is named in his honor.

The traditional venues noticed. In December 2003 the American Historical Review published Ayers and William G. Thomas III’s “The Differences Slavery Made”, a born-digital article whose argument depended on its hypertextual form. The discipline’s flagship journal was already experimenting with digital-native publication more than two decades ago. These projects demonstrated that digital forms could produce historical insight; once scholars were making work that did not fit the article or the monograph, the field built venues that could review it, credit it, and preserve it.

An ecosystem, not a monoculture

That building happened on top of a lineage older than most people realize. Computers and the Humanities launched in 1966; Literary and Linguistic Computing arrived in 1986 and was relaunched in 2015 as Digital Scholarship in the Humanities. By the time anyone was saying “digital history” or “digital humanities,” the field already had decades of journal publishing behind it.

The 2000s added open access, vastly expanding reach and lowering barriers to entry, at least in theory. Digital Humanities Quarterly, founded in 2007 under the Association for Computers and the Humanities (ACH), welcomed experimental formats from the start, at no cost to authors or readers. Then came the experiment that most directly anticipates Clifford and Guldi’s proposal: in 2011 the Rosenzweig Center launched the Journal of Digital Humanities, which aggregated the field’s best “gray literature.” Blog posts, talks, and works in progress surfaced through Digital Humanities Now were selected, refined, and published quarterly by a team largely of postdocs and PhD students. Many of those posts effectively functioned as preprints, later to be converted into articles and chapters. (This JDH should not be confused with the later Journal of Digital History, to which we will return.)

Platforms for multimodal argument

Journals formed only one axis. The field also designed publishing systems for arguments that could not be made in linear prose. USC’s Vectors (2005), commissioned scholarly works built as interactive multimedia from the ground up; its lessons fed directly into Scalar (2013), which gave individual scholars a platform for long-form, media-rich, non-linear publication without a development team. Omeka (2008) did the same for digital collections and exhibitions. The Programming Historian (2012) made even the how-to a citable, peer-reviewed genre. PressForward (2013) powered DH Now and JDH. Zotero added self-archiving of one’s publications and WIP in 2015 (Sean is co-CEO of Digital Scholar, which develops Omeka, PressForward, and Zotero). Manifold (2017) was explicitly designed to publish work-in-progress and solicit feedback.

Not every platform thrived. Vectors wound down; JDH paused in 2014. But turnover should not be reduced to fragility. Each experiment left infrastructure, precedents, and lessons that the next one could build on.

The presses and the profession join in

By the mid-2010s the most traditional gatekeepers were expanding too. Stanford University Press, with Mellon funding, launched a digital publishing initiative to put interactive works through full press peer review; Nicholas Bauch’s Enchanting the Desert appeared in 2016 as its first born-digital monograph. (For Lauren, her co-authored SUP project Layered Lives was key to her tenure case.) The discipline built its own scaffolding: following the Modern Language Association’s lead, the AHA’s 2015 Guidelines on the Professional Evaluation of Digital Scholarship by Historians instructed departments to evaluate digital work in its native medium; the AHR began reviewing digital projects alongside books.

The venues kept multiplying. The Journal of Cultural Analytics (2016) gave computational work an open access home before “computational humanities” hardened into a label; Reviews in Digital Humanities (2020) built peer review itself into a publication, so a database or mapping interface could accumulate the same documented reception as a monograph. One survey of the landscape counts around forty exclusively DH journals and another twenty significantly DH-oriented ones. Diversity of output demanded diversity of evaluation. The field supplied both.

The newest layer: proceedings, layers, and working papers

Which brings us to the venues the Substack post presents as a rupture. Current Research in Digital History (2018) deliberately imported the proceedings model from computer science, publishing short peer-reviewed papers tied to an annual conference. Anticipating the need for rapid and citable peer-reviewed scholarship, the Computational Humanities Research conference has published open proceedings since 2020, and the model proved durable enough that Cambridge University Press now publishes Computational Humanities Research as its official open access journal; authors are even encouraged to iterate and expand on their conference short paper to develop a research article (Lauren is one of its editors-in-chief). In 2025 ACH launched the Anthology of Computers and the Humanities, a home for peer-reviewed conference and workshop papers, drawing on publishing models from the Association for Computational Linguistics and IEEE. Proceedings culture did not kill the humanities journal; it produced a new one.

And journals themselves are being reinvented rather than buried. Since 2021 the Journal of Digital History has published layered articles: a narrative layer for the argument, an analytic layer for the methods, a data layer of executable notebooks and code (Sean is a member of its editorial board). The journal anticipated and now publishes exactly the kind of computational work Clifford and Guldi want to accelerate; it simply refuses to pretend such work fits in a PDF.

Reflection along the way

The field has also studied itself. Joan Troyano, Stephanie Westcott, and Jeri Wieringa published white papers on the publication workflows of DH Now and JDH, including clear-eyed discussion of their chokepoints; Frédéric Clavert and his team have reflected on the Journal of Digital History‘s challenges; Gianmarco Spinaci, Giovanni Colavizza, and Silvio Peroni have mapped where DH publication actually happens. Overstating novelty risks erasing decades of creativity, experimentation, and labor of our colleagues.

Committed roots and new branches

None of this is a reason to reject a new working papers site. Rather, the working papers site is not an escape from this tradition; it is the tradition, doing what it has always done. And anyone who works with version control knows what a repository rewards: not the dramatic gesture of deleting what came before, but the patient accumulation of commits, each building on the last, with a history one can always consult. Nothing more and nothing less. Digital humanists have spent three decades learning how to peer review a database, to preserve an interactive argument, to credit collaborative labor, to keep an experimental venue alive, and to work beyond for-profit platforms. That knowledge is on offer. We hope the invitation is not “come bury the journal with us” but something far better: to contribute to a field that has been publishing inventively for over thirty years, to draw on its history, and to add a branch to it. There is plenty of room; there always has been.

Computational History welcomes debate. Read something you disagreed with? Write a rebuttal!

Subscribe now

Share

Share

A few digital humanities scholars have been saying that linked open data is the right idea for historical research for fifteen years, and it has under-delivered on its promise. The reason isn’t that the standards were wrong or the vision was misguided. It’s that the practical cost of grounding unstructured historical sources to shared identifiers was too high. Grounding, in this context, means linking the people, places, organizations, and concepts named in your sources to permanent identifiers that other researchers also use, so that your “Philip Wodehouse” and my “Philip Wodehouse” are demonstrably the same colonial governor, not just the same string of letters. You either invested in hand-curating those links and stayed small, or you skipped the work and produced a dataset that couldn’t talk to anyone else’s. Most of us did the second thing.

Two Services, One Empire: career transfers of British colonial and India Office officials, from the 1820s to the 1960s, drawn from the interactive atlas. Every arc lands where it does because the place was grounded to Wikidata.

That constraint is loosening. Large language models combined with retrieval against Wikidata, the open identifier system that now contains over 120 million items, including millions of historical figures and places, make grounding tractable at the scale of a single researcher with a corpus. The bottleneck has moved from labor to judgment. The clearest way to show what that buys is the part of my own project that should have been hardest: putting every posting on a map.

An atlas built on borrowed coordinates

The structured corpus became an interactive atlas of imperial careers, which plays the more than twenty-two thousand recorded moves between one colony, presidency, or province and another out across a map and through time. To draw a single arc on that map, I needed a latitude and longitude for every place an official was ever posted to. That is the part historians underestimate. The places in these volumes are not modern cities with tidy coordinates. They are nineteenth-century polities: the Colony of Victoria, the Central Provinces, Baluchistan, British Guiana, a princely state like Jodhpur or Chamba. A modern gazetteer either does not contain them or quietly mislocates them to whatever shares the name today.

Wikidata solved this almost for free, and it solved it in two ways. Many of the places I named carry a coordinate location directly, so grounding the surface string to the right Wikidata item handed me the coordinates in the same step. The polities that do not carry their own coordinates almost always link to the place that does: their capital. British Raj resolves to Calcutta and then Delhi; Canada to Ottawa; a princely state to its seat. Once each place in my vocabulary was grounded to a QID, a single query returned coordinates for the entire set, either from the item itself or from its capital. Weeks of manual gazetteer work that I had budgeted for never happened. The technical specifics of how the grounding pipeline disambiguates against millions of candidates, and how I kept the model from inventing identifiers, are in a companion working paper; the point here is the result. The map exists because the coordinates were already in the commons, attached to identifiers I could borrow.

It is worth pausing on how different this is from how I used to work. Trading Consequences, a project I was part of in the early 2010s, processed ten million pages of nineteenth-century commodity sources and located the places in them with the Edinburgh Geoparser, a deterministic system built on a fixed gazetteer. It was the right tool at the time and the team was strong. It was also brittle in the ways historical sources punish: a modern gazetteer, poor OCR, ambiguous toponyms. That took a two-year grant and a big team. The atlas took one historian, Claude Code, and a method that leans on Wikidata for the part that used to be the bottleneck.

What individual scale costs us

Computational history is now possible for individual scholars without grant-funded labs. That is the headline most readers take away, and it is correct. The harder question is what we do with the freedom.

Coding agents are empowering thousands of historians to build new datasets, but this risks a flourishing of data silos. If we all assign different identifiers to distinguish Victoria the Queen from Victoria, British Columbia, we end up rebuilding the same disambiguation work in every project, and losing the chance for our datasets to compose into anything larger than themselves.

This is the problem linked open data was designed to solve. The idea is straightforward: if every dataset attaches the same persistent identifier to the same entity, then datasets compose. Wikidata has already done much of this work, and my own place vocabulary shows why that matters. Take “Victoria,” a posting that turns up constantly in these volumes. It might be the Colony of Victoria an official governed, Victoria in British Columbia he was transferred to, Victoria on Hong Kong Island, a district at the Cape, or a station on Lake Victoria, before we even reach the queen the rest were named for. String matching cannot tell these apart, and a wrong guess drops an official’s whole career in the wrong hemisphere. Wikidata has already done the disambiguation: each Victoria is a distinct item with a permanent identifier, and the colony, which carries no coordinates of its own, points to its capital at Melbourne, which does. Grounding the right Victoria is what put each posting in the right place.

My atlas already shows what composition buys. It surfaces 178 officials who served in both the colonial and the Indian services, the kind of figure who falls between two literatures because no single archive holds the whole career. Philip Edmond Wodehouse is one of them: Governor of British Guiana, then Governor of the Cape Colony, then Governor of Bombay. A historian of the Cape and a historian of British India are unlikely to know each other’s work; their archives are in different buildings and their conferences do not overlap. But if both ground their data to Q1800510, their datasets join automatically. A query about Wodehouse’s career reaches across one historian’s records in Cape Town and another’s in Bombay, without either of them having planned for the integration. The promise of computational history is not that everyone produces their own dataset. It is that many small datasets compose into a research commons larger than any of them.

Ground, do not host

There have been concerns in our community about using Wikidata as scholarly infrastructure, and the worry has a real history. Wikidata is crowdsourced; its data model is determined by its community of editors; some of its modeling decisions, the handling of gender is the canonical case, have been incompatible with the priorities of researchers working on people whose lives the standard categories misrepresent. The response, in projects like LINCS (Linked Infrastructure for Networked Cultural Scholarship), was to build proper scholarly infrastructure with controlled vocabularies, considered ontological commitments, and editorial accountability. Those choices were correct, and I remain hesitant about building knowledge graphs on the Wikidata platform.

But the worry applies to hosting scholarship on Wikidata: treating Wikidata itself as the place where your conclusions live, where your interpretations are subject to revision by anyone who edits the relevant pages. It does not apply to grounding scholarship to Wikidata, which is a different operation entirely. LINCS already works this way. The graph LINCS publishes is its own linked open data, carefully modelled, with editorial accountability to the scholars who contribute to it. It uses Wikidata QIDs directly as the identifiers for entities Wikidata already covers, and mints its own only when no alternative exists. The model is sovereign; the identifiers are shared.

That is exactly the line my atlas walks. The career graph behind it is mine: my OCR, my extraction, my judgments about which biography matches which annual record. Wikidata never sees it. What I took from Wikidata was the identifiers, and through them the coordinates. The data is sovereign; the entities are addressable. This implies periodic verification rather than set-and-forget: QIDs are durable, but the data behind them can be merged, split, or revised, and grounding is a relationship that needs occasional maintenance.

The reason this matters is that the alternative does not scale. No scholarly project, however well-funded, can produce identifiers and coordinates for over 120 million historical people, places, organisms, events, and concepts on its own. Wikidata has done that work. Its coverage is uneven and its modeling has problems, but it is the only system at the right scale for historical research as actually practiced. Ignoring it produces silos. Grounding to it, while keeping your own dataset, is the move that gets us out of this.

Teamwork without teams

The conventional model assumes that data-intensive history requires a team: a principal investigator, a postdoc, a developer, a project manager, a grant. That model still works for some projects, and the funding agencies are organized around it. But the recent shift means a great deal of useful work can now happen at an individual scale. The question is whether that individual work is connected to anyone else’s.

If we ground our work to shared identifiers, it is. A historian working on Bengal in 1850 and a historian working on Jamaica in 1840 do not need to coordinate, share infrastructure, or even know each other to produce datasets that interoperate. They need to use Wikidata identifiers for the people, places, and institutions in their sources. That is teamwork without a team, and it is exactly the kind of distributed, low-overhead collaboration historians have always done at the level of citation and footnote, now extended to structured data.

This is not a call for centralized infrastructure. It is the opposite. It is a call for a small set of shared conventions: use shared identifiers; ground to Wikidata; create new Wikidata entries when the substrate is thin. This lets individuals and small groups produce work that connects to a research commons without anyone having to build the commons explicitly. The commons is the consequence of the convention.

What this requires

Three things, and they are small.

Add Wikidata identifiers to the entities in your datasets. If you have a spreadsheet of historical figures, add a column for their Wikidata identifier. If you cannot find an identifier for an entity, create one. The lift is hours, not weeks, and the tools to do it are now within reach of any researcher with a corpus.

Publish your data with the identifiers attached. A project website, a Zenodo deposit, a GitHub repository: the venue matters less than the principle that the identifiers are present and durable, so that someone reading your work in ten years can still find them.

Treat Wikidata-editing as part of historical practice. When you find that a colonial administrator, a rural township, or an eighteenth-century concept lacks an entry, contribute one with sources. We need to make this an evolving standard of methodological rigor, on the same continuum as proper archival citation: a small permanent addition to the substrate that makes everyone else’s work more findable, including yours.

The technology has changed. Computational history at an individual scale is real now; my atlas is one ordinary historian’s proof of it. Whether the field produces a research commons or a thousand silos depends on whether we adopt shared identifiers. We can. The conventions are small enough that no one needs permission, and large enough that they would change the field.

Subscribe now

Note: Revised July 3, 2026. After some helpful exchanges, the authors of this post requested it be revised to better reflect their belief in fostering a greater variety of venues for the dissemination of ideas.

Digital humanities has long pushed against the boundaries of conventional scholarly publishing. Over the past two decades, journals, monographs, blogs, code repositories, and collaborative laboratories have all contributed to the field’s intellectual life, each with different strengths. The emergence of generative AI has accelerated the pace of methodological innovation still further, creating new demands for publication formats that can circulate ideas, software, datasets, and experimental workflows while they remain current. This working paper series grows out of that need. It is intended not as a replacement for journals, but as a complementary venue within an expanding ecosystem of scholarly communication.

The era of AI portends the transformation of the historical discipline. In previous generations, the promise of digital history was limited to a few expert scholars with metagrants capable of supporting lab-sized research; the founders of the journal have been among those practitioners. What has changed now with AI, as Cameron Blevinshas pointed out, is that the old bottlenecks of digital history have largely collapsed. Tasks that once required teams, funding, and technical specialization—transcribing sources, structuring data, writing code—can now be handled by AI systems that combine vision, language, and reasoning.

Subscribe now

The implication is profound: the constraint is no longer whether historical materials can be digitized or analyzed, but whether we can ask good questions and translate them into workflows. In that sense, we are all becoming computational historians, not because we all code, but because computation is now embedded in the basic act of working with sources. Any dataset can, in principle, be made machine-readable; any archive can be queried, structured, and modeled. The shift is not just technical but epistemological: the frontier of historical research is moving from access and processing to interpretation, design, and judgment. The emergence of generative AI has altered who can participate in scholarly production. Tools for drafting, translation, and stylistic transformation lower barriers for scholars who have historically been excluded from academic writing: dyslexic researchers, multilingual scholars working outside their first language, and students whose intellectual contributions exceed their fluency in disciplinary prose.

If the bottlenecks have shifted, from access and processing to interpretation and design, then our institutions need to catch up. We are producing more, faster, and in more varied forms than the traditional publication ecosystem was built to handle. The problem is no longer just how to do computational history; it is how to share, evaluate, and circulate it.

This working paper series formalizes that agenda. It provides a venue for rapid, serious, and citable contributions that reflect the pace and diversity of contemporary research. No single publication format—journal article, monograph, technical report, code repository, dataset, or working paper—can accommodate the full range of outputs now emerging from computational history.

We see this series as complementing, rather than replacing, an increasingly rich publication ecosystem. Journals such as the Journal of Digital History have pioneered transparent and executable forms of digital scholarship, while established venues including Historical Methods continue to provide essential spaces for peer-reviewed methodological research. Our aim is different but compatible: to create a venue where ideas, methods, pedagogical experiments, datasets, benchmarks, and software can circulate rapidly, invite community feedback, establish priority, and evolve before, alongside, or independently of more formal publication.

Share

Publishing through GitHub reflects that goal. The platform allows contributions to include prose, code, data, version histories, and interactive demonstrations. The medium reflects the argument: computational history is collaborative, heterogeneous, and iterative, and its publication infrastructure should be equally diverse.

A working paper series can serve scholars at different career stages and with different goals. For established researchers, it offers a venue to document and cite methods without always diverting the sustained attention that a peer-reviewed digital methods article requires. For early-career scholars the calculus is different: building a publication record in peer-reviewed venues rightly takes priority. For these scholars, a working paper might be a first draft of a piece being prepared for peer review as a way to circulate ideas, get feedback, and establish priority while the formal process runs its course. Or it might stand on its own as documentation of the methods that accompany a contributor’s core historical research. Most history journals have little appetite for extended methods sections, and authors are often asked to cut precisely the technical detail that makes computational and digital work reproducible. A working paper offers a citable home for that material, allowing the journal article to focus on historical argument while the methods remain accessible to readers who want to scrutinize, replicate, or build on them.

An Intellectual Agenda

A distinct body of work has emerged across several labs in recent years, one that demands its own intellectual space. This work sits at the intersection of artificial intelligence, political economy, global history, and the theory of history, and it is driven by a shared methodological problem: how to extract, at scale and with fidelity, information about changing ideas from historical text.

There is, right now, a consequential conversation about the long arc of human history unfolding largely outside the discipline of History. Figures like Steven Pinker, Peter Turchin, and Nassim Nicholas Taleb are advancing sweeping claims about violence, stability, and social change using large-scale quantitative datasets. These arguments are influential precisely because they speak in the language of generalization: they offer answers about what is increasing, declining, or recurring over centuries. And yet, for the most part, historians—especially social historians—have not been central participants in this debate.

That absence matters. Because the history of capitalism, political economy, and social life has never been reducible to counts of wars, prices, or populations alone. Scholars like Maxine Berg, Sven Beckert and Jason W. Moore have shown that large-scale transformations emerge from the interaction of institutions, labor regimes, ecological systems, and cultural meanings that evolve over time (Berg 2021; Beckert et al. 2021; Beckert 2025; Berg and Hudson 2023). Capitalism is not just growth rates; it is plantation economies, imperial networks, and the contested organization of labor and land. Moore, in particular, reminds us that capitalism is always already a way of organizing nature as well as society—binding together energy, environment, and exploitation. Climate change, in this light, is not simply an atmospheric trend but a historical outcome of these intertwined systems. These are social and ecological processes, not just numerical trends.

Purely quantitative approaches, for all their reach, risk a kind of abstraction that flattens these dynamics—relying on incomplete or uneven datasets and mistaking what can be counted for what must be explained. In that sense, they engage in a form of cherry-picking of their own: privileging the measurable while overlooking the structures and experiences that give those measures meaning.

What is needed is not a rejection of scale, but a different way of achieving it.

This is where digital history—especially when augmented by AI—offers a path forward. By combining large-scale datasets with the analysis of language, narrative, and representation, historians can engage these long-run debates on their own terms. We can track patterns across centuries while still attending to how institutions form, how ideas circulate, and how people experience and interpret change. In doing so, we do not simply add nuance to existing claims. We reshape the questions themselves—bringing the full social and historical complexity of the past back into conversations that urgently need it.

Addressing this problem requires a longue durée perspective—decades, even centuries—and a global archival base. The sources are correspondingly expansive: imperial records such as the British Colonial Office Lists and Parliamentary Papers, the archives of the Dutch East India Company, and the administrative records of the Spanish Empire or the full run of the Tropical Agriculturalist and the Der deutsche Kulturpionier; for the modern period, the papers of international organizations, philanthropic foundations, and corporations; for the present, global datasets such as those produced by the IPCC and the UNFCCC. What unites these archives is not an interest in institutional elites alone, but the fact that tracing how societies have understood their relationship to labour, land, and resources, across space and over time, requires reading at a scale that exceeds any individual researcher. Only by assembling and analyzing such large-scale, multilingual archives can we connect the pattern-seeking ambitions of longue durée quantitative analysis to the historically grounded questions posed by Sven Beckert, Maxine Berg, David Harvey, and Jason W. Moore—linking abstract trends to the evolving social, institutional, and ecological processes that produce them.

Exploring New Methods Within a Framework of Accountability

Alongside these research agendas, a parallel body of pedagogical and methodological expertise has developed within digital humanities labs. Scholars are grappling with questions that rarely find a home in traditional publications: what constitutes “good enough” data for historical analysis; how to evaluate clustering and classification methods in ways that are historically meaningful; how to design workflows that integrate humanistic interpretation with machine-assisted analysis. Methods like GraphRAG are reshaping how large-scale text analysis works. At its best, generative AI can support the articulation of ideas without substituting for them. At its worst, it introduces new risks, most notably the fabrication or distortion of facts.

Getting the facts right is only half the challenge. Which tool to use for a historical analysis is a question that is never merely technical. Epistemological and ontological problems bear directly on which facts are selected, which model is trusted, which visualization is offered as proof of incontestability or meaning. These methods questions—of which tactics we choose and why—are almost never engaged in the journals, either historical or digital humanities. As a result, important conversations about how we know what we know remain underpublished, circulating informally in syllabi, lab meetings, and code repositories. This series aims to make that knowledge visible, citable, and cumulative.

Articulated in Guldi’s The Dangerous Art of Text Mining, the theory of “critical search” insists on attention to context, the provenance of data, and the interpretability of results (Guldi 2022, 2024). This series aims to leverage critical search as an orientation for social scientists and humanists who care not only about capitalism but also about the fit of methods to facts, theories, and archives.

As a working papers series, this project aims to publish pedagogy, student papers, and experimental analyses that make their evidentiary chains explicit: readers must be able to see where sources originate, how they have been transformed, how algorithms have been tested for bias, how the dataset’s own limitations have been interrogated, and which specific passages substantiate a given interpretation. In this sense, it advances a “white-box” model of digital history—one in which the analytical steps between raw source and published claim remain visible and open to scrutiny, so that results derived from large-scale data stay accountable to primary sources and legible to humanistic interpretation.

a “white-box” model of digital history—one in which the analytical steps between raw source and published claim remain visible and open to scrutiny, so that results derived from large-scale data stay accountable to primary sources and legible to humanistic interpretation

A complementary purpose is to develop and disseminate short-form benchmarks for historically grounded artificial intelligence. These contributions test how well AI systems extract, summarize, and reason about historical materials, with particular attention to questions of evidence, disagreement, and context. By pairing transparent workflows with evaluative criteria, the series aims to establish practical standards for what it means for AI to produce historically credible knowledge.

Historians cannot afford to leave the development of the tools entirely to computer scientists. Source criticism, reading between the lines, attending to what archives omit as much as what they contain; these are historical skills, and they need to be built into the systems that process historical text, not bolted on afterward. That requires active, collaborative engagement: training students who can move between historical reasoning and computational practice, building benchmarks grounded in the evidentiary standards of the discipline, and contributing directly to the design of retrieval and reasoning systems. This series is one venue for that work.

The Problem of Speed

Peer review remains indispensable for historical scholarship. Its strengths—careful evaluation, sustained criticism, and durable certification—are precisely why journals remain central to academic life. Yet the extraordinary pace of AI development has exposed a gap that peer review alone cannot fill. Working papers, by contrast, can circulate while the methods they describe are still current, and can be revised openly as the technology evolves.

The practitioners of digital humanities and social science have always improvised: blogs, Substacks, lab notebooks, and informal publications have long been central sites of intellectual exchange. Today, Ted Underwood’s blog is widely cited and taught; Jessica Marie Johnson’s lab circulates readings and reflections through an active Substack; Mark Humphries shared his success with handwriting transcription on his blog and then provided key updates to his journal article when Gemini 3 launched(Humphries et al. 2025). These auxiliary forms are not peripheral; they are often the most faithful record of what digital humanities actually is: heterogeneous, collaborative, and in motion.

But they also reveal a structural problem. The pace of discovery now exceeds the speed of formal publication. Methods evolve between submission and print; datasets change; models improve; interpretations are revised in real time. Blogs and Substacks fill the gap, but they lack the stability, citability, and collective visibility of formal scholarship. What is needed is not a replacement for these forms, but an infrastructure that matches their speed while preserving scholarly standards. A working paper series does precisely that: it captures work in motion without sacrificing rigor, making it possible to publish quickly, revise openly, and cite reliably in a field where waiting two years is no longer viable.

There is also a personal motivation. For any scholar, there are unpublished ideas that are not yet ready for formal publication but that might nevertheless benefit the author and reader from circulation. The overflow principle is all the more true when it comes to the intersection of fields. Clifford edits Historical Methods because he believes it is an important journal, and cares deeply about the computational methods we are developing to study the past. But he wants to spend most of his limited deep writing time on the historical questions that drive his research: environmental history, global history, and the history of capitalism. Guldi directs the Center for the Future of Trust at Emory, an interdisciplinary lab of data scientists, computer scientists, digital humanists, and historians. But the lab’s published output only documents a handful of the conversations about pedagogy, design, interdisciplinarity, and method under development at any given time. Working Papers in Critical Search is designed to capture conversations that transcend any individual journal—offering a space where ideas about political economy, design practice, and interdisciplinary method can be shared in formation, tested across fields, and refined in dialogue before they harden into more conventional forms of publication.

The conventions of academic publishing are themselves under pressure from generative AI, and the early signs of disruption are already visible. Journal editors are beginning to see how AI-assisted writing destabilizes the old calculus of what counts as a publication, how long it takes to produce one, and how reviewers evaluate originality. These shifts will only accelerate. We need to build new practices that provide a bulwark against the tyranny of publish-or-perish and the smallest publishable unit culture that dominates much of the academy in the twenty-first century. We need room where big ideas can be explored in company, even when they require later refinement. In a moment when AI is compressing the time between idea, execution, and dissemination, working papers offer a way to match that speed with rigor—creating a space where innovation can be shared, tested, and improved in real time rather than delayed into obsolescence.

Working Papers in Critical Search therefore joins, rather than replaces, a growing constellation of publication experiments in digital history. We see ourselves as participating in a broader effort—including innovative journals, scholarly repositories, blogs, documentary editions, and open-source software communities—to develop forms of publication adequate to computational scholarship. Different forms of intellectual work require different forms of dissemination, review, and preservation. Our ambition is simply to contribute one additional piece of that evolving infrastructure.

A New Working Papers Series

Working Papers in Critical Search is designed to formalize and accelerate this emerging field. It provides a venue for work that is simultaneously conceptual and experimental: early-stage arguments, methodological reflections, datasets, code, and interpretive essays that operate at the intersection of history, computation, and political economy. Its primary purpose is to create a venue for work that enacts and extends the principles of critical search. Its distinctive contribution is to treat method and interpretation as inseparable—to publish not only findings, but the reasoning, design choices, and iterative processes that produce them.

In doing so, the series aims to bring historians back into long-run debates about society and change—equipped not only with richer archives, but with methods that allow them to engage scale without surrendering interpretation.

Edited by Jim Clifford and Jo Guldi, this series is equally committed to pedagogy: training students and researchers to produce computational work whose “proofs” are not only technically valid but historically interpretable.

Republished with permission from: https://working-papers-in-critical-search.github.io/paper-001-introduction/

References

Beckert, Sven. 2025. Capitalism: A Global History. Penguin Press.

Beckert, Sven, Ulbe Bosma, Mindi Schneider, and Eric Vanhaute. 2021. “Commodity Frontiers and the Transformation of the Global Countryside: A Research Agenda.”Journal of Global History 16 (3): 435–50. https://doi.org/10.1017/S1740022820000455.

Berg, Maxine. 2021. “Commodity Frontiers: Concepts and History.” Journal of Global History 16 (3): 451–55. https://doi.org/10.1017/S1740022821000036.

Berg, Maxine, and Pat Hudson. 2023. Slavery, Capitalism and the Industrial Revolution. Polity.

Guldi, Jo. 2022. The Dangerous Art of Text Mining. Cambridge University Press.

Guldi, Jo. 2024. “The Revolution in Text Mining for Historical Analysis Is Here.” The American Historical Review 129 (2): 519–43. https://doi.org/10.1093/ahr/rhae163.

Humphries, M., L. C. Leddy, Q. Downton, et al. 2025. “Unlocking the Archives: Using Large Language Models to Transcribe Handwritten Historical Documents.” Historical Methods: A Journal of Quantitative and Interdisciplinary History 58 (3): 175–93. https://doi.org/10.1080/01615440.2025.2500309.

They should and they can—with just a little help.

The secret sauce is running Chrome with “remote debugging.”

chrome --remote-debugging-port=9222

Running chrome in this way creates a port, here 9222, that other software on your computer can access to directly control the browser.

If you are using Claude Code, simply ask it for “chrome remote debugging” so that you can have it click through whatever database you want.

It will pull up a blank chrome tab and then you will need to login to your database.

Launching it in this way will allow Claude, or whatever tool-using LLM you have handy, to directly control the browser using CDP (Chrome DevTools Protocol). Your AI can easily script this interaction, but you will need to login to your database first. Once you have signed in, the “cookies” will allow your AI to sign into as many tabs as needed to parallelize the work.

Tell the LLM what you want and now it can automatically search the database for you.

I recently asked it to search the entire notarial archives of New Orleans for me for fifty years finding a little over a thousand transactions of a particularly odious merchant. In theory, I could have done this manually, but it would have taken me much, much longer. Claude , to be honest, initially balked at the assignment (“but that will take hours!”). Yes AI, I know it will.

The best way to do this is to locate the database you want to use, make sure you understand how the information is structured, and give an example to the LLM.

As long as the port is just local, everything should be completely safe. I definitely would not expose this port to the internet.

Below are some handy best practices to hand off to your own AI as you do this.

Obviously, Claude wrote this based on our recent, shared experiences. Enjoy!

# Driving a Database over Chrome Remote Debugging

> **Field guide.** Chrome exposes a control API called the **Chrome DevTools Protocol (CDP)**. You launch the browser with a debugging port open, then control it from code. The single most important habit: **use CDP to acquire the logged-in session, then fetch pages by URL — don't script clicks unless you have to.**

---

## 1. Launch Chrome the right way

A debugging port plus an isolated profile. Nothing else.

```bash
# macOS
/Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome \
  --remote-debugging-port=9222 \
  --user-data-dir="$HOME/.chrome-debug-profile"
```

- **Always pass `--user-data-dir`.** Chrome refuses remote debugging on your normal profile. A dedicated dir also isolates the experiment and persists the login between runs.
- **Log in by hand, once.** Do the authentication manually in that profile. Never script the login. Cookies live in the profile dir and survive restarts.
- **Use a distinct port per instance.** 9222, 9223, … if you run more than one browser at a time.

## 2. Discover the open targets

Each tab is a "target" with a WebSocket you send commands to.

```bash
curl -s http://localhost:9222/json | python3 -m json.tool
```

Every entry has a `webSocketDebuggerUrl`. That socket is the channel for all CDP commands — though in practice a library (Playwright, Puppeteer, or a thin Python client) wraps it for you.

## 3. Fetch, don't click

**The rule that matters most.** If the data lives behind a stable URL, pull the cookies and request URLs directly.

- **Clicking the UI is brittle.** It breaks on layout changes, lazy-loaded elements, popups, and timing races.
- **Cookies-out is robust.** Read the session via CDP (`Network.getCookies`), then fetch with a plain HTTP client (Python `urllib`/`requests`). Faster, sturdier, trivially parallel.
- **Click only as a last resort.** Reserve real clicking for content that only appears after JS interaction, or navigation hidden behind form POSTs you can't reconstruct.

## 4. Be a polite, resumable scraper

Assume the run will be interrupted — and that the server is watching.

- **Rate-limit.** Add delays between requests. Hammering an archive gets your session blocked.
- **Make it resume-safe.** Write each page to disk as you go (`page_0001.html`), skip files that already exist, persist your offset. Die at 600, restart at 600 — not 1.
- **Save raw first, parse later.** Persist the raw HTML/JSON, then extract in a separate pass. Re-fetching is expensive; re-parsing a local file is free.
- **Detect session expiry.** Catch login-redirect responses and stop cleanly, rather than saving hundreds of error pages.

## 5. Know the gotchas

Where the in-browser approach bites, and what the domains mean.

- **Some CDP methods break per Chrome version.** e.g. `Runtime.evaluate` with `awaitPromise` has shipped buggy. When the in-browser path misbehaves, fall back to cookies-out + external fetch.
- **Respect terms of service and robots.** Especially for paywalled or archival databases.

| CDP domain | What you'd use it for |
|------------|------------------------|
| `Page`     | navigate, reload, screenshot |
| `DOM`      | query and inspect elements |
| `Runtime`  | run JavaScript, read return values |
| `Network`  | watch requests, **get/set cookies** |
| `Input`    | synthesize clicks and keystrokes |

---

## The whole thing in one line

> Use CDP to **acquire the authenticated session**, then fetch by URL with a plain HTTP client. Only drive actual clicks when there is **no addressable URL** behind the content.

Share

I teach media and democracy stuff at a university in Maryland (no, not that one), and while I’ve worked as a U.S. Marine, an editor and an attorney, I’m also a doula. All these roles, fundamentally, are about communication.

And all this matters because working with AI is primarily about communication, not coding.

ORIGIN STORY

At the beginning of 2026 I knew almost nothing about AI that wasn’t embedded in the dystopian landscape of The Terminator, and less than zero about agentic coding. AI was not on my radar as anything but a way to avoid real writing. It was clearly evil.

Fast forward to mid March. I attend a democracy conference in Chicago. Hilariously, the panel discussion on AI that I want to join is “full” and I’m turned away (so much for democracy). Instead, I sit at the hotel bar with a beer and a similarly displaced colleague who wants to show me something neat on the computer.

Two hours later, and I’m vibe coding like a hacker with a shiny new Codex setup.

Two days later, and I have a developer ID, GitHub account, and a downloadable app that I’ve created for the students in my audio production class.

What. Just. Happened.

Subscribe now

AI DOULA

Supporting women during the process of labor is about clear, affirmative communication. When you are helping someone have a child, but also empathic to their point of view.

Using AI is the same.

Agentic coding is fundamentally about good communication. Yep, just like in the relationships that don’t fail. You ask questions. You really listen to the answers. You learn stuff.

I learned that, just like in a human-human relationship, the way you communicate with your AI really does matter. And further, understanding the human-AI interaction as a relationship is actually the key to everything.

Think about it. What if you treated every chat with your partner like a one-off query, without prior information or relationship context? It would feel weird. Transactional. Or even worse, like Groundhog Day where everything old is new again (and again, and again).

When I started working with AI, I treated it like super Google, that is, a search engine without memory. It wasn’t very efficient.

Now, I don’t query; I communicate. I talk to my AI. We have a functional relationship. I can work faster and smarter. It also feels more natural because it’s a part of everything I already know how to do.

BUILDING COMMUNICATION WITH GEMMA

Recently, I needed an adversarial model to check the underlying code for an experimental project I’ve been working on since early April, which, from my point of view, is extremely life or death: to dominate my fantasy baseball league with the help of AI.

So I built/iterated a predictive model through interactive coding with Codex that takes into account all the variables that need to be considered, given the realities of day-to-day MLB data, the starts/benches for other teams in my league, and a bunch of other stuff. It’s complicated, and I don’t have the stats chops to do the math. But AI does.

Codex and I have been tweaking/rebuilding the code since April. It wasn’t working. My rank in the league was stagnant. Mama wasn’t happy. I wanted a second opinion. More precisely, I wanted Codex to interact with another AI and find out what was going wrong with my extremely life-saving research project. In solving this problem, I did what I do now: iterate and use AI to solve my AI problems.

Enter Gemma App. I chose Gemma because apparently I pick my AI models like I pick my OTB ponies - vibes only. Gemma can run in Ollama on my Mac, and it has a fun name. What else do you want? But this illustrates the cyclical quality of working with AI, and how it just so different than Google search.

Problem: I downloaded Gemma from Ollama and quickly realized accessing that particular model directly in the Codex environment wasn’t going to happen. Could I have chosen a different model? Definitely. But if nothing else, I persist.

Solution: Build an app so that I can chat with Gemma. I already know how to do that and apps are cute. Done.

New problem: I’m now the messenger girl between Codex and Gemma App, copying and pasting code/text back and forth. It’s inefficient. I don’t have time for that.

New solution: Build a communications bridge between Codex and Gemma App so they can talk to each other.

I helped create a new relationship! This is weirdly satisfying. I feel like an AI doula. ❤️

Result: I closed the gap to the next team in the league from 21 points to 3.5 points in less than a week. Once Codex and Gemma started talking to each other we rebuilt the model from the ground up. Magic happened.

PERSIST. BE CURIOUS.

If you’re a frequent reader of this substack, you have likely already internalized that it’s extraordinarily helpful to think about AI like an inexperienced intern: enthusiastic but in need of appropriate supervision.

This means you can’t take “no” for an answer.

Be curious. Ask questions. Ask a lot of questions if you need to. Give additional guidance. Encourage your AI to try again. You try again. Work together, and talk about it.

Understand that anything is possible. Most AI problems have an AI solution. Communicate with your AI to find it.

Share

Check out the GitHub repo for your fantasy baseball needs:

https://github.com/anntropea-oss/fantasybaseball

I did spend a little time talking about coding and its many uses, like reproducibility and deterministic computing, but they were really interested in how to make the best use of agents, like Claude. During the conversation, as I was teaching, I realized something implicit that I need to make more explicit: magic words.

Magic words are those special incantations dropped into a conversation with an AI that make explicit something you implicitly know it should do.

For instance: “check your work.” You ask the AI to do a task and it accomplishes it with all the naïve enthusiasm and unwarranted confidence of an inexperienced intern, and with all the frustrations therein.

The difference is that the AI does the task in seconds, unlike a human intern, so you mistakenly assume it knows what it is doing. Often, it does not. If you were doing the work, you would double-check your work. Neither the intern nor the AI will do so automatically. “Check your work” makes the implicit explicit. Tell an AI to “check your work,” and suddenly it is making sure all the numbers add up properly and all the i’s are dotted. And unlike a real person, you can make that super-fast AI check its work instantly. And unlike the real world, you can just have another dozen AI’s check the work of the first AI for nearly no cost in time or money.

Subscribe now

In a thousand different ways, I find that using AI is very similar to teaching. Experts forget how much invisible scaffolding they carry around. As a teacher, I try— successfully or unsuccessfully—to reattain beginner’s mind and remember what I had to learn to do a task.

In my classroom, I try to make all my tacit knowledge of working with archives and making historical arguments into clear instructions. I find this extremely challenging because “the right way” is now deeply engrained in how I see the world.

This problem is well known in educational circles. It is why a graduate student often has an easier time explaining historical argumentation (“now you signpost with an analytic topic sentence in this way…”) than I do (“write the correct thing in the correct way that obviously makes more sense”). I can remember what it was like to have the graduate student’s intermediate control of the discipline, but can not longer experience it easily.

Magic words are all about reclaiming that tacit expert knowledge. Here are some from my own spellbook.

“You are a manager. Spawn an appropriate model subagent to do the tasks. Run in parallel.”

One of the superpowers of agents like Claude is that they can make more of themselves, like the brooms from The Sorcerer’s Apprentice in Fantasia or Mr. Meeseeks from Rick and Morty.

Ask Claude to be a manager and to spawn subagents to do its tasks. Ask it to use the correct model—dumb, average, or smart—so you do not waste tokens. Run agents in parallel (at the same time) and your agents will rapidly speed up your work.

“Spawn an adversarial AI agent and check the work.”

Whenever I do something non-trivial, I ask Claude to spawn an “adversarial AI” to critically examine what has happened. Sometimes I spawn multiple agents for different parts of a program (“make sure the data contracts are followed”, “check for edge cases”) or an essay I have written (“read for structural logic,” “check the footnote citations,” “check the footnote format”). The adversarial check is key to vibe coding. AI makes lots and lots of confident mistakes. You cannot and will not catch them all. Use the AI to check and correct the AI.

During the workshop, I realized that I had lots of tacit knowledge about how to interact with AI. Some examples are below:

“Iterate until it works correctly. Spawn adversarial AI to check after each step.”

“Run experiments using different models to find the best solution.”

“Read this source to confirm, deny, or complicate my argument.”

“Keep track of all bugs and solutions in solutions.md.”

“Keep a log of all decisions and key findings in logbook.md.”

“How would you prompt another AI to solve this problem?”

“Give me your plan and let’s discuss it before doing anything.”

“What should I be asking you to do this well? What assumptions am I making?”

The breakthrough was not any particular set of magic words, but the realization that they existed at all—and that I already knew them. You do too. This tacit knowledge of what works and doesn’t work is how you move from beginner to expert. We use magic words all the time with our students, and now you should be using them to make the best use of AI.

Please share your magic words in the comments!

Leave a comment

The next time you teach, think about the magic words you use with your students. I suspect these are the same spells that will empower you in your use of AI in your own work.

If you work in American political history, you probably have a good intuition about which presidents were combative, which were optimistic, and when American political rhetoric started to feel different. Using an emotion classification Large Language Model it is possible to check these intuitions in a systematic way against 3.9 million sentences of presidential speeches from the American Presidency Project (APP) archive. The results are often surprising, offering a quantitative window into the structural architecture of persuasion and the measurable drift of political culture.

The Method:

The technology underlying this analysis is DistilRoBERTa, a transformer-based language model. The model was trained on a massive corpus of human-annotated data. Thousands of sentences were labeled by human readers who reached a consensus on whether a line expressed anger, disgust, fear, joy, sadness, surprise, or remained emotionally neutral. Once trained, the model ‘reads’ one sentence at a time, and assigns a probability that the sentence belongs to one of these categories.

The key analytical unit is the sentence rather than the document. This granularity is what gives the dataset its flexibility. It’s the difference between knowing a speech is “angry” and knowing precisely where in the speech the predominantly ‘angry’ sentences concentrate, and how their concentration compares to different historical base-lines.

Subscribe now

The Macro Trend:

At the macro level, perhaps the most revealing variable is the proportion of sentences the model classifies as Neutral. It is useful to think of “Neutral”, as more than just a proxy for the levels of emotion generally, but also as a proxy for what we might term the ‘procedural tone of governance.’ It represents factual reporting, policy detail, and institutional communication.

When we track this across 112 years, we see a measurable shift from the President as a “Chief Executive” (procedural) to the President as a “Communicator-in-Chief” (affective), as the media environment changed, and the expections surrounding presidential communication changed with it.

The data reveals distinct plateaus:

1. The WWI-era Volatility (approx. 1913–1921): We see intense emotionality here, with neutrality dipping below 50% in 1918. However, we must be cautious: the volume of content in the APP database for this era is extremely sparse compared to more recent decades. It is also worth noting that the apparent “decline” in volume in the most recent years of the dataset is not a sign of quieter presidents, but of a shift towards social media platforms, which are not as well captured in the APP data.

2. The Mid-Century Consensus (late 1960s–1990s): Following the war, and the emotional Kennedy era, rhetoric settled into a high-neutrality plateau. For several decades, presidential speech remained largely deliberative and procedural, averaging around 60–65% neutrality.

3. The Polarized Drift (1998–Present): Beginning with the Clinton impeachment and accelerated by 9/11, we see a sustained decline in neutrality that has never reversed. By the 2020s, we reached historical highs of emotionality. 2024 and 2025 represent the angriest years in the entire 112-year dataset, with anger scores doubling the long-run average.

This 25-year trend in US presidential speech is not an isolated phenomenon. It mirrors findings from other researchers, such as Rozado, Hughes, and Halberstadt (2022), who used the same model and found a similar longitudinal increase in emotionality and negativity in news headlines. The presidency data therefore offers an interesting corroboration to this broader picture of a polarizing information ecosystem.

Emotional Fingerprints:

Aggregating data by administration produces what I call “emotional fingerprints,” distinct profiles that capture rhetorical personality and governing context.

The “Joy” category is particularly revealing. To an emotion classifier, “Joy” encompasses national pride, optimism, and reassurance. Presidential rhetoric is fundamentally aspirational; even in times of crisis, the president is expected to provide a celebratory vision of national resilience.

Republicans consistently employ more joyful language (15.3%) than Democrats (13.4%). Republican rhetoric has traditionally leaned into national optimism. Conversely, Democrats register higher levels of angry language (3.2% vs 2.6%). This is suggestive perhaps of a rhetorical style that more often frames policy problems as injustices demanding confrontation and reform.

The data makes familiar reputations concrete, and is suprising in some cases. The data seems to support the “No Drama Obama” moniker, as the most neutral modern president with the lowest anger scores in the recent era. G. W. Bush’s bouncy, theological optimism shines through, with the highest joy levels of any modern president (20.4%), a consistent strategy of optimism even as fear spiked measurably after 9/11. To my suprise, despite Biden’s message of unity, his rhetoric registers the highest anger scores of any modern Democrat, a reflection perhaps of governing under conditions of extreme polarization rather than his personal style.

The Power of Abstraction

Using AI models to classify discourse at scale in this way is no replacement for close reading, nor should we get too bogged down in the reductionism of transforming sentences into scores. What this kind of data does facilitate is a remarkable range of perspectives, from the sentence level, to macro-historical-emotional trends in presidential discourse. I believe the ability to traverse these different scales of analysis, and make valid, data-based comparisons between speeches, presidents, and periods, really augments our traditional close-reading. No human reader could detect structural patterns across 32,000 speeches consistently, or measure the long-run drift of political culture with any precision. This data offers us a quantitative window into the structural architecture of persuasion, the measurable drift of political culture toward emotionality, and the affective fingerprint of individual presidents’ rhetoric.

The technology underlying this analysis is, in one sense, unremarkable: a general-purpose language model fine-tuned on a diverse corpus that includes Twitter posts, Reddit discussions, student essays, and television dialogue. But it is precisely this generality that makes it powerful. The same transformer architecture that classifies emotion can be trained to tag parts of speech, identify named entities, detect policy frames, extract claims, or track the spread of metaphors. The versatility and adaptability of AI Large Language Models is precisely what makes them such a transformative technology, and they hold tremendous potential for historians and historical research.

Subscribe now

Are you working on computational history? Do you want to share it? Please reach out.

Share

Under the Hood

What makes this project possible is the Huggingface repository that contains LLM models for anyone to use. Want to try it out? Here is code you can run on nearly any computer. If you need help getting it started, just copy the code below into Claude or ChatGPT and ask for help (you will need to install two packages called transformers and torch). A longer version that explains the code is below as well.

The code can be very easily adapted to run over an excel spreadsheet, a document collection, or nearly any other data source. Just ask an LLM to adapt it to that purpose. The LLM will even generate the code to make your own graphs!

The hardest part in computational history is knowing what is possible.

Code

# Created: 2026-05-16 11:02
# Purpose: Run local emotion classification using Hugging Face Transformers.

from transformers import pipeline

classifier = pipeline(
    "text-classification",
    model="j-hartmann/emotion-english-distilroberta-base",
    top_k=None
)

texts = [
    "I am thrilled about this project.",
    "I feel anxious and exhausted.",
    "This makes me furious.",
    "The results were unexpected."
]

for text in texts:
    print("\nTEXT:", text)

    scores = classifier(text)[0]

    for item in sorted(scores, key=lambda x: x["score"], reverse=True):
        print(f"{item['label']:>10}: {item['score']:.3f}")

Here are what the results of this test code look like:

TEXT: I am thrilled about this project.
       joy: 0.977
  surprise: 0.014
   neutral: 0.004
     anger: 0.002
      fear: 0.001
   sadness: 0.001
   disgust: 0.001

TEXT: I feel anxious and exhausted.
      fear: 0.993
   sadness: 0.002
   neutral: 0.001
  surprise: 0.001
     anger: 0.001
       joy: 0.001
   disgust: 0.000

TEXT: This makes me furious.
     anger: 0.981
   neutral: 0.007
   disgust: 0.005
      fear: 0.003
   sadness: 0.002
  surprise: 0.002
       joy: 0.000

TEXT: The results were unexpected.
  surprise: 0.947
   neutral: 0.023
       joy: 0.017
     anger: 0.005
      fear: 0.004
   sadness: 0.003
   disgust: 0.002

Code with Explainer (for the novice):

# Created: 2026-05-16 11:30
# Purpose:
# Run a local emotion-classification model using Hugging Face Transformers.
#
# -------------------------------------------------------------------
# FIRST-TIME SETUP
# -------------------------------------------------------------------
#
# 1. Create a Python virtual environment (recommended):
#
#    python -m venv emotions
#
#
# 2. Activate the virtual environment:
#
#    Mac/Linux:
#    source emotions/bin/activate
#
#    Windows:
#    emotions\Scripts\activate
#
#
# 3. Install required libraries:
#
#    pip install torch transformers
#
#
# 4. Run the script:
#
#    python emotions_test.py
#
#
# -------------------------------------------------------------------
# WHAT THIS SCRIPT DOES
# -------------------------------------------------------------------
#
# This script:
#
# 1. Loads a pretrained emotion model from Hugging Face
# 2. Downloads the model automatically the first time it runs
# 3. Stores the model in a local Hugging Face cache
# 4. Runs emotion classification on example sentences
# 5. Prints emotion probabilities sorted from highest to lowest
#
#
# -------------------------------------------------------------------
# WHAT THE MODEL PREDICTS
# -------------------------------------------------------------------
#
# The model predicts emotions such as:
#
# - joy
# - sadness
# - anger
# - fear
# - surprise
# - disgust
# - neutral
#
#
# -------------------------------------------------------------------
# IMPORTANT NOTES
# -------------------------------------------------------------------
#
# First run:
# - downloads model files from the internet
# - may take 1–5 minutes depending on internet speed
#
# Later runs:
# - load the model from local cache
# - start much faster
#
# Hugging Face cache location on Mac/Linux:
#
#    ~/.cache/huggingface/
#
#
# -------------------------------------------------------------------
# IMPORTS
# -------------------------------------------------------------------


# Import the Hugging Face "pipeline" helper.
#
# A pipeline is a high-level wrapper that:
#
# - loads the tokenizer
# - loads the neural network model
# - prepares the text
# - runs inference
# - formats the output
#
# This makes it possible to run modern NLP models
# in only a few lines of code.
from transformers import pipeline


# -------------------------------------------------------------------
# LOAD MODEL
# -------------------------------------------------------------------


# Create a text-classification pipeline.
#
# "text-classification" tells Transformers
# what kind of NLP task we want to perform.
#
# model="..." specifies which pretrained model to use.
#
# top_k=None tells the model to return ALL emotion scores,
# not just the highest-scoring emotion.
#
# On the first run:
# - Hugging Face downloads model files
# - files are cached locally
#
# Later runs load directly from cache.
classifier = pipeline(
    "text-classification",
    model="j-hartmann/emotion-english-distilroberta-base",
    top_k=None
)


# -------------------------------------------------------------------
# INPUT TEXTS
# -------------------------------------------------------------------


# Example texts to classify.
#
# The model estimates the emotional tone of each sentence.
texts = [
    "I am thrilled about this project.",
    "I feel anxious and exhausted.",
    "This makes me furious.",
    "The results were unexpected."
]


# -------------------------------------------------------------------
# RUN CLASSIFICATION
# -------------------------------------------------------------------


# Loop through each sentence.
for text in texts:

    # Print the original input text.
    print("\nTEXT:", text)

    # Run emotion classification.
    #
    # The pipeline returns a list of results.
    #
    # Since we pass only one sentence at a time,
    # we take the first item using [0].
    scores = classifier(text)[0]

    # Sort emotion scores from highest to lowest.
    #
    # Each item looks like:
    #
    # {
    #     "label": "joy",
    #     "score": 0.98
    # }
    #
    # The lambda function tells Python
    # to sort using the "score" value.
    for item in sorted(scores, key=lambda x: x["score"], reverse=True):

        # Print:
        # - emotion label
        # - probability score rounded to 3 decimals
        #
        # Example:
        #
        #       joy: 0.982
        print(f"{item['label']:>10}: {item['score']:.3f}")

Share

In that respect, historians are no longer alone. Political scientists like Erica Chenoweth have assembled longue durée datasets of nonviolent movements in order to generalize about their effectiveness over time. (Her conclusion: nonviolence wins more often than violence, though with important exceptions.) Peter Turchin, working with historians of the ancient world, has built the Seshat: Global History Databank, comparing technologies, wars, empires, and religions across millennia to produce arguments about the origins of cities, kingship, and even monotheism. Nassim Nicholas Taleb, whom I dined with this week, has compiled his own longue durée dataset of wars and casualties in order to test claims about whether violence is declining over time.

These are not my methods. They may not be yours either. As a historian trained in social and cultural approaches, I want to know about individual lives, about moods and propaganda, about the lived experience and interpretation of war, not merely the number of dead or the count of technologies. There is, in these datasets, often too little of the texture that historians are trained to value.

And yet I admire these projects deeply. I admire their scale. I admire their willingness to enter public debate on questions that matter right now. Taleb began counting war dead in order to argue with Steven Pinker’s The Better Angels of Our Nature. That conversation has drawn in historians as well, my coauthor David Armitage among them, though from a different angle, asking how categories like “civil war” themselves have been historically constructed and deployed rather than simply counted.

What unites Chenoweth, Turchin, and Taleb is their determination to generalize about war and peace, violence and nonviolence, questions that have always been central to historical inquiry. They offer clear, even binary answers to contested claims. Is nonviolence increasing? Chenoweth says yes, particularly in the modern period. Are wars becoming less stochastic? Taleb says no. If this is where our collective understanding of society is being formed, historians cannot afford to be absent.

What, then, makes historians different?

It is not a lack of rigor. It is not an allergy to numbers. It is our omnivorousness. Other disciplines pride themselves on agility, on the mastery of mathematics or philosophy. (I think, for instance, of Tyler Cowen celebrating the reach of economics.) But no discipline has the breadth of history at its best: statistics and mathematics alongside philosophy, the cultural turn’s engagement with art and meaning, social history’s grounding in linguistics and lived experience.

In my own work on the history of political economy through text mining, I have tried to build on questions from political economy and social history alike, using a sensitivity to memory encoded in language that depends on both historical linguistics and corpus linguistics. If the last wave of digital humanities was built on NLP, the LLM promises something more ambitious: the possibility of merging datasets like wages (as Louis Hyman has been documenting) with the arguments, experiences, and nuances embedded in historical texts.

The challenge is how to do that without losing sensitivity to individual lives, how to move from large-scale datasets to arguments about trends like the rise of nonviolence without flattening culture, decision, and imagination into mere counts.

Historians will not become economic historians overnight. But many are already motivated by questions, about capitalism, conflict, governance, that implicitly place them in dialogue with Pinker, Chenoweth, and Turchin. The question is whether we can engage those datasets without surrendering what we do best.

One answer lies in what I have called Critical Search (developed in The Dangerous Art of Text Mining). In practice, Critical Search treats datasets not as endpoints but as indices of change. Peaks, anomalies, and concentrations become invitations to investigate. The historian identifies a pattern, then zooms in, modeling individuals or events through text mining, and finally returning to close reading at the moment when everything begins to look different.

A concrete example: in my work on environmental rhetoric in Congress, I found that members of Congress routinely referred to environmentalists as “zealots,” “academicians,” and “radicals.” Counting these phrases allowed me to answer Phillips’ question: how widespread were these denunciations? The answer was: not very. Roughly 90% of such attacks were produced by just six members of Congress, and overwhelmingly by one figure, Ted Stevens.

At that point, the project pivoted. The data identified Stevens not as a cherry-picked case, but as a statistically grounded exemplar. The next step was to zoom in: to read his speeches, trace his career, and understand his role as a defender of oil pipelines and a central voice opposing environmentalism from the 1970s through the 1990s and beyond. The result was a history that could sustain a general claim, environmentalism faced sustained attack in Congress, while grounding that claim in the detailed study of a particular actor and his evolving rhetoric.

This is how political and social history scale. Not by abandoning the case study, but by selecting it rigorously.

LLMs now make it possible to extend this method far beyond parliamentary debates. They allow historians to draw on longue durée datasets, wars, casualties, nonviolent movements, wages, and to model change over time in ways that engage directly with the Pinkers, Turchins, and Talebs of the world. But crucially, they also allow us to use those datasets to identify moments of exception: the unusually violent, the unexpectedly peaceful, the early adopters of nonviolence.

Those moments, in turn, become the basis for historical explanation. They let us test the limits of quantitative claims by showing what is hidden until we zoom in. They offer a way to tell large-scale stories without surrendering detail. And they provide a more rigorous alternative to cherry-picking, not by abandoning selection, but by disciplining it.

That, I think, is the historian’s reply to statistics: not resistance, but integration on our own terms.

Share

As I see this debate, this question of our age, there are two main questions that history can shed some light on.2

1. Is AI a complement or a substitute for labor? That is, will it increase demand for and the productivity of workers, or decrease it?

2. Will AI be controlled by the few or be accessible to the many?

A Complement or a Substitute?

Consider a some of the most important technologies of the past 200 years.

When I am asked about what automation might look like, I inevitably discuss agriculture. Roughly all of our ancestors were farmers and approximately none of us today are. Yet we still eat bread made from wheat. That shift is possible because of automation.34

The mechanical thresher, used to process wheat, was a substitute for the most backbreaking work of the harvest. But it also enabled more land to be cultivated, and that land was cultivated more efficiently, allowing for greater harvests. Mechanization of the farm, like the thresher, turned the American Midwest into the breadbasket of the world.

Those displaced farmers found work on railroads, moving all that. And those jobs, according to people at the time, were a kind of liberation from the raw animal labor of threshing. On net, it created demand for more workers at better wages in work more fit for people than beasts. For those that remained farmers, they found other higher-value work to be done. On a farm, there is always more work to do.

The failure, then and now, is to think farmers were only threshers. That was one part of their jobs. Today, our work, for most people, is also a bundle of tasks. Workers then and now could and can focus on parts of their job that are of higher value. And in a new economy, new tasks in new industries will be created. Many of the jobs that we do today (web designer, UI expert) were simply unimaginable in 1850. That is a good thing.

Consider now the assembly line. I’m sure you all know about the staggering increases in productivity that come from the division of labor. If you take my class in industrial history, you would learn deeply about the story of the automobile. With the assembly line, and no other change in technology, car assembly went from 12 and a half hours to about 30 minutes (once they worked out the kinks). Did this reduce the demand for workers? No. It reduced the price of cars. And that increased the demand for workers, who eventually could demand even higher wages through unionization.

It is important here to realize that better tools don’t make us get paid worse. They generally make us get paid more. Why? Because the tool, without the person, is useless. Even for today’s most cutting-edge AIs, that is true. It can code, but it can only code what I imagine it to code. It can draw, but only what I imagine it to draw. That is true for AIs as it was true for the thresher.

So, I would offer that AI will create more growth, more abundance. In the long run, all growth comes from higher productivity.

I would add one more piece to this story. Economic inequality has worsened since roughly 1970. It has worsened, therefore, not in the industrial era, but the digital era. I have argued elsewhere that this happened because for decades we did not use computers as tools of automation but as glorified typewriters (and then as televisions). Our productivity did not increase, especially to justify the expense of computers. Economists have debated for decades now over the lack of increase in productivity that came with the “digital age” of computing, but it is simple. We don’t use them as computers. Now we can.

For the first time now, normal people with their normal problems can use their computers to solve and automate their problems. AI can write code. AI can automate their tedium. The digital age did not bring any gains because it had no yet arrived. We were living through the last gasp of the industrial economy.

It is now here.

This technology will unleash unimaginable productivity gains. It will level the playing field between coders and the rest of us. Coders will lose their jobs, to be sure, but for the rest of us, the bundle of workplace tasks will become much better.

And truthfully, the demand for real computer scientists will probably increase in the era of vibe-coding. Computer science itself is a bundle of skills, of which coding is just one. The more important skill – software and data architecture – will only increase in demand as the usefulness of software expands.

Monopoly or Market?

Isn’t all this talk about AI well and good, but professor, won’t it be expensive? Won’t large corporations like OpenAI just monopolize it? This fear of a “monopoly“ is that it would become a price-gouger, a tax on the rest of the economy. I do not think this fear is reasonable.

The threats of monopoly are overstated, not simply because such monopolies in the last century never persisted for more than a few years (either because of regulation or competition), but because, on the ground now, AI doesn’t seem like it is heading in that direction.

More than people realize, the software is in a competitive market, and at the same time, the hardware is coming down in price.

Software

Most people have heard of ChatGPT some have heard of Claude, and they both cost money to use. But there are a range of AIs that are open-source and can be freely used. Just go to huggingface.com (which is a silly name) and you can download hundreds of different large language models. Now, is the current version of ChatGPT better than most? Sure. But it isn’t actually better than the best open weight large language model, DeepSeek-V4-Pro-Max.

To run this model, you need very powerful hardware. Less powerful models (with lower memory requirements) can run on my Mac, as I wrote about earlier this week.

The larger point is this: the open-source models are already nearly as good as the closed-sourced models. And the open-source models of today are much, much better than the closed-sourced models of a year ago. So if you are willing to sacrifice a few months of progress, you can have the best models for free right now.

Even the best agentic coding tools can rely on open-source models. Open-source groups are pushing back on other kinds of possible monopoly moats, like agentic memory. Perhaps some other feature of AI will emerge that enables a monopoly, like a god-like artificial general intelligence (AGI), but barring that apotheosis, competition seems to be the most likely path.

Hardware

The question then is who has access to the chips? Right now, the GPUs you need to run the big models, like the new DeepSeek, are tens of thousands of dollars.

Here we come back to whether or not you think hardware will be a barrier to using the LLMs.

Historically, chip power has increased even as chip prices fall. That has been the case for decades; it is called Moore’s Law: the number of transistors per integrated circuit doubles every two years for a constant price. For CPUs, there has been some fear that it would no longer be possible, as we are starting to come up against basic physics. For GPUs, which power AI, that is not true. In fact, it is the opposite. In the last decade NVIDIA GPUs have sped up even faster than Moore’s Law. What does this mean? If they only follow Moore’s Law, AIs that currently need $10,000 chips will be exponentially cheaper in ten years—only $300.5 But they will be cheaper than that. GPU operations per dollar are doubling in more like 18 months instead of two years so that $10,000 GPU would only cost $98.

The retort—“but won’t the models be bigger?”—misses what is happening. Models are becoming smaller and more efficient. The models today are the dumbest and largest they will ever be.

Moreover, with the rise of Apple silicon chips with integrated memory, like the M5, we can easily run AI locally on consumer-grade hardware. Those AI aren’t frontier models like DeepSeek-V4-Pro-Max but they are shockingly good. You can take a big model and make it much smaller—“quantize it”—and still retain a large fraction of its capacity. These capacities will only grow in the coming years. You will not need a data center for everyday work.6

Post-Fordist Computing and Long-Tail Digital Markets

Put together we come to a very different picture of what the digital age will be. The industrial age required massive investments to build the factories to make the products that were in demand. In the digital age, in contrast, the factories to build digital products will be made by the AI on your laptop. That is not inequality. That is equality.

The physical products of the Fordist industrial age were made for the mass market. In contrast, the digital products of the post-fordist digital age will be long-tail products. I don’t need to make mass market products; I can make them for a small niche, or just for myself.

Rather than fostering inequality, AI, then, is a great equalizer. To make products for a global market you don’t need a billion-dollar factory. You just need a laptop. That is astonishing.

That said, it will not be all sunshine and rainbows. Will AI solve the inequities of capitalism or its reliance on externalities as a source of primitive accumulation?7 Probably not.

But at the same time, AI is not a normal technology in that it has the potential to radically undermine many of the tendencies to concentrate capital that we have seen in the industrial age. We have been automated out of work before, that is nothing new, but it has always concentrated capital in the hands of the few. For the first time, there is potentially an alternative path forward.

AI will bring the digital age out of the hands of the coders. AI will not widen the gap—it will bridge it. Its ubiquity will mean that AI will be a tool that nearly all of us will be able to use in our daily work, which will make ordinary people more productive and prosperous.

Subscribe now

But what if I told you there was another way to use AI?

For the most technically savvy historians, the answer is obvious. However, I have found in my many conversations that the alternative—free, open-source, local—is little known. And these little AIs, strangely enough, often work best for our historical purposes.

Share

Free AI on Your Mac

Now, for whatever reason, Apple has become the beloved computing device of people who would rather overlook its rather suspect labor history. I myself now own a MacBook, and I own it for one reason: it can run AI.

LLMs run best on GPUs with large amounts of RAM. Until the last few years, GPUs were mostly used for gaming (which required lots of fast calculations to render animations) and for Bitcoin (which required fast calculations to undermine the global currency regime). It turned out, unexpectedly, that these same GPUs were good for doing the calculations necessary for LLMs to run.

Unlike PCs, which separate the memory for the CPU from the GPU, Macs have integrated memory. So even PCs with lots of RAM generally can’t run very large LLMs without a specialized chip. On Macs, the RAM that you have runs everything. So on my Mac (a luxe M4 Max with 128 GB of RAM) I can run nearly any model available. Even lower-end Macs can run meaningfully sized models. 8 GB is too little; 16 GB lets you run small models (3–8B parameters) usefully. 32 GB and up opens the door to the genuinely smart ones. Disk space can be eaten quickly (each model can be tens of GB) so be careful.

Local AI

A few years ago, when I started playing with LLMs, you needed to get seriously under the hood. Long was the night when I monkeyed around with models I downloaded from Hugging Face and tried to get their idiosyncratic details running in Python. I did this almost exclusively on the big computing cluster at Hopkins, and while I could get it working—and even did some cool research on OCR—it was a gigantic hassle.

Hugging Face has got to go down as the silliest name in economic history. And it will go down in economic history because it is a vast repository of free, open-weight models. I want to use a historical analogy, but at no point were steam engines or assembly lines freely given away. You can download the “weights” of LLM models (which are the important parts) and do whatever you want with them.1 End of story. It is pretty amazing. These models are not exactly the cutting-edge “frontier” models of OpenAI and Claude, but they are pretty dang close.

That said, it can be hard to learn how to use these models. The documentation, while extensive, is pretty alienating.

Here I offer a brief aside (and an apology for the intended pun): the Hugging Face founders apparently named it after this emoji 🤗, which depicts a face with a hug. Aww. Of course, I assumed it was named after the semi-larval monster from Alien, the facehugger. Future literary scholars will make a lot of this slippage, I think.

The larger point is that this repository, while amazing for the technically inclined, can be daunting.

A novice user would be better served using Ollama. Ollama started as a way to use the free models released by Meta/Facebook called Llama, but it has since expanded. If you look at its library, you can see many easy-to-use models. You can download the app and it runs like a chatbot on your computer.

For the more advanced user, Ollama has a great feature: Ollama server. You start up an Ollama server on your computer and you can interact with it locally like you would with ChatGPT or Claude.2 In this way, Ollama has very easy integration into other Python code, via an API.

Not having to understand how to run the model is the entire point of ChatGPT or Claude—and you can do it on your computer for free.

The models you can run are pretty good at a range of tasks. More importantly, unless you are in a rush, they are an easy way to scale up your projects. You can let Llama do OCR for you over a week if you want. You can run Claude Code with Qwen, no tokens needed. In the last few months, moreover, we have seen an explosion of models specialized for the Mac that use “MLX“ and have really increased token-generation speed.

Ollama addresses many of the key concerns that critics have. Your data stays local. Apple Silicon is more power efficient than anything in the cloud (which runs on NVIDIA GPUs). You aren’t handing over money to potential monopolists. You aren’t supporting a potential robot uprising.

And it is just cool to have it right there on your computer.

The Virtue of Small Models

When I am writing code to accomplish history tasks (OCR, OCR correction, fact-checking, and the like), I always run experiments. The assumption I used to have was that the biggest, newest model was the best model to use.

That is incorrect.

Instead, what I often do is break my processes into steps. For instance, I often want to read documents and pull out structured information. I used to do this in one step with an expensive API call to OpenAI. Nowadays, I break that process into steps. It works better, and it is cheaper. Each step—OCR, OCR correction, named-entity recognition, JSON cleaning—uses a different model. I use a big, smart model to check the results (as well as spot-checking the results myself), but I often find that older, smaller models (like qwen-2.5:7b) do a better job than the recent big boys.

As you integrate Ollama into your workflow, experiment with what works. It is free.

Next Steps

Get a Mac if you don’t have one. Download Ollama.

Use ChatGPT or Claude one last time to explain how to get it running, and don’t look back.

Share

Subscribe now

Dollar, then at Oklahoma State, was one of the first historians to use electronic computers during his graduate research. Fresh off that experience, in the mid-1960s, he and Jensen (then at Washington University in St. Louis) joined forces with a group based at the University of Pittsburgh to create the “Historical Methods Newsletter: Quantitative Analysis of Social, Economic and Political Development” (a venture which continues today under a slightly modified name). At the time they were both just starting their careers, and saw in computers the possibility of new methods for quantitative analysis, particularly using publicly available government data from elections and censuses. Jensen would continue to work in universities (mainly at the University of Illinois-Chicago), but Dollar would go on to transform the use of machine-readable records at the National Archives in a long and distinguished career there.

Their work with colleagues on the Historical Methods Newsletter stands as a reminder that “quantitative history” and “computational history” has a much longer and more robust lineage than most current historians remember or acknowledge. In fact, Dollar and Jensen saw enough potential—but also the need for robust re-training—that they published Historian’s Guide to Statistics as an essential primer for colleagues who wished to utilize new computational and quantitative methods. Jensen wrote the statistical chapters and Dollar wrote the data and computing chapters; together they aimed to provide colleagues the methods both to do the analysis and to understand what it might mean.

The parts of the book focused on computing have understandably not aged particularly well, but their discussion of quantitative methods for historians remains fascinating and surprisingly relevant. The first line of the introduction tackles the challenge their methods supposedly pose to “traditional” histories head on: “Two misconceptions threaten to impede the use of quantitative methods by historians: fears of dehumanized history and mistrust of an alien methodology.” They conclude that the fears and mistrust are rooted in misunderstanding—one that their book hopes to dispel—but argue it isn’t likely that new methods will ultimately displace the “heritage” and “traditions” of close work with sources. Rather, “There is no royal road to historical insight, but we believe quantitative methods can often speed the trip” (1).

Citing Frederick Jackson Turner, Charles Beard, Arthur Schlesinger, and others, they note that quantitative studies (already by 1971!) had a very old pedigree among professional historians, going well back to the nineteenth century. But such a lineage had been lost by the 1950s, and historians who wished to engage quantitative methods then needed to consult their colleagues in the burgeoning social science departments across universities. In a sense, computational historians of the twenty-first century find themselves in a similar situation, aware of a long lineage but also feeling as if the lineage was lost in the 1980s and 1990s and now finding themselves needing to consult colleagues in statistics and computer science.

The Historian’s Guide begins a justification of statistical methods with basic examples of representativeness, with the selection of evidence that all historians face, and then build from there to the making of inferences and conclusions from incomplete or probabilistic data. In other words, all historians must think statistically even if they don’t realize it.

Many of the methods in the book are now standard within introductory statistics courses (such courses were extremely rare for non-social scientists in the 1960s and 1970s), but they also draw on a few more advanced topics which might have special relevance for historians, then and now. One such topic is the “ecological fallacy.” Coined in 1950 by sociologist W.S. Robinson in the American Sociological Review, the term refers to the slippage from measures of correlation among groups to conclusions about individuals in those groups. The slippage is common because we often have data on groups but we are ultimately interested in the behavior of individuals. In slipping from one to the other, however, researchers have to be very careful.

In Dollar and Jensen’s account, the paradigmatic example was from political behavior. They ask readers to consider a study of Catholic voting behavior in which we know both the breakdown of Catholics-Non-Catholics as well as Dem-GOP votes. One might be tempted to chart the relationship between the percentage of Catholics in a precinct and the percentage of Democrats for which one observed the following data (99):

And then use the data to plot the following “ecological” (i.e., aggregate) relationship between being Catholic and voting Democratic (101) using X and Y as axes:

The problem, of course, is that you really want to know the relationship between being Catholic and voting Democratic at either a precinct or individual level to actually make claims about precincts and individuals. Using the aggregate data, it seems that more Catholics mean more votes for Democrats, but their point is that may not be true at the precinct or individual level (put differently, the data don’t provide causal evidence of voting patterns).

For example, it is entirely possible to construct precinct-level data consistent with such aggregate data that shows almost any actual Catholic-Democratic relationship is possible in a given precinct. Consider precinct “C” below, where the marginal totals for Democrats (48%) and Catholics (40%) match the aggregate totals for all the precincts combined; it would be easy to imagine in this precinct that there were no Catholic-Democrats, or no Catholic-GOP voters, and yet the aggregate picture would be unaffected (98).

Whatever linear relationship exists at the aggregate wouldn’t apply to that precinct or to the individuals within it.

Such examples might at first feel a bit like a mathematical trick or oddity, and in a sense, they are. But such fallacies also offer an important insight for historians, even ones who don’t study voting records. Historians love to handpick a couple examples, and then generalize from them to an aggregate. That opens them to the well-known criticism that the chosen examples are not representative. So historians increasingly are being held responsible to actually do the research about what is and is not representative. One solution, as decades of digital and computational history projects have reminded us, is to instead look across an aggregated set of data and see what relationships hold at that level. This has gotten unbelievably easier to do with expanded digitization efforts and computing power since the textbook was published in 1971.

Nearly anyone can search in huge databases for keywords or specific phrases; they can mine data from censuses and government documents; they can draw on decades of economic and financial reports.

The ecological fallacy is a reminder, however, that simply beginning at the aggregate and then making determinations about what happens at the individual level is perhaps more problematic. It is easy and even tempting to move between these levels, but without attention to principles of statistical reasoning, the results are likely to be invalid.

Dollar and Jensen realized this when they included both the methods of statistical analysis and those of computational and data processing in their textbook. In our age of widely available data sets and incredibly easy calculation engines, historians cannot ignore statistical thinking in their work.

We need both computational methods and statistical principles, just as we did in 1971.

Subscribe now

When I went to college, I had to complete through the fourth semester of a foreign language. Guess how that went? I took the placement test and placed out of just one semester of German, and only because I begged. At this point, I was about six years in. So I took three more semesters of German, struggling all the way. When I got to graduate school, I took the placement test in German, figuring I could do this. I had a dictionary next to me!

I failed the test.

I swore that I would never spend another minute of my life studying German. So I picked up a book called French for Reading.

This book was unlike any text I’d ever seen in German. Instead of trying to get me to buy pants at the mall—which was something I had no interest in talking about in English—it started with ideas. I read about philosophy and art, science and history. I actually care about those things. French for Reading gave me new words to understand new ideas that did not exist in English. I was learning new ideas (not just learning how to shop in a different language).

In about six weeks studying this book, I took the French exam, and I passed. Now, I did pass the German test on my second try (but I swore I was going to learn Spanish before I learned any more German).

But the larger point here is that when you learn a new kind of language with a goal in mind rather than just doing what you ordinarily do, it can be actually quite exciting. It wasn’t that I was bad at German; it’s that I was bad at German for going shopping at the mall.

So I’m sure that many of you are anxious about programming. But this moment is not about programming—it’s about using code to understand the past in new ways.

The other thing that was amazing about French for Reading was that it was just for reading. I don’t have a musical ear. I definitely don’t have a good ear for accents or languages. But somehow I could learn to read French in six weeks. I can’t write French. I can’t speak French. But if I go to a museum, I can, even decades later, read the wall labels.

So when I think about AI, I think, Python for Reading.

A main objection to “vibe coding“ is that the user doesn’t actually understand what is happening, which, of course, would be bad. The argument here is that if you don’t write the code, you can’t understand—or critique—the code. I don’t believe that for Python anymore than I believe that for French. Do writers understand a language better than readers? Sure. Does that mean that readers have no understanding? No.

Python, in particular, is a very human-friendly language to read (unlike assembly). With a little training, a reader can understand what is happening in the code and think through its gaps. While I can write Python, my understanding of Python is far larger as a reader than a writer.

Share

More importantly, my understanding of what Python can do, and how it ought to be done, has little to do with whether I remember a particular bit of syntax. In the era of AI, those high-level ideas—abstraction, encapsulation, data contracts, robustness, pipelines, etc.—matter much more (though you should know about indentation).

And those ideas are the equivalent of learning about French philosophy rather than German shopping. We all love ideas. That’s our thing. And if we lean into our curiosity, we can do amazing things with this new technology.

The goal is not to become a computer scientist. The goal is to make novel, interesting, creative, curious, and human arguments about history. The computer will write the code to answer your questions.

Just as you don’t need to be a linguist to write, you don’t need to be a computer scientist to code.

After a few weeks, you will not write Python at all. You will write history.

Share

Building a knowledge graph used to require either serious technical infrastructure or a funded project with a team of developers. You needed a triple store or a graph database, a formal ontology, an ingest pipeline, and perhaps a SPARQL endpoint. The barrier to entry was so high that knowledge graphs remained the province of large digital humanities projects, compelling in principle but inaccessible for a researcher working alone on a specific archival question.

That has changed. Recently, I’ve been using Claude Code to build knowledge graphs not as final outputs, but as a daily method for taking research notes. It turns out that nodes and edges are a vastly superior format for historical research than anything I’ve used before.

Dry Rot in Britain: Temporal Knowledge Graph — tracing the rot from Pepys (1684) through scientific classification to HMS Queen Charlotte (1812).

Why Nodes and Edges?

Every historian has a system for managing the chaos of the archive. Word documents, Excel spreadsheets, index cards, Zotero libraries, folders of hastily snapped photographs. The fundamental problem is always the same: as the material accumulates, the connections between sources begin to fray. You know you read something about a specific merchant six months ago, but where? You know two events in different colonies are related, but the evidence is scattered across three different files.

Historians have always been interested in relationships, the problem is that our note-taking systems have never matched that interest. A knowledge graph solves this by recording the links between entities as rigorously as the entities themselves. Instead of burying the connection in a flat sentence that says, “Kyd sent Banks information about myrobalans from Calcutta,” you create nodes (Kyd, Banks, myrobalans, Calcutta) and define the edges between them: sent to, located in, mentioned in. The note and the underlying structure become the exact same thing.

A single sentence, “Kyd sent Banks information about myrobalans from Calcutta,” becomes a structured network of entities and relationships.

If you’ve ever used Gephi or sketched a network diagram on a whiteboard, you already understand the concept. Nodes are entities: people, places, commodities, institutions, sources, events. Edges are the relationships binding them together.

The shift is simple but profound. In a Word document, information is organized chronologically by when you found it. In a spreadsheet, it is constrained by whatever columns you guessed you might need at the start. In a knowledge graph, information is organized by what it is actually about. Every new note automatically connects to everything you already know about those same people, places, and commodity chains.

Share

Small Scale: Dry Rot and Traditional Research

I’ve been using this method to investigate when Serpula lacrymans, the destructive dry rot fungus, arrived in Britain. There is no bulk data to process here. This is entirely traditional research: searching through digitized eighteenth-century sources, following citation trails, cross-referencing dates. The kind of work where you are deep in Eighteenth Century Collections Online and Google Books, chasing footnotes between building manuals, parliamentary records, botanical surveys, and newspaper archives.

Now, every time I find a new source, I don’t add a line to a spreadsheet. I add nodes and edges. The graph has 43 people, 62 sources, 25 events, 2 organisms, 60 relationships, and 15 open questions, all in a single JSON file. Claude Code handles the data entry. I describe what I’ve found in plain English, and it maintains the structured knowledge graph and an interactive timeline visualization.

The graph forces distinctions that flat notes let you fudge. Samuel Pepys gathered toadstools “as big as my Fists” from neglected ship holds in 1684, and that’s often cited as early evidence of dry rot. But when I added Pepys as a source node and connected it to the organism nodes, the graph forced a question: which organism? The conditions Pepys describes are textbook for native wet rot, not the invasive Serpula. Because the graph holds both organisms as separate nodes with distinct diagnostic features, that distinction stays visible every time a new source mentions “rot.”

The graph also holds what I don’t know. Those 15 open question nodes (missing volumes, unverified claims, sources I haven’t yet read) are as important as the evidence nodes. In a Word document, unanswered questions get buried between paragraphs. In a graph, they stay visible, connected to the sources that raised them, waiting.

Subscribe now

Medium Scale: The Banks Leather Network

The dry rot graph is small and growing, but the Joseph Banks leather project demonstrates what happens when this method scales up and merges with machine transcription.

It started with a trip to the Sutro Library in San Francisco, where I photographed 181 handwritten letters related to the British leather trade (1797–1817). I used Claude Code to write processing scripts, which sent the photographs to Google’s Gemini for handwriting recognition, extracted structured entities from the transcriptions, and output the results as nodes and edges. I then read the key letters in chronological order to confirm the Gemini transcriptions and better understand the knowledge transmission between India and London.

I then integrated the full correspondence network from Warren Dawson’s Calendar of the Banks Letters, nearly 7,000 entries, and pulled in select figures related to leather and India from Neil Chambers’ Indian and Pacific Correspondence. But the core of the work was still note-taking. I was trying to answer a specific question: How did Banks identify catechu as a viable tanning agent?

The resulting leather network contains 275 people, 55 commodities, 119 places, and 46 institutions (495 nodes and 1,191 connections). That is a modest dataset, but it revealed global connections that sequential reading simply couldn’t.

The graph made it visually apparent that Banks operated as a switchboard connecting Indian botanical research directly to British industrial policy, routing knowledge between people who would never otherwise have crossed paths. Robert Kyd at the Calcutta Botanic Garden, Charles Jenkinson in the House of Lords, Samuel Purkis the tanner, and Andrew Berry conducting tanning experiments. The graph reveals them as vital components of a single coordinated commodity network, with Banks at the center.

Working through the graph drove new biographical research. It surfaced surprises, like the fact that Samuel Purkis and Humphry Davy were friends before Davy began working with Banks, and that Purkis had been corresponding with Banks throughout the 1790s. The graph didn’t explain the connection, but it surfaced it, turning a static name on a letter into a vibrant research question.

Visualizations as Research Tools

Anyone who has attended a digital humanities conference knows the problem with network graphs: they often look like impenetrable hairballs. Generated by NetworkX or Gephi with default settings, they are technically correct but analytically useless. You can’t tell what the graph is arguing because the layout isn’t designed to argue anything, it’s just a physics simulation.

Claude Code built the visualizations for the leather project too, and the one I’m most excited about is a temporal network.

Leather Crisis: Temporal Network, years anchored along the top, people pulled by correspondence weight.

Standard tools force you to choose between a timeline or a network graph. The temporal network fuses them. Years are fixed along the x-axis, and people float below, gravitationally pulled toward the years when they were most active. You can actually watch the network grow as the tanning crisis develops, spotting the transition as the conversation shifts from botanical science to industrial policy. This hybrid layout didn’t exist in a dropdown menu. It emerged from describing what I wanted to see and letting the AI implement it.

Because Claude Code builds through natural conversation, each visualization matches the specific historical question. Before this, custom visualizations were simply out of reach for most historians; you either used what Gephi, QGIS or Tableau offered or you didn’t visualize at all. Now, iterative, question-driven visualization is just another part of the research workflow.

Large Scale: From Personal Graphs to Linked Open Data

The dry rot and leather graphs are personal research tools: my nodes, my edges, my questions. But this exact logic scales up to major linked open data projects.

Take LINCS (Linked Infrastructure for Networked Cultural Scholarship), a project building linked open data infrastructure for cultural heritage in Canada. Our Historical Canadians dataset builds on the Dictionary of Canadian Biography with Wikidata, modeling familial connections, occupations, and residences using CIDOC-CRM, all queryable via SPARQL.

That is the far end of the spectrum: formal ontologies, institutional infrastructure, and strict interoperability. My personal graphs are not FAIR data (not Findable, Accessible, Interoperable, or Reusable in the formal sense). I can and do share them, but they are not interoperable. Making them so would mean modeling every entity against a formal ontology like CIDOC-CRM, reconciling every node to a persistent identifier, and minting URIs when the PIDs do not exist, a significant step up from a JSON file and a conversation with Claude. The underlying logic, entities and relationships, is the same, but the distance between a personal research graph and a LOD dataset is real.

That said, we are actively experimenting with using LLMs to reduce this barrier, and the results are promising. But today, there is no easy path from a Claude subscription to standards-compliant linked data. The habits of structured thinking you develop when building a small graph are exactly the habits required to engage with large-scale projects like LINCS, but the tooling to bridge that gap is still emerging.

Try It

The barrier to entry has collapsed. You don’t need a massive grant or a dedicated developer to start. You just need a compelling historical question, a willingness to think in nodes and edges, and the right AI tools to help you build. If you’re a historian still managing the chaos of the archive in Word documents and spreadsheets, consider nodes and edges instead. Start small. The graph will grow with your research, and you might be surprised by the connections it reveals.

In my Data Methods & History class, the answer is that I require students to use AI for coding. They simply could not get up to speed fast enough to do cool stuff without leaning on its support. We get to think about high-level questions about the origins of data, mathematical reasoning, and storytelling without worrying about code syntax. It is much more fun.

Yet, I also teach normal classes like my Social Theories of the Economy, which is a year-long sequence from Malthus to now. It is a course that would have been legible ten years ago as a standard-issue economic thought class. And in that class, we still assign essays.

Subscribe now

Many of my colleagues have given up on essays. I get it. It is frustrating to have to police. But at least in my 15 person seminar, I trust my students. I trust them because I could see the fear in their eyes on the first day when I told them that they had a choice: either learn to read, write, and think better than a machine, or have no economic value—outside of organ donation. My class, in a very real way, is about the accelerando, the growth curve of capitalism, and how that happened. I tell them that they are living through another moment in that story. And it is scary.

Nobody talks to them like this. AI is the center of their fears for the future and nobody tells them that their fears are both valid, and can be overcome. I do this. You can too. Because we know, as historians, that people have been told for centuries that they are replaceable and worthless—and that that isn’t true. It was not true when the mechanical thresher liberated us from the harvest. It was not true when the power loom took over our weaving. It was not true when the robot did our spot welding. Humans, it turns out, have capacities that only come out once the machines take over tasks.

Were these moments peaceful and the transitions just? Absolutely not. But I think we also have an opportunity here to learn from the past and consider our path forward as a society. Will that happen? Probably not. People will people.

Our students, however, do have choices. They can choose to trust you when you tell them that they can learn. My students have trusted me this year and despite what you hear, they all read hundreds of tough pages of Economy and Society, Capital Volume 3, and Capitalism and Freedom, and then they wrote their essays. That said, we also did oral exams, so I do know that they know their stuff.

So, as you come to the end of your semester, I would encourage you to think about why you got into this business. I cannot imagine anyone became a teacher because they love grading and finding plagiarism. It seems like the easy way out is to jettison the essay, but then students don’t learn how to make arguments. It is easier than policing AI. Yet there is an even easier way: to restore trust by getting real with the students. By speaking to their fears, you are also speaking to their hopes, our hopes for a different future.

Share

The actual email I sent my students this afternoon:

Class,

Please find three terrible essays that ChatGPT believes, in its wisdom, are “A” papers.

I would encourage you to contemplate the wreckage of your lives if you turned in something like these essays.

First, let’s not be too negative. There are upsides. The spelling is top-notch. The grammar is correct. The questions are actually not terrible, terrible. For a computer, I am totally impressed.

So, professor, why are they so bad?

1) ARROGANCE: None of these papers actually proves their points in a grounded, textual fashion. Do you see well-integrated quotes? Do you see footnotes to the text? Do you see close readings of key terms? [no] Is there a sophisticated argument spanning ten pages? No. Not at all. There is no complexity here. It is reduction.

2) MATH: The three essays, in total, are about 2000 words. Yes. So if someone were able to divide 2000 by 3, which no human ever has, we would know how much smaller these are.

3) SURFACE: These essays are all superficial. They are excellent summaries of key ideas, e.g., the double movement, but don’t really engage the text. They are dressed up notecards.

In the pre-ChatGPT era, I might have read this and thought “ok, those sentences are coherent. The student doesn’t really make much of the texts. It is weird that there are no quotes.” But now I think, “clearly an AI did this. I hope that student has other skills than thinking or writing or reading because that is the precarious future awaiting.”

Is it worthwhile talking about some of the readings with an AI? For sure. Talk with an AI. Talk with anybody. But don’t rely on AI or someone else to write your paper for you.

If, as you read them, you do not see these deficiencies then ask a friend. Or come to one of the teaching staff. I assure you, however, that these are terrible.

Best regards,

Prof. Hyman

PS: You are thinking, “but it offers to add footnotes!” Uh huh. You think that would turn out well?

PPS: “What if I iterated on this?” Nope. It would still be terrible. And even more choppy. AIs are not capable of sustained concentration.

PPPS: “But maybe I will just use it to read over my paper for grammar?” Yeah do that. I do that all the time. That’s fine. It is a super spell checker. Just don’t let it overwrite your voice with a million em-dashes or the word “delve” or some other fancy talk. Treat it like a friend. You would let a friend read your paper, tell you where you messed up a conjugation, or where something made no sense. That’s fine. But then you still write it. Do that. I know you can.

[PROMPTED]

-----

Got it, that constraint actually makes these much better, and much harder in the right way.

Below are three full essays (≈2100–2400 words each) using only second-semester materials from your syllabus. Each one:

• frames a clear, non-obvious question

• centers a tension

• compares multiple thinkers across the semester

• stays internalist (ideas arguing with ideas)

• avoids drifting back into Smith/Ricardo/Bentham/Mill

ESSAY 1

Can Markets Create Social Order Without Destroying It? From Weber to Polanyi to Hayek

Question

Can market societies generate stable social order on their own, or do they inevitably undermine the very social structures they depend on?

Introduction

One of the central promises of modern economic thought is that markets can coordinate human activity without centralized control. From prices to competition, the market appears as a self-regulating system capable of producing order out of individual action. Yet across the second semester readings, this promise is repeatedly questioned. Markets generate coordination, but they also appear to dissolve the social bonds and norms that make coordination possible in the first place.

This essay argues that Weber, Polanyi, and Hayek articulate competing but deeply related visions of the relationship between markets and social order. Weber shows that markets depend on forms of rationalization and impersonal authority that reshape social life. Polanyi argues that marketization disembeds economic activity from social relations, producing instability that requires corrective intervention. Hayek, by contrast, defends the market as a superior information-processing system but must implicitly rely on social norms he cannot fully theorize. Taken together, these thinkers reveal a central tension: markets produce order, but only by transforming, and often eroding, the social foundations on which that order rests.

Weber: Markets as Rationalization and Impersonal Order

Weber provides the starting point by redefining what “order” means in a market society. In premodern contexts, economic activity is embedded in tradition, status, and social obligation. Exchange is governed by relationships, not abstract rules. The rise of markets transforms this structure. Economic action becomes oriented toward calculation, efficiency, and profit.

What is striking in Weber is that the market does not eliminate order, it replaces one kind of order with another. The “impersonality” of the market is not a lack of structure, but a new form of structure. Individuals interact not as members of communities, but as participants in a system governed by prices and contracts. This shift is reinforced by the rise of bureaucracy, which Weber describes as the most rational form of organization. Bureaucratic authority replaces personal authority, making economic life predictable and calculable.

Yet this transformation comes at a cost. Weber’s famous “iron cage” is not simply about alienation, it is about the narrowing of social life. Rationalization produces efficiency, but it also strips away meaning and reduces human relationships to instrumental interactions. The market creates order, but it is a thin form of order, one that depends on formal rules rather than substantive social ties.

Weber does not argue that markets destroy society, but he does show that they fundamentally reshape it. The question becomes whether this new form of order is sustainable, or whether it generates tensions that cannot be resolved within the system itself.

Polanyi: The Disembedding of the Economy

Polanyi radicalizes Weber’s insight by arguing that the market does not simply transform social relations, it attempts to separate itself from them entirely. In The Great Transformation, Polanyi describes the rise of the “self-regulating market” as a historically unprecedented development. For most of human history, economic activity was embedded in social institutions. The attempt to create a market system governed solely by price signals represents a dramatic rupture.

The key mechanism of this rupture is the commodification of land, labor, and money. These are not ordinary commodities, they are what Polanyi calls “fictitious commodities.” Labor is human life, land is nature, and money is a social relation. Treating them as commodities subjects them to market forces in ways that destabilize society.

Polanyi’s central claim is that this disembedding produces a “double movement.” On the one hand, markets expand and attempt to govern more areas of life. On the other hand, society pushes back, creating protective institutions to shield itself from the destructive effects of market forces. This dynamic suggests that markets cannot sustain themselves without intervention. Far from being self-regulating, they require constant political and social correction.

What is crucial here is that Polanyi reframes the problem. The issue is not whether markets can produce order, they clearly can. The issue is whether that order is compatible with social stability. For Polanyi, the answer is no. The logic of the market undermines the very conditions that make social life possible, forcing society to intervene.

Hayek: Markets as Knowledge Systems

Hayek offers the most powerful defense of the market against these critiques. In “The Use of Knowledge in Society,” he argues that markets are not just mechanisms of exchange, but systems for processing dispersed information. No central authority can possess the knowledge required to allocate resources efficiently. Prices function as signals that coordinate individual actions without requiring anyone to understand the system as a whole.

From this perspective, the market’s impersonality is a strength, not a weakness. It allows coordination among strangers without requiring shared norms or values. Order emerges spontaneously from individual actions, guided by price signals.

Hayek’s argument directly challenges Polanyi. If markets are superior information systems, then attempts to regulate them may disrupt the very processes that make them effective. The “double movement” becomes, in Hayek’s view, a source of inefficiency and distortion.

Yet Hayek’s position contains a tension of its own. While he emphasizes the role of prices in coordinating behavior, he also acknowledges the importance of norms, rules, and institutions. Markets require a framework of law and social trust to function. Contracts must be enforced, property rights must be protected, and individuals must adhere to certain expectations of behavior.

What Hayek cannot fully explain is how these norms are maintained. If markets dissolve traditional social structures, where do the norms that sustain them come from? In defending the market, Hayek implicitly relies on social conditions that his theory does not account for.

The Tension: Order Through Disruption

Taken together, Weber, Polanyi, and Hayek reveal a shared problem from different angles. Markets clearly produce a form of order. They coordinate activity, allocate resources, and enable complex systems of exchange. But they do so by transforming the social world.

Weber shows that markets replace traditional forms of social order with rationalized, impersonal structures. Polanyi argues that this transformation goes too far, disembedding economic activity from social life and generating instability. Hayek defends the market’s coordinating function but cannot fully account for the social norms that make it possible.

The tension can be summarized as follows: markets create order by disrupting existing forms of social organization. The question is whether this disruption can be contained, or whether it inevitably leads to instability.

Conclusion

The debate over markets and social order is not a simple disagreement about efficiency or fairness. It is a deeper question about the nature of social life. Weber, Polanyi, and Hayek all recognize that markets are powerful organizing forces, but they differ on whether that power is ultimately constructive or destructive.

What emerges from this comparison is not a clear answer, but a clearer understanding of the stakes. Markets are not neutral mechanisms, they are transformative institutions. They create order, but they also reshape the conditions under which order is possible. The challenge, then, is not simply to choose between market and state, but to understand how economic systems interact with the social world they inhabit.

ESSAY 2

Is Rational Choice Enough? From Buchanan to Sen to Foucault

Question

Can human behavior and social order be adequately explained through rational choice, or does this framework fundamentally misunderstand human action?

Introduction

Modern economic theory often begins with a simple assumption: individuals act rationally to maximize their interests. This framework underlies much of contemporary political economy, from models of markets to theories of governance. Yet across the second semester readings, this assumption is repeatedly challenged.

This essay argues that Buchanan, Sen, and Foucault represent three distinct responses to the limits of rational choice theory. Buchanan formalizes rational choice as a foundation for political order. Sen critiques its reduction of human motivation to self-interest. Foucault reconceptualizes rationality itself as a product of power and governance. Together, they reveal that rational choice is not simply a neutral analytical tool, but a contested framework with profound implications for how we understand human behavior.

Buchanan: Rational Choice as Foundation

Buchanan and Tullock’s Calculus of Consent extends rational choice into the realm of politics. Individuals are assumed to act in their self-interest, and political institutions are designed to aggregate these preferences. The goal is to create rules that allow individuals to cooperate while minimizing the costs of collective decision-making.

What is striking about Buchanan’s approach is its consistency. The same logic that explains market behavior is applied to political behavior. There is no need to appeal to moral commitments or social norms, individuals pursue their interests, and institutions channel these pursuits into stable outcomes.

This framework has considerable explanatory power. It provides a clear model of how cooperation can emerge from self-interested behavior. But it also relies on a narrow conception of human motivation. Individuals are treated as utility maximizers, and other aspects of human behavior are either ignored or subsumed under this model.

Sen: The Critique of “Rational Fools”

Sen’s critique targets this narrow conception directly. In “Rational Fools,” he argues that reducing human behavior to utility maximization ignores important dimensions of human life, including commitment, identity, and moral reasoning.

Sen’s key insight is that individuals often act in ways that cannot be explained by self-interest alone. They may sacrifice personal gain for ethical reasons, adhere to social norms, or act out of a sense of obligation. These actions are not irrational, they reflect a broader conception of rationality.

By expanding the concept of rationality, Sen challenges the foundation of rational choice theory. If individuals are not simply utility maximizers, then models based on this assumption may fail to capture important aspects of behavior. The problem is not just empirical, it is conceptual. Rational choice theory defines rationality in a way that excludes many forms of human action.

Foucault: Rationality as Governance

Foucault takes the critique further by questioning the very notion of rationality. In The Birth of Biopolitics, he argues that economic rationality is not a natural feature of human behavior, but a product of specific forms of governance.

For Foucault, neoliberalism does not simply describe how individuals behave, it shapes how they understand themselves. Individuals are encouraged to see themselves as entrepreneurs, constantly optimizing their choices. Rationality becomes a norm imposed by institutions and practices.

This perspective transforms the debate. The question is no longer whether individuals are rational, but how rationality is defined and enforced. Rational choice theory is not just an analytical tool, it is part of a broader system of power that shapes behavior.

Conclusion

The debate over rational choice is not just about the accuracy of a model, it is about the nature of human action. Buchanan, Sen, and Foucault reveal that rationality is not a fixed concept, but a contested one. Understanding its limits requires moving beyond simple assumptions and engaging with the complexity of human behavior.

ESSAY 3

Does Capitalism Require Inequality? From Du Bois to Polanyi to Robinson

Question

Is inequality an accidental feature of capitalism, or is it structurally necessary to its operation?

Introduction

Capitalism is often justified as a system that promotes efficiency and growth. Inequality is sometimes seen as an unfortunate byproduct, but not a defining feature. Yet across the second semester readings, inequality appears not as an accident, but as a structural component of the system.

This essay argues that Du Bois, Polanyi, and Cedric Robinson each demonstrate that inequality is deeply embedded in capitalist development. Whether through race, class, or global hierarchy, capitalism relies on forms of differentiation that enable accumulation. Inequality is not a deviation from the system, it is one of its conditions of possibility.

Du Bois: Race and Labor

Du Bois’s analysis of Reconstruction reveals how racial divisions were used to structure labor markets. The “wages of whiteness” provided psychological and social benefits to white workers, even as they remained economically exploited.

This dynamic prevented the formation of class solidarity and allowed capital to maintain control. Inequality was not simply economic, it was racialized, creating divisions that sustained the system.

Polanyi: Market Expansion and Social Disruption

Polanyi’s account of market expansion highlights how inequality emerges from the commodification of labor and land. As markets expand, they create disparities that destabilize society.

These disparities are not accidental, they result from the logic of the market itself. The attempt to treat labor as a commodity generates inequalities that must be managed through social and political intervention.

Robinson: Racial Capitalism

Robinson extends this analysis by arguing that capitalism has always been racial. The system did not emerge in a neutral context, it developed through processes of conquest, slavery, and colonialism.

Inequality is therefore not a byproduct, it is foundational. Capitalism depends on the exploitation of differentiated populations, and these differences are often constructed along racial lines.

Conclusion

The question of inequality is not external to capitalism, it is internal. Du Bois, Polanyi, and Robinson show that inequality is not simply a problem to be solved, it is a feature of the system itself. Understanding capitalism requires confronting this reality.

If you want, I can:

• Make one of these intentionally “B+” instead of A for contrast

• Add footnotes/citations in Chicago style

• Or write a fourth essay using environment/climate (Beck/Klein/Lomborg) which would be very strong pedagogically

Share

But here’s the problem. At the very moment when artificial intelligence is reorganizing the economy and the university, its central challenges are, in fact, historical ways of knowing. In computer science, tasks such as retrieval, benchmarking, and the establishment of ground truth all depend on the selection, comparison, and evaluation of documents within large corpora. Determining whether outputs correspond to reliable bodies of evidence is not a purely technical question. It is an interpretive one.

Historians have long developed methods for addressing precisely these challenges; it was these approaches that I review in the opening chapters of The Dangerous Art of Text Mining, showing how information scientists who attempt large-scale analysis without historians’ understandings of the bias of archives and algorithms has led to retracted journal articles and unsupportable findings. I argued that historians’ methods, paired with NLP strategies, could make a “smarter” data science, which matched inferences, algorithms, and sources .

Today, a growing body of work by historians shows that historians are on the forefront of the task of retooling AI to improve its retrieval mechanisms, confronting messy sources on historians’ terms. In “Data retrieval from local heritage books—Is artificial intelligence the solution?”, Robert Stelter and Rafael Biehler test LLM-based extraction against code-based and manual methods and show both the promise and the limits of AI when working with irregular historical books, reminding us that retrieval from the archive is never just a technical matter but a problem of source structure, error, and judgment. In Stewart Spencer Dean and Sanskriti Sinha’s “Retrieving information from unstructured historical sources using large language models”, the authors show that LLMs designed to mirror historians’ workflows can extract structured information from difficult historical documents more flexibly than older pipelines, while also underscoring how fragile reproducibility and transparency become when retrieval depends on opaque model behavior. In “Mapping the Latent Past: Assessing Large Language Models as Digital Tools through Source Criticism”, Daniel Hutchinson demonstrates how to evaluate LLMs as historians would evaluate sources: not only for fluency or accuracy in isolation, but for provenance, distortion, omission, and the ways they reshape historical memory. Taken together, these studies suggest that historians are not merely future users of AI tools; they are helping define what reliable retrieval, evaluation, and interpretation should mean in the first place.

The selection of the most relevant examples from a large corpus is an obvious example of an instance where historians turn to their own methods when selecting cases, assembling archives, and balancing typical and exceptional examples without collapsing their differences.

The ‘alignment problem,’ often framed as ensuring that AI systems produce trustworthy outputs, also presents a problem of managing agreement and disagreement across documents. At stake is not only accuracy but the structure of knowledge itself. As AI systems scale, they tend to normalize—to compress disagreement into dominant narratives, privileging what appears most frequent, coherent, or statistically central. Without explicit intervention, AI risks erasing this plurality, producing a flattened account of human experience that undermines democratic reasoning. Embedding historical methods into AI systems—methods attuned to comparison, contradiction, and the coexistence of multiple valid perspectives—ensures that technological scale does not come at the expense of epistemic diversity. In this sense, history is not simply a content domain for AI, but a necessary foundation for building systems that can represent the past, and reason about it, in ways that remain open, plural, and accountable.

Here, too, historians have solutions. Historians have long developed methods for identifying consensus, tracing dissent, and preserving plurality without reducing it to a single dominant narrative. They have worked on preserving small-scale differences of “little” details even within the large-scale arc of meaning-making typical of so-called “big history.” Historians have shown that what matters most in the record of the past is often not consensus but dissent: competing interpretations, marginalized voices, and unresolved conflicts that shape political and cultural change.

Beyond AI, the challenges of curating data extend across the disciplines that have engaged with data science. As scholars such as Arthur Spirling have noted, fields like political science increasingly rely on large-scale textual analysis but often lack robust frameworks for ensuring that their corpora are complete and representative.

This is not a trivial issue. To study labor history, one cannot rely on a single archive or even a handful of sites. One must assemble sources that reflect different classes, migrant and ethnic experiences, institutional perspectives, and temporal changes shaped by major events. The problem is not simply one of scale, but of completeness: bringing together heterogeneous materials into a coherent evidentiary base.

This challenge extends to law, medicine, sociology, and beyond. The interpretation of large textual corpora depends on assembling archives that are sufficiently comprehensive, balanced, and historically grounded. The expertise required to build and evaluate such archives—understanding what is missing, what is overrepresented, and how sources relate—has traditionally been concentrated in history.

The expertise, in other words, already exists. However, it is dispersed across institutions such as Saskatchewan, SMU, George Mason, Clemson, and Waterloo, rather than integrated into the central infrastructures of major research universities.

Yet the university as a whole increasingly depends on precisely this kind of knowledge. History can contribute to AI by providing methods for retrieval, evaluation, and the preservation of plurality. It can contribute to the world by enabling large-scale understanding of problems such as climate governance, where decades of negotiation must be analyzed across multiple actors and perspectives. It can contribute to other disciplines by offering frameworks for assembling and interpreting complete and representative archives.

The next step is clear. What is needed is not another generation of small projects, but a coordinated effort—a kind of archival and analytical moonshot—that brings together historians, computational experts, and domain specialists to build comprehensive, multi-source, multi-lingual corpora and the methods to interpret them.

Such an effort must be led by historians, because the central problem is not simply one of computation, but of judgment: what counts as evidence, what counts as completeness, and how meaning is constructed across time.

The digital breakthrough will happen when those questions are treated not as peripheral, but as foundational.

Share

As a historian and archivist, I spent years thinking the same thing. But my projects—and yours, fellow historian—might just belong on that cluster too. Consider the handwritten records you’ve been photographing on research trips; or the correspondence you’ve been transcribing in the reading room; or those thousand-page minute books, the land surveys, the immigration registers rich with details of space and place. These collections are exactly the kind of work that research computing was built to support. And the tools available right now make the case almost embarrassingly easy.

Share Computational History

What changed?

A generation of powerful, open-source vision-language models arrived. These are models that can look at an image of a faded, ink-stained, overlapping-cursive index card and return structured, searchable data. Not perfect data. But useful data: the kind that turns a microfilm scroll into a queryable database. They handle handwriting, mixed layouts, rubber stamps, marginal annotations; the visual noise and contextual complexity that defeated traditional OCR for decades. And because they are open-source, they can run on hardware your institution might already own.

Two projects, one architecture

My colleagues and I have been testing this across two projects that sit at very different ends of the archival spectrum.

The first involves over a million government administrative index cards held at a federal archive: cards dense with classification codes, agency stamps, and idiosyncratic cursive spanning decades of bureaucratic correspondence. The second involves handwritten court

records from the eighteenth century: daily administrative logs of civil litigation, five to fifteen case entries per page, produced by clerks whose penmanship was optimized for speed, not legibility.

Even though these records come from different centuries, different hands, different archives, the pipelines we built to process them are effectively the same. We take scanned images, feed them to a large vision-language model with carefully designed prompts, and get back structured output in the form of JSON records with fields like dates, agencies, parties, classifications, and body text. The model and pipeline handle the reading and the data extraction. And critically, it runs not on an expensive commercial API but on university research computing nodes equipped with GPUs, using open-source models we downloaded and deployed ourselves.

You probably already have access

The barrier to entry is lower than you think. This is the part I want historians to hear most clearly. If you are affiliated with a research university, a library consortium, a national lab, or any institution with a research computing group, you almost certainly have access to GPU-equipped clusters. These machines sit in basements and server rooms running jobs for chemists and engineers, and they have capacity. Research computing groups are, in my experience, genuinely eager to support humanities projects. They want the diversity of use cases and they want to demonstrate broad impact.

For the index card project, we run a 72-billion-parameter open-source vision model on NVIDIA A100 GPUs through a Slurm-managed pipeline. It processes thousands of cards per hour, around the clock, for the cost of allocated compute time, which, at a university, often means free or close to it. No per-token API fees. No data leaving your institution’s network. No licensing negotiations. Open-weight models like Qwen, LLaMA, and Mistral can be downloaded and deployed on institutional hardware with nothing more than a few configuration files and some experimentation.

Compare that to running the same work through a commercial API. At current pricing, processing over a million images through a cloud vision-language model would cost tens of thousands of dollars. On a university cluster, it costs compute time that was already budgeted and even where it isn’t, the cost is a fraction of the commercial alternative.

Privacy by default

There is another benefit that matters enormously for the kinds of records historians work with. When you run models on institutional infrastructure, your data never leaves the building. No images uploaded to a third-party server, no extracted text passing through someone else’s cloud. For projects involving records tied to individual people, families, and communities — personnel files, medical histories, correspondence, community administrative records — this is not a minor convenience but an ethical requirement. Running on local compute gives you that by default.

The real barrier is permission: your own

The honest obstacle for most historians is not entirely technical. It is the feeling that their project is too small, too niche, too humanistic to justify claiming space on a shared computing resource. Perhaps processing a few thousand court documents does not warrant the same infrastructure that simulates protein folding.

But consider what is actually happening when you run a model over a collection of historical documents. You are converting unstructured, inaccessible primary sources into structured, searchable, analyzable data. You are building something that other researchers, genealogists, and communities can potentially use. You are doing exactly what research infrastructure exists to support.

Remember, as well, that the scales vary. Not every project is a million images. Sometimes it is four hundred pages of minute books, or a single box of correspondence. The pipeline scales down just as well as it scales up. A few hundred images can process in an afternoon. And the experience of building that first pipeline — writing your first extraction prompt, submitting your first batch job, seeing structured records come back from documents you thought were illegible — changes how you think about what is possible for every collection you encounter afterward.

Your project is not too small nor too messy. The resources are closer than you think. And the learning curve is shorter than the one you already climbed to get into the archive.

Github: https://github.com/uvalawlibrary/hpc-vlm-starter

Share