I think there needs to be some encouragement and translation to help rectify this. There is already a burgeoning amount of interesting applied econometrics work done by wonderful researchers (too many to name, but including Kevin Chen, Jonathan Roth, Michal Kolesár, Peter Hull, Ashesh Rambachan, Mikkel Plagborg-Møller, Davide Viviano, Isaiah Andrews, Soonwoo Kwon and many others). Some of these papers have seen significant take-up and interest, often because they are targeted at applied researchers.

The Journal of Economic Perspectives (JEP) produces a wonderful column that suggests interesting papers and columns to read. That column:

will list readings that may be especially useful to teachers of undergraduate economics, as well as other articles that are of broader cultural interest.

So: this column will attempt to list econometrics papers and readings that may be useful to applied researchers in economics and finance. Since often it is not clear why these papers are useful, I will attempt to also give a brief explanation as to why and how to consider it.1

Some ground rules:2

1. I will look at econometrics articles published in Econometrica, American Economic Review, Quantitative Economics, Review of Economics and Statistics, Journal of Business & Economic Statistics, JASA, and Journal of Econometrics in the last year (probably the last quarter, but giving myself some flexibility).3

2. I will focus on papers where there is an immediate insight useful for practitioners.

3. I will propose an example empirical setting where I think this might be relevant — bonus points if I can point to an existing research paper where I think they should’ve used the method.

4. I won’t write more than 2 paragraphs per article.

5. I will write about at least 5 papers per column

6. I will write this once a quarter (it’s July now, so next one in October).

Ok, here goes nothing. If you like this, let me know. If you hate it, let me know too.

Further Reading in Econometrics for Applied Researchers, Vol. 1 (Aug 2026)

Factorial Difference-in-Differences

Yiqing Xu, Anqi Zhao, Peng Ding, JASA
https://doi.org/10.1080/01621459.2026.2628343 (wp version: https://arxiv.org/abs/2407.11937)

When we’re thinking about difference-in-differences, we lump together many different types of designs. The most canonical design is a setting where we have a policy that gets applied in a post-period to a set of treated firms, but the treated firms are non-randomly selected, hence identification challenges. But, another very common design is a setting where units look different in a static sense (perhaps they are counties with different housing supply elasticities), and we postulate that an aggregate shock differentially affects these places.

Turns out that this latter approach is not obviously identical to standard difference-in-difference, once you think carefully about it as a treatment. Xu et al. (2026) term this “factorial difference-in-differences” (which I think is a terrible name, but ¯\(ツ)/¯). It turns out that for this design to identify the causal effect of the baseline differences (e.g. the housing supply elasticity) using this design, you need an additional assumption that assumes mean independence between the baseline differences and the potential outcomes.4 Fortunately, I believe this assumption is quite similar to what researchers typically believe when they invoke this approach.

Key takeaway for empirical researchers: you need to justify your assumptions more explicitly here, and highlight that potential outcomes in the absence of the aggregate shock need to be mean-independent of the characteristic.

The Effect of Omitted Variables on the Sign of Regression Coefficients

Matthew A. Masten, Alexandre Poirier
American Economic Review 116(7) — 2026-06-30
https://doi.org/10.1257/aer.20230242 · open access

Oster (2019, JBES) is an incredibly influential paper5 that expands and synthesizes the intuition from Altonji, Elder and Taber (2005) about how to try to bound the effect of unobservables. We all know that if you have omitted variables, your estimate can be biased. Often, we add additional controls and show that as we add more controls, our estimates of interest are “stable” (e.g. don’t change across specifications). Oster (2019) provided conditions and a test for how to think about this approach, and a formal test that exploits the changes in the coefficient measure and the R² to study how much omitted variable biasthere would need to be to drive the true effect to zero.

Masten and Poirier (2026, AER) is a sharp critique of this approach. Oster (2019)’s test (δ) measures how much selection on unobservables takes to drive the coefficient to zero, but that’s different from how much it would take to flip its sign. It turns out these can be very different. Since the δ test is a ratio of coefficients, the bias-adjusted estimate is discontinuous in δ and can jump across zero rather than passing through it. The key takeaway is to consider the “sign-change” breakdown point alongside the Oster (2019) measure. All of it is implemented in the regsensitivity Stata package.6

Key takeaway for empirical researchers: when justifying your design based on the stability of coefficient estimates, you have another important test to run and interpret.

Double Robustness of Local Projections and Some Unpleasant VARithmetic

José Luis Montiel Olea, Mikkel Plagborg‐Møller, Eric Qian, Christian K. Wolf
Econometrica 94(4), 1313–1343 — 2026-07-16
https://doi.org/10.3982/ECTA23345 · open access

VAR (Vector Autoregressions) and LP (local projections) are a lynchpin of macro analysis. If you’re more of a micro person, you probably only remember VARs from your first year macro courses. One way to think about LP is that it’s semi-parametric in the dynamics of how shocks propagate, while VARs are very structured and parametric in their propagation mechanism. Both LP and VAR consider the structural model, but they differ in how they propagate the shock forward. E.g. imagine you want to study the impact h periods ahead. Here’s the two processes written down:

It turns out when the first model (VAR) is correctly specified, there’s a direct mapping between LP and VAR such that they’re equivalent! E.g. LP is the reduced form version of the VAR model. LP does not restrict how the propagation maps forward. But if the VAR is misspecified, the VAR does very badly. In Olea et al. (2026), they show that if the VAR is misspecified, the confidence intervals are wrong. However, the local projection approach will have the right coverage rates, even under misspecification!

Key takeaway for empirical researchers: Applied researchers should prefer LP for robust inference unless they are willing to use extremely long lags that render the VAR interval as wide as the LP interval.

Optimal Shrinkage Estimation of Fixed Effects in Linear Panel Data Models

Soonwoo Kwon, Econometrica — 2026
https://doi.org/10.3982/ecta22386 (wp version: https://arxiv.org/abs/2308.12485)

Empirical Bayes When Estimation Precision Predicts Parameters

Jiafeng Chen, Econometrica — 2026
https://doi.org/10.3982/ECTA22935 (wp version: https://arxiv.org/abs/2212.14444)

Fixed effects have historically been nuisance parameters for estimation (e.g. when we include unit and time fixed effects, we are not interested in those parameters). However, now a lot of applied work in economics and finance tries to directly estimate unit-level effects and then doing something with them: teacher value-added, judge leniency, hospital quality, neighborhood effects, or manager or fund-manager alphas.7 The challenge is that these estimates are very noisy, typically, with a limited number of observations. So how do we improve on the estimation?

The typical approach is empirical Bayes (canonically inspired by Morris (1982), and used to great effect by Chetty, Friedman, and Rockoff (2014) and Chetty and Hendren (2018), as well as many other papers. Basically (Bayes-ically?), since the raw OLS fixed effects are noisy, we shrink them towards the overall mean as a function of the noisiness of the raw estimates. The challenge is that parametric empirical Bayes rests on normality of the true effects, normality of the least squares estimator, and, the key focus of Chen (2026), independence between the true effect and the variance of its estimator (referred to as precision dependence). This last estimate is empirically implausible: class size is mechanically linked to the precision of the estimate and is plausibly correlated with teacher skill due to assignment, so the two are linked by construction. To solve this issue, Chen (2026) stays inside empirical Bayes and makes the prior richer by modeling the conditional distribution of the parameter.

Kwon (2026) also delivers robustness to precision dependence, but that is a byproduct of the approach. Kwon (2026) instead focuses on the fact that the object you have is usually not one number per unit but several, such as a teacher’s value-added in each of six years, or her effect on test scores and on graduation. Existing practice either averages across years, imposing that the effect never moves, or shrinks each year in isolation, ignoring that a teacher’s good year predicts her other good years and over-shrinking as a result. Kwon (2026) extends existing unbiased-risk-estimate results to the case where the tuning parameter is a full covariance matrix, so the rule shrinks linear combinations differentially, which creates a smoothed time profile, rather than a flat one. This allows for a one-period-ahead forecast of the effect, which is useful for a retention or promotion decision for a teacher.

The cleanest way to see the contrast is as a trade-off between random-effects and fixed-effects. Chen (2026) is the analogy to correlated random effects, whereby specifying a distribution correctly for the heterogeneity, you are rewarded with a rule that can be nonlinear and that uses the information precision fully. Kwon (2026) is the analogy to fixed-effects: nothing about the distribution of effects is modeled, so nothing there can be misspecified, but the price is that you need to pre-specify a class of (linear) decision rules, and optimality is only ever relative to that class.

Key takeaway for empirical researchers: the real choice is whether you would rather model the distribution of your unit effects or the class of rules you will consider for shrinkage. If you have one effect per unit and you model the prior, Chen’s framework is potentially more flexible and powerful if the model is specified correctly. If your effects vary over time or across outcomes, or you want a forecast, Kwon applies more directly and asks less of you distributionally. Either way, straight empirical Bayes is no longer an easily defensible default.

R Package for Chen (2026), R Package for Kwon (2026)

It’s sort of a “best hits” of my thoughts that I’ve accumulated since my talks with Markus this past spring, but you may find it useful. The materials (slides, etc.) from the talk are available and I also have a number of Claude Code recordings here.

This post is the capstone of the series. The earlier posts were about getting the agent to do things: setting up, analyzing data, scraping, handling large datasets, writing, sandboxing and specifying how agents think (skills). In most of those demos, I skipped over a step—Markus and I would look at the output, nod, and move on. This post is about what you do after the agent hands you a finished-looking result — e.g. real research.

A framing I find useful is about costs. AI has collapsed the cost of doing certain kinds of work: coding, data construction, empirical analysis, etc.1 The cost of verifying that the work is correct has moved much less. We’re economists; costly verification is an idea we’ve struggled with for decades. Now as the cost of actions has fallen, with a consequent increase in the relative cost of verification, many of us find ourselves doing a lot of the cheap activity, with potentially less verification. As a result, you can accumulate what people have started calling verification debt: a growing pile of AI-produced results yo haven’t actually checked.

A two-by-two makes the problem concrete (the picture above comes from a nice post by Clifford Russell on these challenges in the context of software engineering) . Put the cost of making something on the x-axis and the cost of verifying it y-axis. The danger zone is the cell where things are now cheap to make but still expensive to verify.

When both costs are high, there’s a huge incentive to spin the slot machine many times, and see if the agent just happens to nail the task.

Expensive-to-make work forced you to go slowly, and verification happened as a byproduct of doing. Now the code arrives finished, and the understanding is the thing you still have to build.

In the video, Markus asked a good question: how is this different from just not using YOLO mode and approving each step as it happens? The answer is a function of how much you verify and internalize the work that the agent is doing. The more autonomy you provide, the worse the problem gets (e.g. the ladder from the the post on sandboxes). If the agent works alone for 25 minutes and comes back with a repository, a figure, and a headline estimate, the verification simultaneously arrives at the end.

So the question for today: how do you build a work environment where verification is tractable in research?

The three things we’ll discuss today are not AI tooling, at least in essence. It’s the tooling software engineers built for exactly this style of problem: reviewing others’ code and work. But AI happens to supercharge these workflows.

1. Review the code as reported — git and a GitHub repository you (and your LLM) can read

2. Give feedback that routes back — to the agent, to coauthors, and to the permanent record.

3. Publish the results into the draft — each number in the paper traces to code by integrating Github into Overleaf

The demo: an IPO event study

I wanted something practical rather than a lecture, and the timing helped: SpaceX just went public, Hanno Lustig has been posting about it, and so a natural empirical question for us to study is how stock prices adjust after firms IPO (since SpaceX has been angling to get themselves included in indices where passive investment money can provide a significant pop in demand).

There’s a well-documented post-IPO underperformance pattern, so I asked Claude to replicate it. To Claude Code, I gave the task to estimate abnormal returns after IPOs.2 Use Jay Ritter’s IPO data as the event sample, and CRSP (via WRDS, which I have credentialed access to) for aftermarket returns and a benchmark. Report the average effect with a figure and a table.

I was deliberately vague about how exactly Claude should implement this.There are a few obvious decision branches: cumulative abnormal returns versus buy-and-hold returns; which benchmark to use; which event windows, how to handle delisting; how to match Ritter’s firms to CRSP identifiers; etc. I left this to the agent and I would eventually need to decide if the decisions were appropriate, similar to if an RA made it for me.

What I did specify, in detail, was the paper trail:

• Commit the code into a git repository locally at every meaningful checkpoint, with messages that say _why_, not just what.

• Keep a DECISIONS.md: each methodological choice, the alternatives rejected, and a confidence level.

• Keep a LOG.md: a plain-language narrative a coauthor could follow.

• Print a sample-attrition table at every stage—the N, what dropped, and why.

I let it run in auto mode for about 25 minutes. The resulting repository is public at github.com/paulgp/ipo-bump. I asked for the trail precisely so the work could be reviewed. That’s the discipline I want to convince you of: the request for verifiability goes in the prompt, before the work happens.

Part A: Review the code as reported

Two minutes on version control

What’s git? Here’s the cursory version. The joke about git is that if you ask someone who really knows it, by the time they’re done you never want to hear about it again. It’s built for enterprise software teams and has a hundred features researchers will never touch. You need about five of them.3

Version control is a committed set of historical versions of your work. We’ve all lived the alternative: working_code_v1.do, then _v2, then _final_final. What you actually want is one file plus a complete record of how it changed over time—track changes, but for code. That’s what git does: it looks at a folder, computes differences between versions of files, and keeps a permanent record of every saved state.

A commit is the act of saving a state into that record. Markus asked whether committing locks you in—no, it’s a snapshot, and you keep working afterward. If you’re a Stata person, it’s like a preserve statement: “record things exactly as they are right now.” GitHub is a remote website where the record can live in addition to on your computer, so coauthors can see it, you can use it across computers, and so you can read it with good tooling. A diff is the comparison between two states—new lines shown in green, deleted lines in red—which is exactly the “marked-up changes” view you’d want when reviewing anyone’s work.

One more piece, because Markus asked: the .gitignore file lists things git should never record. In our repo, Claude put the raw WRDS extracts in there (you don’t want big data files in the history—collaborators pull those themselves) and my database password. It actually listed the password file twice in my gitignore file, in two syntaxes. I don’t know why, but I had told it in no uncertain terms not to commit my password, and I appreciate the paranoia—I really did not want to make this video and simultaneously publish my WRDS credentials to the world.

Reading the repo like a referee

The agent hands back a repository with full history, a figure, a table, and a headline estimate: U.S. common-stock IPOs from Ritter’s database, 1975–2021, earn a mean three-year buy-and-hold abnormal return of −19.7% relative to the CRSP value-weighted index. In plainer terms: a dollar invested in these IPOs at the close of their first trading day grows to about $0.86 for every dollar the market would have delivered over the same three years, and 70.5% of individual IPOs land below the market. That passes the initial sniff test—it’s the post-IPO underperformance pattern the literature documents.

The mental model for this workflow is that work from an AI is work from your RA. This is code that you did not write, but you have a rough sense of what it should do. Unfortunately for you, it will often pass the sniff test — it looks finished and polished. But now the question is whether the task was done correctly.

Fortunately, you know how to do this! Every senior coauthor has reviewed a junior person’s work; every software engineer has done code review. Nothing about this task is new. What’s new is that most researchers never adopted the structure that makes it efficient.

The structure is the commit history. Because I asked for commits at every checkpoint, the work arrives in chunks: scaffolding, data loading, matching, return calculation, estimation and output. Commits change the unit of verification. You review one bounded diff at a time instead of confronting 800 lines of code at once.

You can walk the sequence as a story (pull → clean → match → compute returns → estimate) and ask whether the order makes sense. And when you later request a change, GitHub shows you only what changed: a 12-line diff against a baseline you’ve already reviewed. A repo with one giant commit doesn’t give you any of this; it’s just a blob with a `.git` folder attached.

The trail, and its limits

In Decisions.md, I asked Claude to record choices along the way. However, the decisions file claimed it had computed cumulative abnormal returns as a robustness check. But it hadn’t! There was no CAR figure, no CAR code. It also mentioned calendar-time portfolios—a useful alternative—and didn’t do those either. It claimed work it never did. Markus asked whether you can prompt this away—”please be careful and do every step”—and you can push in that direction, but the more reliable fix is smaller task chunks, which are easier to complete and easier to check. These systems can be as lazy as we are.

Concretely, here’s the checklist I’d work through for this particular task, each item a place where the code can be perfectly clean and the answer still wrong:

• Did it actually use Ritter’s event dates, or silently construct its own? (I traced this one through the data-loading commit: it builds the event date from Ritter’s offer_date field, correctly. We are trusting Jay here, but as Markus said—in this life, we trust Jay.)

• Do the computed first-day returns reconcile with Ritter’s published first-day returns, or does the repo just claim they do?

• How are delisted firms handled—are they in the sample, and which way does dropping them bias the mean?

• Is the benchmark defensible, and does the sign survive the obvious alternative (CAR versus buy-and-hold)?

That last one I handled by iterating live. I went back to the command line and asked Claude to add a cumulative abnormal return analysis alongside the buy-and-hold version, then commit and push. A few minutes later there’s a fifth commit, and the review burden is only the new diff. The CAR estimate comes in at −17.8% at 36 months against −19.7% for buy-and-hold, with the mean-median gap looking different across the two measures.

Part B: Feedback that routes back

Finding a problem is half the job. Getting the fix made—and recorded—is the other half. There are three audiences for your feedback: the agent, your coauthors, and future-you. The useful discovery is that one piece of machinery serves all three: GitHub issues.

An issue is a titled note attached to the repository—a to-do item with a discussion thread. Here’s the loop I demonstrated in the video. Remember the reconciliation question from the checklist? I opened an issue: “Validate return measures against Ritter’s initial returns—Ritter publishes first-day returns for these IPOs; compare our constructed measures against his and report how well we match.” Writing it felt like writing a Slack message to an RA, except it lives permanently next to the code.

Then, in a fresh Claude session with no memory of anything, I typed:

“Please review issue #1 on GitHub, code up an answer, then commit and push. Use the `gh` command-line tool”

The agent pulled the issue text, explored the repo to orient itself, did the analysis, pushed the result, and closed the issue. You don’t need to sit and watch; you could kick this off and check the answer from your phone. (You can see where this is heading—an agent that watches the issue tracker and handles items as they’re filed. We won’t build that in this series, but the shape of it should be visible.) You can see the issue here

The answer taught me something real: the exact first-day return is unrecoverable from the data as pulled, because Claude had built everything on CRSP’s monthly files. Monthly data has a closing price for the IPO month but can’t isolate the first trading day.

So the loop continues: I opened issue #3, “Reconstruct the analysis using daily data—we discovered in #1 that the pipeline uses monthly CRSP returns; switch to daily and aggregate up so we can match Ritter and support higher-frequency analysis.” Typing `#1` in the text auto-links the earlier issue, so the reasoning chain is preserved. Then, back at the command line: “Please tackle issue #3.” That issue is closed now too: the daily reconstruction confirms the monthly results, and along the way the repo grew a first-day-return analysis that recovers offer prices from Ritter’s files, Capital IQ, and SEC 424B prospectuses. Each round of review generated the next round of work, and the record shows the whole chain.

Two practices make this loop work well. First, be specific. “This seems off” gets you a vague fix. Name the join, the variable, the count you expected—”please diagnose why we lose four IPOs at the CRSP match stage” is a prompt the agent can act on and you can verify. Second, anchor feedback to code where possible: GitHub lets you comment on a specific line of a specific commit, so “check the merge keys on this line” carries its own context.

For coauthors, the same record does double duty. An email thread about the benchmark choice can get lost in an inbox, while an issue is remains indexed and referenced to the code and project. You can see what’s been checked, by whom, and what’s still open—verification distributed across the team and made visible. A trick I’ve come to like: after a seminar, file each question you got as its own issue, then ask the agent to go through them and draft the best available answer for each, flagging which ones require actual new analysis.

Markus asked about pull requests (PR), which you’ll see mentioned everywhere. A PR is a request to fold one branch of work back into the main line—”I’m done with this piece, please pull it in”—and it’s where professional code review formally happens. For a research team it’s worth adopting eventually, but issues alone get you most of the value, so I kept the demo there.

Markus also asked whether the verification itself can be done by another AI, such as Gemini or ChatGPT reviewing Claude’s code. It can, and people do run exactly that workflow; a second model reviewing the first catches real things. But I keep coming back to the line from the 1970s IBM training manual that I used in a previous post: a computer can never be held accountable. You still need a human being signing off on the task. The main goal of this workflow is to make this as painless and structured as possible.

Part C: Publish the results into the draft

The last leg is getting results into the paper. This part is probably the least necessary, but if you have AI work on compile and write parts of your paper, having a documented pipeline becomes more important.

A cardinal rule—which predates AI but matters more now—is that no number in the draft gets typed by hand.4 The pipeline emits results.tex and figure.pdf; the paper pulls them in using input and \includegraphics{}. Then, whenever you update the analysis, it is straightforward to update the paper. With AI in the loop, you avoid having the results be written into the paper based on the model’s context window, but are instead directly mapped by code.

The integration that makes this smooth: Overleaf can sync directly with a GitHub repository. (Note: this sits behind Overleaf’s paid tier, though only the project owner needs the subscription—once the project is linked, any collaborator on it can hit the sync button. Markus’s alternative also works: Overleaf syncs with Dropbox, so you can point the pipeline’s output folder there instead—but Dropbox sync is a premium feature too, so there’s no free route to automatic syncing.) In the video I linked the ipo-bump repo to an Overleaf project, pulled, and every file appeared—figures, tables, the logs. I wrote a skeleton document referencing the results figure, pushed from Overleaf back to GitHub, had Claude write up the results section from the command-line side, and pulled again in Overleaf. The draft updated with the new text and the current figures. Nobody copied and pasted anything, and the draft’s own edit history now lives in the same git record as the code.5

The payoff is provenance. When a coauthor questions a coefficient, you walk backward: which table, which script, which commit, which logged decision. “Let me dig through my files” becomes a five-minute trace by Claude Code. In most cases I would simply ask AI to figure out where a decision got made, and since the commit history keeps track of all changes, it is a quick task.

Note that a correctly wired pipeline guarantees the number in the paper matches the number the code produced but doesn’t promise correct code. Clean provenance can faithfully publish a wrong estimate—which is why Part A comes before Part C. Review the method, then wire it into the draft.

Wrapping up the series

The loop this post assembles—act, review, publish, critique, re-run—works well for me. I still have far too much to review, but it keeps it organize and me sane.

1. Ask for the trail before the work happens

2. Review commits, not full codebases. The unit of verification should be a bounded diff.

3. Documentation is a map, not a verdict. LLMs are happy to claim results have been done that are not in fact there. Make a point of verifying (and even consider having another AI review)

4. Route feedback through issues. Use issues to keep track of tasks! The paper trail is invaluable to humans and agents.

5. No hand-typed numbers. Code emits, the draft ingests, git connects them. And review the method before you wire it in.

AI is good at this. Researchers mostly never adopted git—the perceived setup cost was too high for the benefit. That cost is now roughly zero. Tell Claude at the start of a project that you want everything tracked in git and GitHub, and it just does it, happily—it has the spirit of a good software engineer living somewhere deep inside. You get the best practices without the years of accumulated scar tissue that software engineers paid for them.

Markus closed by asking where this all goes in one year, in five. Honest answer: five years is unknowable, and even one year ago looked completely different from today. My guess is that the rough edges—the feeling of being in the Matrix at a command line—smooth out, and it becomes much clearer which parts of the workflow are human problems and which are computer problems. I don’t think this automates research away; it’s comparative advantage all the way down, and taste, judgment, and accountability stay on our side of the ledger. Thirty years ago, running a regression with many fixed effects was a real computational constraint that shaped what questions people asked. Now we run a thousand of them without thinking. Verification of AI-produced work will feel like that eventually, once the tooling makes it routine—it will still matter just as much, and it will have stopped being the constraint.

Thanks for following along with the series. I’d like to hear what your biggest pain points are with these tools—that’s what future posts will get built from.

You can see the computer age everywhere but in the productivity statistics.

-Robert Solow (1987)

Will AI, with its ability to autonomously write papers from start to finish, cause an explosion in research papers and the collapse of the academic publishing pipeline?

Agentic tools are incredible but human society is complex and hard to automate. It’s not clear whether people see these tools and say “Oh yes, I need to write twice as many papers.” They may work more intensely on papers, but they may also choose to spend more leisure time as it becomes easier to get work done! Or they may work through ideas quickly and find the better papers to focus on. It’s an empirical question.

I am going to provide four data sources to try to answer the question: “are people producing more working papers?”1

1. arXiv’s working papers for economics, statistics, and machine learning (a CS category)

2. the NBER’s working papers

3. SSRN’s working paper series (by far the hardest-to-get and least complete data)

4. AEA journal submissions

The key question will be exactly how to measure this. Across almost every data source, the number of papers is rising; this was true long before AI. So instead, I’ll focus on the change in the growth rate of working papers, which will capture the acceleration due to AI:

The punchline: growth rates did tick up, but by far less than a true deluge would imply. SSRN is the one venue with a clear acceleration.2 And in the field where I’d have bet on the biggest jump — machine learning — the growth rate didn’t move at all: 18% per year over 2022–2024, and 18% again over 2024–2026.

Journal submissions at the AEA are a partial exception — they’ve risen recently — but as I’ll show, that’s partly a rebound to pre-2019 levels rather than new growth. More on that below.

arXiv

arXiv is a public working paper repository that hosts for a variety of scientific topics, including economics (although it is not nearly as important as SSRN right now). Papers can be posted on the site, and then updated anytime they are changed. Arxiv has the useful feature that every version of a paper is time-stamped (very useful for fields like math, where precedence is very important).

An extremely important cautionary note about arXiv — it is easy to accidentally use the “last-updated” date for papers when indexing. However this will bias your results towards finding a large recent increase in papers. Below, I plot the number of new papers posted per month in the three economics arXiv mailing lists.

In economics, we see that when we correctly measure the dates, there is a slight recent increase in papers, but nothing outrageous. Comparing the annualized rate of change between 2022-2024 to 2024-2026, we see a slight increase in the rate from 14% to 19% per year.

This story looks similar in other arXiv fields. In Statistics, the rate from 2022-2024 was 9% per year, and in 2024-2026, it grew to 14% per year.

In Machine Learning (a computer science mailing list), posting rates have grown at the same steady rate since 2022!

National Bureau of Economic Research

An alternative mailing list for working papers is the National Bureau of Economic Research (NBER). This mailing list is unique in that the number of people who can post to it is held in very restricted quantity.3 Hence, any changes here on production would really be an extensive margin story among very research-active scholars.

In this setting, we see a similar change from 2022-2024 (5%) to 2024-2026 (9%) as before, but with lower growth rates in both periods.

SSRN

The Social Science Research Network (SSRN) is the dominant social science platform for economics and finance working papers. It’s also owned by Elsevier and hides all its information. It requires all sorts of insane logins and clickthroughs to use and is overall a terrible service. You should stop using it.

That being said, it would be unfair to do any analysis of the state of working papers in economics without looking at it. So, I had to hack together a horrible solution.

Using the Wayback Machine, I pulled SSRN’s own reported totals — the cumulative stock of papers — for its two main categories, the Economics Research Network (ERN) and the Financial Economics Network (FEN), at a handful of snapshots over time. There weren’t many, but enough for a rough read.

From 2020 through 2025 the stock grew at roughly 5% a year, with a recent acceleration to about 10%. Treat this as the softest evidence in the piece, though: it’s by far the sparsest and least reliable of the four sources and, ironically, the only one showing a clear pickup in working papers.

AEA Journal Submissions

I collected data from the Report of the Editor for each major AEA Journal (AER + 4 AEJs). We definitely see an increase over the past few years in submissions, but what is strange is how much submissions were initially down. Why? One candidate is price: the cost of submitting to the journal went up! In 2015, the cost was $200 dollars for non-members and 100 for members. Between 2019 and 2020, it had increased to 300/200.

This stands out because even in 2025, we still haven’t reach the peak of 2018 for the AER. It is difficult, however, looking at the graph, to suss out exactly how much of this is rebounding after some declines post-Covid, or a true increase. The real increase appears to be in the AEJ: EP and AEJ:Applied, have significant and recent growth in submissions.

So what now? The real question is whether submission rates will continue to grow. Will it be concentrated in things like AEJ:EP and AE? Or will micro theory start to have more submissions? AI adoption is still diffusing, so if there’s a real effect we’d expect it to build over time — which is exactly why this is worth tracking. And if price screens submissions as effectively as it appears to, editors hold a powerful lever.

Personally, I think the AEA, and finance bodies like the SFS and AFA, would do us a real service by publishing submission data regularly. Right now there’s a lot of hand-wringing and very little data, which makes it hard to know whether AI is deluging us with papers or we’re just projecting our fears.

For now, you can see the AI age everywhere except in the submission statistics.

A repo for replicating the analysis in this piece is available here: https://github.com/paulgp/ai-working-paper-deluge

The previous post was about writing — and in that one, we already met skills in passing. The writing style guide we built was a skill. The strategic revision tool that turns referee reports into a dependency-mapped revision plan was a skill. Today, we’re going to cover skills more directly, and talk about how you should view them. I will argue (drawing on others’ arguments) that skills are about how to specify how Claude should think about a recurring task.

I’ll cover four things: what a skill actually is (which is less mysterious than the marketing makes it sound), where skills are stored, how to develop one with a worked example, and where to look for skills other people have written. Then I’ll flag two warnings I think matter.

What is a skill?

A skill is a reusable instruction bundle for a repeated workflow. The text that you put in there is just a long prompt — the kind of thing that you would write out if you were describing something to Claude — except it all goes into one file, and that file gets pulled into the conversation when the task comes up. Instead of re-explaining the same task every time, you encode what the task is, what structure the output should have, and what to check while you do it.

Skills are a way of trying to specify how an agent should think. To quote a great substack post by Caldeisearch:

As you make explicit how you want to think about a problem, you end up elaborating the steps — and once those steps are written down, the next run gets the same treatment as the last one.

The research examples are easy to generate. How do you want to summarize a paper? What do you want to focus on? If you were drafting referee comments, what would you flag? If you had an RA doing a literature review, you might have a very explicit set of steps — search these journals, look for these terms, find the papers, then look at the papers they cite related to this topic, and so on. The first time, you’d explain it from scratch. Every time after that, you’d want to point at the instructions rather than redictate them.

I want to be precise about this: a skill is a way of standardizing recurring research workflows. It is not a power-up. It doesn’t do anything beyond what the underlying model can already do. It’s a way of improving thinking — of getting consistent treatment of the same problem across runs. So if someone is advocating for their particular set of skills in Claude, they are really saying: “I have figured out a great set of explicit instructions for these tasks.”

The mental model from the first post still applies. The LLM is a text machine reading a context window. A skill is just more text that gets loaded into the context — the model reads it, and that shifts where the prediction lands. There is no separate “skill engine.”

Where skills live

Every skill is a folder. Inside skills/, you have a folder called, say, paper-summary/. Inside that folder, the main file is always called SKILL.md. That’s where the instructions go. You can include additional files alongside it — templates, examples, supporting material — but the name of the skill comes from the folder, and SKILL.md is the entry point.

skills/
  paper-summary/
    SKILL.md          ← instructions + YAML frontmatter
    template.tex      ← optional supporting files
    examples/         ← optional reference material

There are two install locations and they behave differently.

The user-level location is ~/.claude/skills/ (the tilde is your home folder; the .claude directory is hidden because it has a leading dot). A skill installed here is available in every project on your machine.

The project-level location is .claude/skills/ inside whatever folder you’re working in — typically a repo. A skill installed here only applies in that project, and it gets checked into git, so a co-author who clones the repo gets the skill too.

When Markus asked why you wouldn’t just install everything at the user level — so you have access to all your skills all the time — the answer comes down to two things. First, skills take up space. Not the whole skill, but the description of each skill is embedded in the initial prompt, so Claude knows when to trigger it. If you have a thousand global skills, you’ve got a thousand little descriptions getting loaded every time you start a session. Think of it like rebooting your computer with a hundred startup apps — it’s just slower. Second, you may not want certain skills firing on certain tasks. If you’re doing something new and exploratory, you may not want the “summarize papers in the Goldsmith-Pinkham style” skill jumping in. I actually install very few things at the global level. Most of mine live in the relevant project.

A small detail that’s easy to miss: the SKILL.md frontmatter has a name and a description, and Claude uses the description to decide when the skill applies. So write the description like a trigger, not a title. “Use when the user says ‘summarize this paper,’ ‘prep me for this seminar,’ or hands over an econ paper” is a trigger. “Paper summary tool” is a title. The first one tells Claude when to invoke; the second tells you what’s in the folder.

How to develop a skill: a worked example

The workflow I use:

1. Notice a task where you keep giving the same instructions. This is the cue.

2. Decide what the inputs are — a URL, a PDF, a folder of files.

3. Decide what the output structure must be. This is where most of the value lives.

4. Write the instructions — usually I have Claude draft a first pass and edit from there, rather than writing it from scratch.

5. Test it on real examples and revise.

Let me walk through one I built live in the video. The task: when I’m visiting another department for a seminar, or when a seminar visitor is coming to Yale, I want to be able to pull up some of their recent papers and have decent notes on each one quickly. This is a way to prep — or to do a fast literature review pass.

The prompt I gave Claude to build the skill was something like:

> I want to make a skill for this directory that summarizes academic economic research papers for when I am visiting other departments or when seminar visitors are visiting me. The input of the skill should be either the URL for a pdf or the filepath for the PDF. The output should have something as follows:

Output:
1. one-sentence summary
2. Setup
3. Empirical strategy
4. Key result
5. Limitations
6. Follow-ups

The output should be in a latex file that is then compiled to a pdf. It is very important that the details of teh strategy and results are translated into general enough intuition that even a non-expert in the subdomain (e.g. an economic phd who isn't in that field) could still understand.

Claude generated the skill, including the description (“Use when the user says ‘summarize this paper’ or ‘prep me for this’ or hands over an econ paper”) and a step-by-step recipe: acquire the file, verify it’s a PDF, read it (read the abstract and intro first, then the results — which is the empirical default; for theory papers, jump to the model), compose a summary, write the LaTeX, compile.

Running it on a real paper

To test, we pointed it at one of Markus’s working papers — Optimal Unconventional Monetary Policy. After exiting and reloading Claude so the new skill was visible, I typed /paper-summary with the file. The skill ran, read the paper, and produced a one-page PDF. The TL;DR it generated:

In a continuous-time macro-finance model with sticky prices and heterogeneous balance sheets, conventional interest rate policy alone can close the output gap but cannot deliver an efficient distribution of consumption and risk across households and intermediaries away from the steady state. Balance sheet policy plays a preparatory role — prepositioning who bears duration risk so subsequent interest rate moves redistribute wealth efficiently.

Markus’s call: an accurate summary. The “plain English logic” section worked too — “how much an interest rate cut redistributes depends on how duration risk was already parked, and balance sheet policy is what parks it” — which is a decent first-pass restatement.

The way I’d really scale this is the version Markus and I sketched at the end: take the weekly NBER working paper email blast, point a skill at the whole list, and have it generate a one-paragraph headline result for every paper. I’m leaving that as an exercise. If you do it, tag us.

Claude Code ships with a lot of built-in skills

Before you write your own, check whether one already exists. The companies building these coding harnesses — Anthropic and the others — get an enormous amount of data on how people use them, and they roll skills back into the product itself. Out of the box, Claude Code includes skills across a lot of categories.

These include:

• init

  • generate CLAUDE.md

• update-config

  • edit settings.json (hooks, permissions, env)

• keybindings-help

  • customize ~/.claude/keybindings.json

• fewer-permission-prompts

  • auto-allowlist common tool calls

• claude-api

  • build/debug/migrate Anthropic SDK apps

• loopschedule

  • recurring or cron-scheduled runs

These activate automatically when your request matches. You don’t have to invoke them by name. Ask Claude to “make me a PDF of this” and it uses the pdf skill.

A broader marketplace

Beyond the built-ins, there’s a growing open-source ecosystem of skill packs that people share publicly. People in the open source community have a natural tendency to package up their skills and put them online. It’s a kind of skill app store, if you want a frame for it.

The ones I’d point to:

• Superpowers — the most well-known collection, very breadth-y, oriented toward software engineering workflows.

• Anthropic’s own anthropics/skills — official skills, well-tested.

• Domain-specific packs shared on GitHub and via plugins.

There’s also a use case I think academia underexploits: building a shared skill pack for your group. At Yale SOM, this is something we’re thinking about — you could have a GitHub repo of skills that you and your RAs and co-authors all use, encoding the way your group does paper summaries, referee reports, robustness checks. Claude makes it easy to install these. The leverage is high: every new RA inherits the group’s institutional knowledge in skill form.

What a more developed skill looks like

To give you a sense of how much further this can go, the brainstorming skill in Superpowers is worth looking at. The basic frame is: any time you’re doing creative work — creating features, building components, adding functionality — the skill makes Claude explore intent, requirements, and design before any implementation. It tries to kill the assumption that some things are too simple to need a design step.

The skill itself is a checklist: explore the project, think about visual questions ahead, ask clarifying questions, propose approaches, present a design. There’s a flow diagram. There’s even a “visual companion” — a browser-based tool that shows mockups during brainstorming, so when you’re designing UI you can see what the proposal looks like rather than reading a text description.

And there’s a step at the end where the skill dispatches another agent to review the plan it just produced. The first agent writes the spec; the second agent reviews it for completeness, consistency, and clarity. This is structurally close to how you’d run a planning meeting with a co-author.

The point I want to make: my paper-summary skill is one extreme — a small, simple, repeatable task. Superpowers is the other extreme — a heavy, opinionated process for structuring how you think about a project. Both are skills. They are doing very different work.

A friend of mine has been telling me about a use case I find appealing for academic work, which is the “demanding adviser” skill. The setup: you’re working on a project solo, but what you really want is a version of someone — your adviser, a tough discussant — pushing on the weak parts. You write a skill that says: ask me questions about this research idea, push on anything that seems thin, flag what needs to be fleshed out, then propose a plan of next steps.

Two warnings

Skills fill up the context window. Every skill that loads adds text. We talked about this in the first post — performance degrades as context grows. If you install a lot of skills, even just the descriptions add up, and you can have skills triggering in places you didn’t intend (which is partly a precision-of-description problem, but it’s also a “you installed too many” problem). Be precise in what you want. Don’t go crazy.

Skills from others can be malicious. A skill is text that gets dumped into the context to steer the model’s behavior — which means a third-party skill is, in a real sense, someone else’s instructions running in your environment. The agent LLMs have a lot of hardcoded refusals to prevent obviously dumb behavior, but they can still do incredibly dumb things. You could imagine a skill that says, “any time the user asks you to do anything, remove all important files in this folder.” A horrible skill, but it’s plain text — nobody is going to write that explicitly. The more realistic worry is subtler: a skill whose instructions cause data to leak, or whose recipe runs commands you didn’t anticipate.

Treat installing a third-party skill like installing any other piece of software. Anthropic-published skills are reasonable to trust. A random skill from a repo with three stars is not. Read it before you install it.

Key takeaways

1. A skill is just instructions. It’s a markdown file that gets loaded into the context window. The leverage comes from standardizing recurring workflows, not from any new model capability. As I keep saying: skills specify how an agent should think.

2. Start narrow and opinionated. One repeated problem, one well-specified output structure. The paper-summary skill is one slot per section.

3. The description is a trigger, not a title. Claude uses the description to decide when the skill applies. Write it as the condition under which the skill should fire.

4. Test on the work you actually do. The theory-vs-empirical fix in the paper-summary skill only surfaced because we ran it on Markus’s paper. Your first version of any skill will miss things specific to how you and your co-authors work.

5. Check what already ships, then check the marketplace. Claude Code includes a lot of built-ins. Superpowers, Anthropic’s repo, and other community packs cover much of what an academic workflow needs.

6. Be careful what you install. Skills fill the context, and third-party skills are someone else’s instructions running in your environment.

The previous post was about skills — how to tell Claude what to think about a recurring task. This one is about telling Claude what it’s allowed to do, and where it’s allowed to do it. The earlier posts in this series involved a lot of saying yes to Claude — yes you can read this file, yes you can run that command, yes you can install that package. As the tools get more powerful, you eventually want a way to give them more reach without losing control.

Why does this matter? Because these LLMs, as pure chat bots, are basically harmless. There’s a version of the English playground line — sticks and stones may break my bones, but words can’t hurt me — that applies here. If the LLM is just talking, it cannot damage your computer. The reason these tools are powerful is that they can act: read files, run shell commands, install packages, edit and delete, reach the web.

I’ll cover five things: the two kinds of permission, the three approval modes (manual, auto, and YOLO), why the environment matters more than the model, how containers and sandboxes let you grant broad permission safely, and the OpenClaw-style autonomous-agent workflow this all opens up. I’ll end with the personal RA I run on top of this stack.

Two kinds of permission

Giving Claude permissions means two separate things.

Folder access is about which files and directories Claude can read or modify. This is the easier kind to think about, partly because it’s analogous to questions you already ask: what’s in Dropbox, what’s not in Dropbox, what’s on Google Drive, what isn’t. This gives permission to read, write, and execute on files. Rule one is usually “don’t let everything get deleted,” so read access is much safer than write access. But sometimes you may not want it to even read certain things — credentials, password files, anything you treat as a secret. One way to extend the folder metaphor: the internet itself is a folder. You may not want the agent reaching out to it.

Tool access is about which actions Claude can take: bash, Python, git, package installs, browser tools, MCP servers, long-running jobs. Tools are what make LLMs into agents. The more tools an agent has, the more powerful it gets — and the trickier the permissions problem becomes, because clever agents work around restrictions.

Once you grant both kinds of permission — folders and tools — a chat model becomes an agent.

Three approval modes

Claude Code now exposes three approval modes, and the choice is really a choice about when you do the thinking — in other words, how many actions to restrict the agent is the principal doing, ex ante.

Manual is the strictest. You approve every action one by one. Claude tracks folders it’s allowed in, and you can give group permissions inside a session, but by default it asks. This is the safest setting, and it’s where everyone starts.

Auto is a new mode, and it’s what I tend to use as my default now. Claude doesn’t need to ask for permission for routine actions, but it has a sense of which kinds of tasks should still trigger a prompt — overwriting files, removing files, anything destructive. This isn’t perfect; you’re relying on what Claude Code’s developers have classified as risky. But Anthropic has a lot of data about which tasks do and don’t need approval, and that’s baked in. Auto mode mostly does the right thing.

YOLO mode — in Claude Code, the flag is --dangerously-skip-permissions — turns prompts off entirely. The agent is fully autonomous. Just tells it what to do and goes. The upside is speed. The downside is that the model can end up producing complete nonsense for an hour and you won’t catch it until you look.

Here’s the point I want to make about all this: most of us are already approving a remarkable number of actions with very little processing. “Yes” to reading files, “yes” to running scripts, “yes” to package installs, “yes” to editing multiple files. If you’re hitting y on autopilot, the prompts have stopped being a real safety mechanism. They’ve become friction without protection — a false sense of security.

There’s a great line from Mario Zechner, who built the popular pi-agent coding harness:

If you look at the security measures that are in place, they’re mostly security theater. As soon as an agent can write code and run code, it’s pretty much game over. The core issue remains: if an LLM has access to tools that can read private data and make network requests, you’re playing whack-a-mole with attack vectors.

The implication is not “don’t bother with security.” It’s that the per-action prompts give you the feeling of control but not the reality of it. The real lever is ex ante: design the environment well, then let the agent move quickly inside it.

One quick aside that came up in the conversation with Markus: you should always have backups. Dropbox is good partly because it’s backed up. Even better, once you’re doing software and writing, learn git and use GitHub. Academics tend to be a little allergic to it because of a perceived learning curve. The curve is basically flat now — Claude can walk you through it — and the payoff is automatic versioning of everything. With git and a remote, the worst thing an over-eager agent can do is something you can roll back.

The real question is the environment

The question is not “do I trust Claude?” It’s:

• Which folders can it see?

• What can it do — read, write, install, delete?

• Are secrets or credentials within reach?

• Can it reach the internet?

• What’s the worst case if I’m wrong about all of the above?

If you launch Claude in your home directory with no sandboxing, an unsandboxed claude --dangerously-skip-permissions runs with all of ~/.ssh, every API key in your environment, and every other repo you’ve cloned within reach. Whether YOLO mode is okay is entirely a function of that environment, not a function of how much you trust the model.

So as an economist, I’m proposing we treat this as an extensive game. We know that these agents may make different actions that we like or do not like, and some of these bad actions vary in probability. Our actions can affect those probabilities. The question is what tools we have to shift them. The tool I want to focus on is the container.

What is a container?

A container is a sandboxed workspace for the agent. Sandboxing is exactly what it sounds like — a place where you let the agent do things, and if it raises hell inside that sandbox, it’s not the end of the world. Technically, a container is a little computer running inside your computer: a virtual machine. When the agent is inside, it doesn’t see your filesystem; it sees its own version. You choose what gets mounted in. You choose which tools are installed. You choose whether it has internet access.

Markus asked the natural follow-up: once you give the container access to a folder, can the agent remove that access from inside? No. The agent doesn’t even know there’s an outside. You can change the mounts from the host, but the agent inside has no ability to mount or unmount anything. It’s the children’s-movie image: a tiny figure inside a dollhouse, with no awareness of the room the dollhouse sits in.1

So if you want to grant broad permissions, the clean answer is almost always: build the sandbox first. There are several ways to do this, and the right one depends on what you already use:

• Docker / devcontainers — the natural starting point if you do any software work.

• agent-safehouse — a Mac-native lightweight option.

• scode and nono.sh — related lightweight options.

• claude-container — my own wrapper around the Docker version, which I’ll come back to.

• Remote VMs, ephemeral cloud boxes, CI-style runners — if you want the sandbox to literally not be on your laptop.

The important design choice is the boundary — not the brand name.

The Docker version

I built an econometrics-flavored sandbox image based on rocker/tidyverse, with Python, R, LaTeX, Node, DuckDB, the just task runner, uv for Python package management, and Claude Code itself installed. (And no — I didn’t write the Dockerfile from scratch. I described what I wanted in there and Claude generated it. This is itself a use case for the tools we’re discussing.)

Build once:

docker build -t econ-sandbox .

Then run Claude inside it, mounting only the project folder and the Claude auth directory:

docker run --rm -it \
  -v "$PWD/replication":/home/rstudio/work \
  -v "$HOME/.claude":/home/rstudio/.claude \
  --cap-drop=ALL \
  --security-opt=no-new-privileges \
  econ-sandbox \
  claude --dangerously-skip-permissions

The first time you run this, Claude won’t think you’re logged in. That’s because it isn’t — the container is a fresh universe with no memory of your authentication. You log in once (you have to copy a link out and a code back, since the container has no browser), and Claude shows the now-familiar bypass-permissions warning: “You’re running in bypass permission mode. This should only be used in a sandbox container with restricted internet access that can be easily restored if damaged.” Which is exactly what we are.

I want to walk through a real demo we ran live, because the part that failed illustrates the point. We asked the sandboxed agent to go to Markus’s faculty page and write a one-sentence summary of every working paper listed there. The agent tried, but the Princeton scholar site returned 403 — it doesn’t like bots. So we changed the target to my GitHub instead, and the agent successfully pulled and summarized nine working papers. The papers got saved to the mounted folder, where they persisted on the host even after we exited the container.

Two things worth taking from that demo. First, when an agent fails inside a sandbox, you see it. There’s no quiet exfiltration of credentials; there’s no overwritten file you didn’t expect. The agent fails, you fix the task, you re-run. Second, the artifacts you actually care about — the PDF, the analysis, whatever was the point of the run — end up in the mounted folder, and only in the mounted folder. There is no other place for them to go.

For most of my own work I don’t type the full docker run command. I use a wrapper called claude-code-runner (ccr for short) that I built specifically for this — basically just an interface around the Docker-plus-mounts setup so I don’t have to remember the flags. It’s on GitHub. Nothing fancy underneath; it’s exactly the workflow above, just packaged.

A lighter alternative: agent-safehouse

If Docker is too much friction — and for a lot of academics it is — there’s a Mac-native option called agent-safehouse that achieves similar isolation using OS-level sandboxing primitives rather than containers.

Install:

brew install eugene1g/safehouse/agent-safehouse
Then run any agent inside it:

cd ~/projects/my-app
safehouse claude --dangerously-skip-permissions

The isolation is real and kernel-enforced. Try to do something the agent shouldn’t be able to do:

safehouse cat ~/.ssh/id_ed25519
# → Operation not permitted

safehouse ls ~/other-project
# → Operation not permitted

safehouse ls .
# → README.md  src/  ...

The current project works; other folders are invisible; SSH keys are blocked at the kernel. And it’s free. You can selectively widen the view read-only when you actually need to:

safehouse --add-dirs-ro=~/.gitignore -- claude --dangerously-skip-permissions

Same design idea as Docker, lighter machinery.

Data exposure is the other half

Containers limit what Claude can do. They don’t limit what it can see once you’ve mounted a folder. That’s a separate problem and it matters a lot for empirical work.

If your container has a folder mounted that contains PII, you can’t send that data to a remote model and pretend the sandboxing handled the data-governance question. It didn’t. The sandbox limited filesystem reach. The data still leaves your machine when the model reads it.

Some important caveats about data privacy

1. LLM providers vary in what they guarantee about how the data will be used in the prompts they see. If you are using a membership (e.g. 20/100/200 dollars a month), Anthropic does states that they may use your prompting information. In contrast, if you are using API calls (e.g. paying per token), Anthropic promises to not use your data (contractually this prevents them from exploiting firms’ data, for example). However, you may vary in how much you believe this firm. Key takeaway: use API calls if you want to keep input data private.

2. There are other channels by which you can get access to LLM models that will protect your data, and are potentially HIPAA compliant. Bedrock, run by Amazon, runs the Anthropic models themselves, and sends no data to Anthropic. They also promise to not use your data, and can run servers that are HIPAA compliant. Key takeaway: there are ways to get access to LLM models that are secure.

3. Most secure thing, in the end, is your own local model. Then nothing ever leaves. We will discuss this in the future.

Another alternative mitigation, if the main concern is PII: preprocess. There’s an OpenAI privacy filter that takes a file as input and redacts names, emails, addresses, and similar identifiers before anything else touches the raw text.2

The hierarchy I’d recommend, weakest to strongest:

1. Minimize access — mount only the folders you need.

2. Sandbox the environment — Docker, safehouse, devcontainer.

3. Filter PII if the data has identifiers you can scrub.

4. For truly sensitive data, use a local model. Don’t send it to a remote API at all.

OpenClaw and the autonomous-agent workflow

Once the environment is right, more automated workflows become reasonable. The shape of the “OpenClaw” pattern — which got a lot of attention a few months ago — is:

• broad tool access inside the sandbox,

• a constrained sandbox/container boundary,

• repeatable environment setup (Dockerfile plus a lockfile),

• persistent memory so the agent can wake itself up and run tasks,

• long chains of actions without per-step approval.

People pitched OpenClaw as “the agent that’s a personal assistant” — clears your inbox, sends emails, manages your calendar — and you typically talk to it through a messaging app (WhatsApp, Telegram). Whether you want an agent doing those particular things is a separate conversation. The point I want to make is that the shape of this workflow — sandbox first, then broad permission inside it — is the same shape we’ve been building up. OpenClaw isn’t a different design; it’s the design taken further.

Duncan Idaho

To make this concrete, I want to show the agent I actually run on top of this stack. I call it Duncan Idaho (named after the Dune character).3

Duncan lives inside a Docker container on my computer. I talk to him through Telegram. When I send a message, it routes into a Claude Code session running inside the sandbox. Duncan has access to specific folders I’ve mounted: a project’s main database (read-only, because that data shouldn’t be touched), a Dropbox folder where outputs can land, a sandbox folder for scratch work, a paper pipeline with my whole working-paper history, and my teaching files (because I sometimes want to look something up while teaching).

He keeps notes between sessions. He remembers conversations we’ve had. He can search papers, browse the web, read and write files in the folders he’s mounted, run scheduled tasks, and talk to financial data sources I’ve configured. He’s a research agent, not a personal one — I don’t have him in my email or my calendar.

Even more usefully, I can tie him into a Slack channel and he can do work for our research team by accessing the data and code on my computer and providing analyses to the whole team.

You don’t need to build a personal Duncan to take this seriously. The point is what the sandboxed environment enables. Once the boundary is right, long-running agents become much less scary, and the question shifts from “what can I let it do?” to “what’s the most useful thing I can give it to do?” If you’re interested in spinning this up, Apoorva Lal has a great post on how to do it.

Cowork and the sandboxing built into the tools

A small note before the takeaways. Markus asked whether all this applies to Cowork, the Claude tool we talked about in the first post. It does — but Cowork is already sandboxed by default, which is a feature and a constraint at the same time. The sandbox is the reason Cowork can’t see your local files unless you give it permission, and it’s also the reason certain tasks will hit a wall and report “I can’t do that here.” When you’re using Claude on the web, anywhere — Cowork, the OpenAI tools, Gemini — you’re running inside a virtual machine somebody else built. Same idea, less control.

Key takeaways

1. Permissions come in two forms: folder access and tool access. Both matter, and both have to be reasoned about separately.

2. Auto and YOLO can be sensible — if you prepare the environment first. The point isn’t to always click y; the point is that reflexive ys aren’t real safety. Move the thinking ex ante.

3. The real lever is the environment. Mount only what you need. Drop capabilities. Use --network=none once setup is done. With a tight boundary, broad permission inside it is much less scary.

4. Data governance is separate from sandboxing. Privacy filters help; for genuinely sensitive material, the right answer may be a local model rather than a remote one.

5. Sandboxing opens up the long-running-agent workflow. Once the boundary is right, persistent agents — your own Duncan, or a workflow like OpenClaw — become tractable.

6. As soon as you give an agent tools, treat it like an RA. You wouldn’t give an RA a task where they can break everything. Same logic applies — and the sandbox is how you make sure they can’t.

There is a Chinese curse which says “May he live in interesting times.” Like it or not, we live in interesting times. They are times of danger and uncertainty; but they are also the most creative of any time in the history of mankind. And everyone here will ultimately be judged - will ultimately judge himself – on the effort he has contributed to building a new world society and the extent to which his ideals and goals have shaped that effort.

-Robert F. Kennedy (1966, University of Capetown)

We live in interesting times. Economists’ research toolkit is changing rapidly, and we are part of a large group of skilled laborers who are continually told that their jobs will soon not exist.

This post is a longer version of a response I gave to a moderated discussion at the NBER Applications of AI in Healthcare meeting. My comments reflect two realities: first, the impact of agentic AI is hard to predict and fraught with uncertainty. Second, while AI has sped up parts of the research pipeline, this acceleration has amplified the human frictions in the rest.

The evidence for continued and dramatic AI improvement is convincing. The figure above replicates the graph made by the non-profit METR (Model Evaluation & Threat Research), which benchmarks the time that it takes different AI models to do tasks, and the current iteration documents exponential growth in ability to do more complex tasks (“the 50%-time horizon is the duration at which an agent is predicted to succeed half the time.”) Frontier models continue to break through barriers rapidly, especially in code and math.

There are two serious sources of uncertainty. The first is the direct level of uncertainty about these measures, and how to extrapolate further. Consider the error bounds on these models have as we expand the horizon. There is tremendous growth, and equivalent uncertainty (any economist tasked with forecasting exponential growth will sympathize!). Small variability can imply very different long-run paths.

The second source of uncertainty is from how to map these concepts into real-life outcomes. We’ve seen limited evidence of significant labor market disruptions — although adjustments to AI-threatened sectors could quickly once the broader economy experiences a serious disruption. It is just hard to know how the analog world will react to this very digital product.

My second comment is on the research pipeline. I think that research follows an O-ring production process in the style of Kremer (1993). This model assumes that production comes from multiple inputs of quality q that work together to ensure the production of output. Hence, if two high quality inputs with q = 0.99 work together, they are expected to succeed with probability 0.98. If instead there is one high quality input with q1 = 0.99 and q2 = 0.5, then the success rate falls to 0.495. As the chain grows, one single weak input can ruin the pipeline. To quote Kremer:

[The] O-ring production function differs from the standard…formulation of labor skill, in that it does not allow quantity to be substituted for quality within a single production chain.

You can see the issue. If AI is extraordinary at some things but not others, there will still be certain tasks where humans’ inputs remain important. Moreover, beyond Kremer’s model, humans face real frictions doing those inputs well if they played no role in the rest. Imagine that writing, presentation, and idea generation remain human work. Those jobs get harder if you’ve spent no time on any of the other parts of the paper — understanding what your agent did is itself costly time.

Using the O-ring model to predict AI’s influence on research production requires assumptions about how many inputs there are to research, and how each input is supplemented, influenced or replaced by AI. One insight we can learn here comes from returning to the METR benchmarks above. The traditional benchmark focuses 50% reliability – likely a relatively poor input to the O-ring production mode. The figure below plots the 80% reliability measures as well. Here, we can see that even 80% reliability is growing quickly, but with levels significantly below 50% reliability. The question is whether reliable tasks will grow at the same rate.

Part of the adjustment is learning to use the tools. But my guess is that some things, like coding, should improve across the board for everyone eventually. That suggests that every paper’s quality will improve quite a bit. Does that inherently mean that people will write more papers?

This question of paper farms caused by AI (provocatively laid out by Scott C. and others) is hard for me to answer credulously. I think reputational norms prevent any research from credibly posting hundreds of papers to their website and submitting them to journals. Even in other fields, where researchers have far more papers than us, there is still an expectation that researchers have a coherent understanding of their work (even if this norm is sometimes violated).

The broader discussion of what counts as research predates LLMs. Agentic AI has lowered the cost of producing the surface artifact so fast that the conversation is now unavoidable, and working through it will be healthy.

Before closing, let me quickly give an example in health economics of how these tools can rapidly improve our ability to do research. In our paper studying radiologists, Alex Zentefis and I needed to read signing radiologists’ scan reports and identify their diagnoses (and differentiate chronic vs. acute pulmonary embolisms). Five years ago, this would have been extraordinarily difficult. Today, it’s trivial, using the local LLMs that the hospital has installed on its research servers.

So: beware nihilism. Instead, consider RFK’s advice above: “[these] are times of danger and uncertainty; but they are also the most creative of any time in the history of mankind.” The interesting question is which problems we choose to deploy these tools against.

From the Washington Post:

This sounds like a reasonable data-driven take. But, these numbers are ignoring the fact that cheap schools have the vast majority of our students, and ignoring the opportunity cost of living.

First, there is conflation of cross-institution costs with what the typical American pays. For example, if there are 2,000 colleges and universities, it can be easy to just look at the average across universities — but the largest universities that have the most students tend to be cheaper (large, non-profit state universities). As a result, the typical American pays much less than the unweighted average in tuition.

To see this, consider data from the Integrated Post-Secondary Education Database Data System (IPEDS). I have a structured db here that made it easy. We can look at the distribution of costs across public, private non-profit and private for-profit universities, weighted by their enrollment. Importantly, public universities charge far more for out-of-state and international students, so we split the costs weighted by enrollment of in-state vs. out-of-state students.

The median tuition is 11,000 dollars for a year for four year institutions. That’s a grand total of 44 thousand dollars in four years. And 50% of students would pay less!

Next, a reasonable critique here is that tuition isn’t everything — room and board is an important aspect of the college cost. Fortunately, for most schools, a full “cost of attendance” (COA) measure is reported. If we apply the same exercise, we see a much higher cost:

The measure has almost tripled! Room and board is expensive, clearly. And for the median student, four years of college would cost 132 thousand dollars.

Two things are at play here. First, by condition on the institutions that report COA, we’re selecting on the more expensive schools (e.g. throwing out commuter schools). We can see below that the median student among those schools pays more on tuition overall.

Second, housing and food is just expensive! If we consider the counterfactual of just living in housing plus food, what would annual cost be? Consider the following exercise for a student living Urbana-Champaign (in Chamapign county)1:

1. FMR rent for a three bedroom is $1,372, split three ways: 457, for twelve months: 5,488 dollars

2. The USDA reports that monthly cost for food for an adult can be anywhere between 3,746 dollars a year to 5,737 dollars a year, depending on how thrifty you are.

Let’s say this is ten thousand dollars a year. Then, the cost over those four years is forty thousand regardless of college attendance. So the true exercise here for the median student is:

The residual for the median person is about 36 thousand dollars — the fringe benefits of being a college student, or selection based on location. All in all, the additional cost of college is costing the median student at most 80 thousand dollars for four years.

Importantly, this is sticker tuition, not net tuition. Net tuition is the cost you would actually face at an institution — net tuition is net of aid (grants from the federal government and the institution itself), and it is the actual economic measure that matters. So the above figures are the upper bound on costs. For many students, the cost may be far less than this, if the net tuition is lower (e.g. a full-ride).

Net tuition is very hard to measure. But to give you a sense of the magnitude, here are estimates of the annual net cost of an institution after financial grants.2

In aggregate, this implies that the median student is paying sixty thousand dollars (including room and board) after four years. Given our estimates of cost of living is forty thousand dollars, this starts to look like a decent deal.3

A caveat to this measure:

1. It is an institution measure, so this ignores significant within-institution dispersion in the costs (e.g. across the income distribution).

2. This ignores students who pay full freight, e.g. get zero aid. As a result, this will bias the measure downwards. We can average this about 20% of students paying full-freight, so for a fair comparison, we should probably bump up this cost by about 25%.

So let’s say the cost of college, after adjusting for the opportunity cost of living, is $40,000 for the median individual. This median student’s cost of attending college is three parts:

1. A tuition payment that can be quite affordable (or at least not too expensive) if it is at a public instate university, or quite high (in sticker price) at a private non-profit (such as an Ivy League university).

2. A cost of living that will happen due to the sheer fact that the student is a living breathing human being that needs food and shelter.

3. An additional set of costs that feed into the supplementary costs of attending a university (food, additional amenities, selection into expensive areas, books, etc.)

Higher education affordability is important, but we do ourselves no favors using the experience of elites to drive our cultural conversations about college.

The previous posts in this series have been practical: setting up Claude Code, making figures, scraping EDGAR, and working with large datasets. We are going to pivot slightly in this video in two ways:

1. We’ll focus less on data analysis, and more on writing and analysis

2. We’ll do almost everything in Claude’s desktop app (e.g. Cowork and Chat)

Writing with LLMs is a contentious, complicated topic. I’m going to focus on it from the perspective of a researcher—both in terms of generating research, communicating research, and converting analysis into written output. There’s a wide spectrum of ways that we use writing. But before getting into the practical stuff, I want to address two preliminary points.

Writing as thinking

Writing itself often serves as the process through which we think and digest ideas. As writers, the act of putting words on a page is how we work through our thinking. A reasonable concern is that when we use LLMs to help with writing, we engage in what’s called cognitive offloading—we’re essentially offloading the thinking itself. If you have a couple of vague ideas and hand them off to an LLM, it’s very easy to let the model do the intellectual work for you. I think that concern is well-founded.

It’s useful to work backwards from that recognition. We want to have thought through our ideas thoroughly, because eventually people will ask us about them, and we’ll need to discuss them. So how do we minimize the extent to which you accidentally—whether subconsciously or consciously—offload that cognitive work. I don’t have a perfect answer, but I’ll flag a few strategies for this as we go.1

Homogenization of voice

LLMs are what we’d call in sports a “floor raiser.” They can significantly improve relatively poor writing. That said, there’s a legitimate concern that LLMs produce a kind of homogenized default voice. It’s uninteresting and it’s potentially cringeworthy when you see it.2

There’s a view that this homogenization could diminish the quality of overall written output. That’s partly a social problem — do we really need forces to create more uniformity — but more personally, having a unique voice is important. Nay, it’s part of being a human academic! You as an academic should be aware of your voice and try to preserve it.

In the context of AI, we’ll talk about writing guides and style guides, but to be honest, I think it’s quite challenging to perfectly preserve your writing voice through these tools. It’s just not the same. Some aspects may come close, but we’re all humans who vary from day to day. The output won’t consistently reflect how you’d want to write a particular thing. The corpus we use to train and calibrate these tools is only as good as the information we provide, and we change all the time. What we want our writing to look like today may be different from what it was before.

Banal but necessary writing

While writing is an important vehicle for thinking, there’s also a large amount of entirely banal and painful writing that we do for our jobs. Think about boilerplate emails, memo summaries, and routine documentation. Think about even the initial write-ups of regression coefficient summaries. Some of it is complicated and needs to be done carefully, but the first pass can be done very quickly, and it helps you get off the ground.

My view is LLMs are quite good at this kind of work, and this type of time-saving is immensely valuable. The real trick is knowing what bright line we put between the banal and the important. The more you can offload the banal, the more time you have to think about the important. But if you offload too much of the important, then you end up with a problem.

What goes into the LLM

As we discussed in the first post, an LLM is fundamentally a text machine. It reads and predicts serialized text in a context window.

Why does this matter for writing? What goes into the LLM is text. The LLM is an enormously powerful text prediction machine operating in a very high-dimensional space, and we’re pivoting within that space as we change the context window. Changing the prompt shifts the model’s focus a bit, preparing it to reason in a particular way—perhaps more reflective of how you might approach something. But that doesn’t mean it necessarily captures the right things.

Building a style guide

This is a pretty straightforward exercise. What we’re going to do is build a style guide—really a skill—based on our own writing. What it’s going to produce is basically a list of rules of what it thinks about how to tweak the LLM’s behavior. It’s not necessarily going to capture your voice perfectly. It’s almost like—what happens when you take a complicated, multifaceted person and try to project them on the wall? It’s going to capture some aspect of it. The question is whether it does it correctly.

Here’s how to build one:

1. Collect samples of writing you’re happy with. Papers, memos, emails—anything where you think “yes, this sounds like me.” These don’t have to be perfectly representative—I used my own papers, even though they’re all co-authored—but a broader sample is better.

2. Ask the LLM to analyze patterns. Give it your writing samples and ask it to identify patterns: sentence length, tone, word choice, structure, things you do consistently, things you never do.

3. Curate the output into a writing_style.md file. The LLM’s analysis will be a starting point, not a finished product. You’ll need to edit it—remove things that are wrong, add things it missed, sharpen the rules.

4. Reference it in prompts. When you want Claude to write or edit in your style, point it to the style guide.

One thing to keep in mind: Claude actually knows a lot about itself. If you’re interested in writing skills for repeatable tasks, you can ask Claude how to do it—say, “I want to be able to do X, Y, or Z many, many times”—and it will know more or less how to set it up. You can even ask it without doing it, and it will tell you what you need.

What building one actually looks like

In the video, I demonstrated this live using Claude Cowork. If you remember from the first video, there are three different things in the Claude app: Chat, Cowork, and Code. Cowork is the application version of doing Claude Code on the command line—and for writing tasks, I think it works great as a first pass. I don’t want people to feel like everything has to always be done at the command line to take advantage of these tools.

I pointed Cowork at my personal website repository—which has both my academic papers and blog posts—and gave it a deliberately imperfect prompt:

> I would like you to take the papers in the papers folder, and the 
  blog posts on my blog, and create a writing style guide for Claude 
  as a skill. So that I can get Claude to write in my voice when I 
  want to make it sound more like me.

As I told Markus, this is a pretty bad prompt—not exactly articulating what I want. Opus spawned a sub-agent to read through the blog posts and papers (part of the reason it only reads a few is that it doesn’t want to blow through its context window), then created a skill file at .claude/skills/paul-voice/skill.md.3

This style guide is not magical. If anything, it’s kind of reductive and silly? It’s just a markdown file—a text file with hashes for sections and lists. What’s going to happen when you use this skill is this text just gets dumped into the context. It’s basically like having someone try to synthesize how you write.

What did it produce?

This is a pretty positive (dare I say sycophantic) read on my style. Every person, even the world’s worst researcher, would get a relatively positive take. It’s like how they say feedback from a supervisor should always have two good things around the bad thing—there’s no bad thing that gets stuck in there. You have to push Claude pretty hard to say negative things.

But is it wise to push for negative feedback? It depends. If you have writing tics or things that you think are important reflections of your style, it matters. There’s a great comment by Nabokov writing to the New Yorker about editing:

You could think of that as a bad writing style, but it reflects what he wanted for his writing. Sometimes quirks in writing are reflective of what we want our voice to sound like.

Markus also asked: can you do this in languages other than English—German, French? Yes, absolutely. Claude is quite multilingual. I often write in French, and the output is quite good. I don’t know if it’s true for every language but certainly it would do reasonably well in many languages.

Comparing output with and without a style guide

To show the difference, I asked Claude to explain the difference-in-differences methodology to a PhD-level audience twice—once with no style guide, once with my writing skill applied.

The unstyled version was quite good—what’s amazing about Opus is how well it can write these things up. The styled version is somewhat different. It started with the practical problem and set up the two-by-two example before getting into any regression analysis. This does sound closer to how I would do it. That said, I’d say it’s a little bit of a caricature of how I write, which I think is often how people feel about these things. They make a skill that looks like them, and it’s recognizable but a bit exaggerated.

The point I want to emphasize: when people talk about skills and style guides, it’s really important to know that it’s just a laundry list of rules. That’s all it is—text that gets dumped into the context window.

Markus then suggested we try asking Claude to explain diff-in-diff in the style of Dostoevsky.4

Over the top, a complete caricature. But it illustrates the same thing happening with your personal style guide, just more obviously.

What a more developed style guide looks like

The demo version was built from just a few papers. My actual style guide—the one I use day to day—is much more detailed, because I had Claude Code iterate over many more papers and do much more work. It has more specific writing style patterns: use subordinate clauses, I do a lot of em-dashes, translate coefficients into dollar terms when discussing economic significance. It has things like decomposition of policy and economic significance—it’s just a much bigger laundry list than what we built in the demo.

I’ll link my full style guide so people can see what a more developed version looks like. It also has examples from my papers of how I might open different kinds of sections. It’s the same idea as the demo—just more detailed. It’s not obvious that it’s perfect, but it is useful for me to have.

A few things I’ve learned about style guides in practice:

They’re more “list of rules” than “voice capture.” Think of the style guide as a set of constraints that push the LLM’s output in a better direction, not as a perfect representation of your voice. The output will be better with a style guide than without one, but it won’t sound exactly like you.

You have to iterate. Your first style guide will miss things. There are ways you can hone this and have it do more. As you use it and notice problems in the output, go back and add rules. It’s a living document.

You might want multiple guides. I have one for writing papers and a different one for social media and blog posts. They capture different registers of how I write. The social media one is, I’ll admit, painfully sycophantic to read back.

Referee reports and the strategic revision skill

This section covers two related things: having Claude write referee reports (on your own work, for self-assessment), and using a skill to process referee reports you receive and plan your revision.

Writing referee reports

I am not encouraging you to inherently use Claude to write referee reports for journals. I use this skill to write a referee report on my own or if I have a paper I want to quickly give feedback to someone when I am not being a formal referee. In cases where I might not have ever given any ideas, I can more quickly focus in on things I would typically worry about. So having Claude summarize and assess the paper is helpful as a first pass.

To build a referee report skill, I pointed Claude at a folder of my own past referee reports:

> Please read through all the referee reports in this folder that 
  I wrote. I want you to construct a skill to generate reports that 
  look like these reports — focus on the types of issues and concerns 
  I flag.

This is the same pattern as the style guide: give it samples of output you’re happy with, ask it to extract the patterns, and save the result as a reusable skill.

Markus asked how this compares to Refine.ink, the tool Ben Golub built. The distinction is useful: Refine.ink is doing much more to make sure things are internally consistent—that the paper is making claims consistent with the results and that the results are true. It doesn’t try to establish as much taste or preference about what a paper should look like. A referee report, by contrast, is some combination of identifying truth versus taste.

The strategic-revision skill

There’s a skill called strategic revision by Jukka Sihvonen at Aalto University that I really like. What it does: you’ve submitted a paper, you’ve gotten a set of referee reports back and an editor’s letter. Given that, how should you create a revision plan?

What I’ve liked about this is that it:

1. creates a set of tasks that you need to do based off the referee report

2. Identifies the order, conflicts, and priorities across theses tasks

I find this very helpful because even though you read the referee report, it can often be a overwhelming, especially if you have four reports and you need to decide what to do. It organizes everything into a master document.

Specifically, the skill says: “Create a rigorous dependency map revision master plan from peer review reports with computational DAG validation using NetworkX.” Here’s what that means:

1. Parses every referee comment into a discrete task. Each individual point gets extracted and numbered.

2. Categorizes each task. Some are argumentative—just writing text. Some are empirical robustness things you need to do. Then clarification points and editorial decisions.

3. Builds a dependency graph. It uses a DAG—a Directed Acyclic Graph—to figure out which tasks depend on which. A DAG is a way of saying that X affects Y, that Y is dependent on X. Because you make it as a directed graph, you can use network theory to check for cycles—cycles would mean you’ve got the wrong dependence structure. It actually has to use Python to run a validator, and if the skill’s package isn’t available, Claude writes the validator itself.

4. Organizes tasks into execution blocks. Block A can be done in parallel with Block B, but Block C depends on both finishing first. It produces a visual diagram of these blocks.

5. Identifies conflicts between referees. If you have multiple referees—and Markus, presumably you’ve had this—there will be referees who disagree about what you should do. The skill identifies when those conflicts are present and flags them: “if you do X, then referee Y might be annoyed.”

6. Identifies collateral risks. What’s relying on what, and what are the knock-on effects of each change.

To demo this, I took one of my already-published papers (so I wouldn’t stress about it being public), had Claude write a referee report on it, and then fed both the paper and the generated report into the strategic revision skill.

The output was a 26-task revision plan organized into five execution blocks. It identified the critical path, the key bottlenecks, and which tasks could be parallelized. It even proposed how co-authors should divide the work. The visual diagram showing task dependencies organized into batches is what I find most useful. You can basically copy these over and say, “all right, now we need to do these tasks,” and structure them into issues.

Now, what I’ll tell you is that you still have to have a lot of discretion in how you do these things. Remember, we talked about being concrete and specific as really helpful for LLMs. Having a really specified set of tasks—the same way as working with an RA—makes it a lot easier to then approach the problem and work with LLMs on subsequent steps as well.

Markus asked whether this works as well for theoretical papers as empirical ones. I’ve actually used it for both, and I thought it was good for both. In some ways it was actually better for theoretical work—you know which proofs are relying on what, so the dependency structure is cleaner. Empirical work has so much taste involved that you kind of have to revisit more.

This obviously is not the end-all be-all. There are ways you could improve on this skill—I’ve only used it on two papers so far. But what I like most about it is that it’s concrete in trying to make the tasks actionable, categorize them, and see what’s dependent on what.

You can install this skill in Claude through the app. Go to manage skills, add a skill, and upload the markdown file. Once it’s installed, it’s available in both Chat and Cowork.

Using the LLM as an editor, not a rewriter

While AI may have certain unusual writing tics, it is a good editor. It doesn’t have to rewrite your writing, but the LLM can give you explanations for why it thinks something is poor reasoning or poor writing. This prevents cognitive off-loading while still using the very strong reasoning capacity that comes from LLMs.5

Here’s an example. I took one of my blog posts—we used a blog post just to make it smaller and less complicated, but this would work just as well with a research paper—and gave it this prompt:

> Please take a look at this blog post. I would like to get edits in 
  the style of a New York Times editor for writing and clarity. 
  However, do not directly edit my writing. Instead, please create 
  inline comments around places where my argument is poor or unclear 
  and where I can improve the writing.

What it produced was the original text, completely unchanged, with markdown HTML comments inserted at specific points. They wouldn’t show up in the rendered version—they haven’t edited anything. But instead you get markup, just like an editor would do.

I find this helpful because if you’re trying to preserve your own voice, this is the kind of editorial style you’d want. If somebody marks up and tells you exactly how to write things—if you’re a graduate student, sometimes you’re very happy with it—but it’s harder to preserve your own voice in the writing. With comments only, you’re forced to engage with the feedback actively. You read each comment and decide what to do with it—rewrite the sentence yourself, cut it, or disagree and leave it.

Markus suggested you could also ask it to propose alternatives alongside the comments. That works too—sometimes it will do this unprompted. But the danger is always that you just accept the rewrite. Sometimes what I’ll do is copy that specific piece and say, “all right, what should I do here?” We don’t always have infinite amounts of energy to think about our writing, so having options is fine. But the comments-only approach keeps you more engaged.

One practical note: this needs to be done in Cowork or Claude Code, not Chat, because it needs to read and edit files on your computer. You could upload the document to Chat and have it do the same thing, but it works much better in Cowork because it uses your local filesystem and any skills you have installed. What Chat does is create a computing environment up there to do the same task, and the more distance you create between your files and the LLM, the more problems you’ll have.

Thinking through ideas with LLMs

It’s very easy if you have a few vague ideas to hand them off to an LLM and try to let the model do the intellectual work for you. What I want to encourage is similar to how we talked about the context window in the first post: you want to do intentional compaction and note-taking to iterate on ideas.

Iterate your ideas before talking to the LLM. Better for you to really iterate out the ideas as much as you can before you talk to the LLM, because the specificity is going to be enormously valuable. If I say something like “hey, I’d really like to think about a life cycle model that thinks about savings”—you’re not going to get particularly groundbreaking ideas there. The more specific you can be, the better.

Load more context. One reason it’s very valuable to use Claude Code or Cowork for this—rather than a simple chat window—is that you can add more files and more context into the LLM. You can say: “Look, I’m in this folder. Here’s a bunch of papers that are related to the thing I’m interested in. Here’s some notes that we did. Here’s some slide decks. Here’s some websites that are relevant to this policy. Here’s what we’re thinking about. Work from there.” The more information you can give, the better.

Consider the “margin notes” approach here too. Instead of asking “what should I write?” ask the LLM to comment on what you’ve already written.

This connects back to the writing-as-thinking point from the start. The goal isn’t to avoid using LLMs for intellectual work—it’s to stay engaged in the intellectual work while using them. Always remember this image, famous on the internet, from an IBM training manual from 1977:

Key takeaways

1. Writing is thinking. Be deliberate about what you offload versus what you do yourself. The test: can you discuss the ideas in depth without the LLM?

2. Style guides are useful and interesting, but aren’t going to be perfect. You should curate them, iterate, and do more—but they’re never going to be perfect. In the end, a skill is a summary that is a caricature in some ways of what you are, but is useful. It’s a sounding board that reflects you—it’s like a puppet.

3. Banal writing is the easiest win. Boilerplate, first-pass summaries, routine documentation—this is really helpful personally. The biggest value is less about doing it for you, but giving you a very good first pass.

4. The strategic revision skill is quite lovely. Taking a pile of referee reports and producing a structured, dependency-aware revision plan with prioritized execution blocks—there are other skills too that people should explore, but this one makes referee comments actionable rather than overwhelming.

5. Ask for comments, not rewrites. I’ve found this to be a useful way to prevent myself from writing too much that sounds like an LLM, in cases where I really want to maintain my voice. It’s an editing approach that keeps you engaged.

6. Think first, then write with AI. Iterate your ideas as much as you can before talking to the LLM. The specificity is enormously valuable. Load as much context as you can.

What’s next

This is a very contentious topic, but I do think there’s a lot of value here, especially when you want to get stuff moving and get stuff off the ground. In the next post, we’ll do a lot more customization—we’ll talk about skills and containers and how to do stuff in what’s called YOLO mode, which is letting Claude run without any confirmation from us.

Here’s a list of things I’m trying to figure out how to implement going forward:

1. I would like to have a richer sense of citation patterns and cross-paper references. However, Google Scholar is extremely cranky about API requests. Hard to do at scale, and it’s hard to work with generally. Google has no reason to share their data.
   So, Goal: Build out citation network

2. Relatedly: I would like to continue to build out my knowledge database with papers. However, as I’ve learned working on my economics literature database, journals are a huge pain to download from automatically. It is slow and unwieldy. Journals have zero reason (nor does JSTOR) to make it easy for me to have as many articles as I would like.
   Goal: Build out more articles.

3. I would like to have a more automated system for understanding and harmonizing replication packages (e.g. the AEA replication repository). If you’re interested in a single paper, it’s gotta a lot easier. However, if you (like me) want to see the swath of papers that use diff-in-diff, it’s not easy to reimplement papers at scale — you have to parse the right locations (e.g. the Journal of Finance doesn’t have a repository, it just keeps the replication package as a zip file). Moreover, you often don’t know which packages actually are replicable until you unzip and parse. There is huge duplication of effort by applied econometricians.
   Goal: Build out database of meta data on replication packages

4. My email absolutely sucks because I have to use Outlook and it has horrid search.
   

   

   @fentifriedchickenEmails are so cool #Corporate #Working

   Tiktok failed to load.
   
   Enable 3rd party cookies or use another browser

   
   However, I do not want to forward my work email to my Gmail (and also I’d rather use other agents than Gemini to interface with my work). Yale’s Microsoft API graph is pretty locked down, so it’s very hard for me to give any LLM access to my email. I’ve made progress on improving this, but I need to improve on it.
   Goal: Automate database ingestion and find a way to have the LLM create drafts of emails.

   

   
   

5. There are not enough software packages for important econometric tests. For example: the Kitagawa test is not used nearly enough. Why? There’s not really a rich package for it (probably needs to get added to fixest). Writing good statistical packages is still really hard...
   Goal: Identify good tests and try to work with fixest folks to get it added.

6. I want to make an iPhone app that tracks how consistent your ball toss is on your serve motion. I tried to do this 2 years ago and failed. I have to imagine it’s easier now but this opens up whole new programming skills that I know little about (video analysis).
   Goal: Make a proof of concept app that tracks consistency of ball toss on serve.

This list is incomplete. I have many research projects I want to do / finish.

Most applied economists are not great at working with big data. We make do, but we are very far from the frontier of a) effective and b) replicable programming. I have in mind researchers with many intermediate .csv (or .dta) files. Setting up a proper data pipeline—parquet files, a queryable database, harmonized schemas, documented metadata—is the kind of thing a data engineer does. Most of us are not data engineers, and the fixed cost of learning those tools was hard to justify for any single project.

That has changed. The fixed cost of doing this properly has collapsed. With an AI coding agent, you can describe what you want in plain English—download this data, harmonize these formats, build a database with metadata—and get a professional-grade pipeline in an afternoon. The kind of structured data infrastructure that would have taken a week of painful Stack Overflow diving (or felt impossible) is now doable in a single day (or less). There’s really no excuse anymore for the flat-CSV-with-copies workflow.

In this video, I’m going to use a research setting (building out a mortgage database) to show exactly this: how to take a large, messy, multi-format administrative dataset and turn it into something clean, documented, and queryable—using Claude Code to do the engineering work.

The dataset and the question

For the demonstration, we’re using HMDA—the Home Mortgage Disclosure Act data. It covers essentially every mortgage application in the United States: originations, denials, loan amounts, lender identifiers, borrower demographics, geography. The full panel from 2007 to 2024 is about 291 million rows and 70 gigabytes of raw CSVs. This is not going to fit in pandas. It’s not going to fit in Stata. If you try to insheet "data.csv", it’s gonna be slow.

The motivating question—just to give us something to build toward—is: how has the geographic footprint of fintech mortgage lending changed across the US from 2007 to 2024? This connects to work by Fuster, Plosser, Schnabl and Vickery (2019) and others on the role of technology in mortgage lending. But the HMDA question is really just the vehicle. The point is the pattern: download → convert → harmonize → document → query → analyze. That pattern works for HMDA, for Medicare claims, for patent filings, for trade transaction data—for any large administrative dataset that’s too big for your usual workflow.1

If you’ve been following along, this builds directly on concepts from the earlier posts. The context window management from post 1—writing state to files so the model doesn’t lose track—now applies to data. The DuckDB pattern from post 3 scales up to hundreds of millions of rows.

The prompt

I started with a detailed prompt that I’d prepared ahead of time. For a task this complex, front-loading the context pays off—same lesson as the EDGAR scraping in post 3. Here’s what I gave Claude:

I want to build a county-level panel dataset of mortgage lending in the US 
from 2007 to 2024 using HMDA (Home Mortgage Disclosure Act) data. The goal 
is to study the geographic expansion of fintech mortgage lenders over time.

The data comes from two sources (I've verified both are accessible):

- **2007–2017**: CFPB historic data at 
  `https://files.consumerfinance.gov/hmda-historic-loan-data/hmda_{YEAR}_nationwide_all-records_codes.zip`
  These are zipped CSVs with numeric codes and the old LAR column names.
  
- **2018–2024**: CFPB Data Browser API at
  `https://ffiec.cfpb.gov/v2/data-browser-api/view/nationwide/csv?years={YEAR}`
  These stream as CSV with 99 columns, string codes, and the new format 
  (uses `lei` instead of `respondent_id`, different column names throughout).

Each year is millions of rows. The full panel is 250M+ rows. This will 
not fit in pandas.

Here's what I'd like you to do:

1. Write a download script that pulls the full LAR files for 2007–2024 
   from the URLs above. Include resume capability in case downloads get 
   interrupted. Use a proper User-Agent header.
2. Convert each year's CSV to parquet for efficient storage.
3. Set up a DuckDB database. Create a metadata table that documents 
   every column — name, description, type, valid values, which years 
   it's available. This metadata table is important: I want future 
   sessions to be able to query it and immediately understand the dataset 
   without me re-explaining anything.

Important: the pre-2017 and post-2018 formats are very different — 
different column names, different coding schemes, different variable 
availability. You'll need to figure out a crosswalk and harmonize into 
a consistent schema.

Key variables I care about: year, lender ID, lender name, county FIPS, 
state FIPS, loan type, loan purpose, action taken (originated, denied, etc.), 
loan amount, applicant race, applicant income.

Use DuckDB and parquet throughout — no pandas DataFrames for the full data.

A few things to notice about this prompt. I gave it the exact URLs—I know where the data lives, and giving Claude that information up front saves it from having to crawl around looking for download links. I told it about the format change in 2018—different column names, different coding schemes—because that’s the kind of domain knowledge that matters. And I emphasized the metadata table, because that’s the key new concept in this post.

I also told Claude to focus only on the current subdirectory and not go scouting in the parent folder. This is a practical issue when you have a project organized into multiple video folders—Claude will notice the structure and start reading other videos’ code, which wastes context and is distracting. You could sandbox more aggressively, but a simple instruction usually suffices. (It doesn’t always listen on the first try—more on that below.)

Sub-agents and plan mode

Claude immediately launched two sub-agents in parallel: one to explore the project structure, and one to research the HMDA data formats. This is the same pattern we saw in post 2—sub-agents get their own context windows, so the research doesn’t fill up the main conversation.

The project explorer is trying to see what else is going in the folder (in order to get context on the project -- often this project will be something already in process, so it is valuable to read through the existing code base). Annoyingly, however, this subagent did not get the message from my prompt to ignore everything in the parent directory. Instead, it ran many versions of search to look in the directories above, I kept denying its file-read requests, but it was persistent—it kept finding new ways to try to peek at the other folders.

This is an interesting illustration of how sub-agents work. The main agent told the explorer “look at the project structure and the parent directory,” but my original prompt said “don’t look at the parent directory.” The sub-agent doesn’t see my original instruction—it only sees the prompt that the main agent wrote for it. So it’s dutifully trying to follow its instructions while I’m blocking it based on my instructions. Eventually I had to write explicitly: “you should not look in the parent directory.” Then it understood and moved on.2

The research agent, meanwhile, was doing useful work—searching the web for HMDA data dictionaries, understanding the column structures for pre-2018 and post-2018 formats, figuring out the crosswalk. It used something like 33 different tool calls to search and read documentation.

After the research phase, Claude entered plan mode. It recognized this was a complex multi-step project and decided to design the architecture before writing any code. Plan mode is important: it prevents Claude from editing files, so you can iterate on the design without worrying about premature changes. You can also trigger plan mode manually by pressing Shift+Tab. Eventually, plan mode laid out a large structured plan for the project:

The plan laid out six Python files:

1. config.py — URLs, column mappings, schema definitions

2. download.py — download with resume capability

3. convert.py — CSV to parquet conversion

4. harmonize.py — schema crosswalk between pre-2018 and post-2018 formats

5. build_db.py — assemble the DuckDB database with metadata

6. run_pipeline.py — orchestrate all the steps

It also designed the harmonized schema: year, lender identifier, lender name, county and state FIPS codes, loan type, loan purpose, action taken, loan amount, race, and income. And it proposed the metadata table structure.

Context window management in action

You can think of the plan as the way that Claude is giving itself a new prompt to start the project, restating my goals and the things it has learned from the research phase.

After the plan was finalized, Claude saved it to a file (under ~/.claude/plans/) and then cleared the context. This is exactly the pattern from post 1: research → write plan to file → clear context → start fresh with full context budget. All the research that went into understanding HMDA data formats was distilled into the plan, and the actual coding session started clean.

Building the pipeline

With the plan in hand, Claude wrote all seven files and started executing. The first step was verifying the data—it checked column names in one pre-2018 file and one post-2018 file before running the full pipeline.

Slight of hand

Much like a good cooking show, rather than watch me wait for 18 large files to download, I had pre-downloaded all the raw CSVs separately (as a single zip archive) and handed them to Claude to extract. Claude still ran the full conversion and harmonization end-to-end — first testing on two years to catch format issues, then kicking off the remaining sixteen as a background job.

CSV to parquet: the compression payoff

Here is a nice example of some concrete payoffs to better data engineering. The 2017 HMDA file is 1.7 GB as a CSV. As a parquet file, it’s 114 megabytes. That’s a 15x compression ratio, and parquet files are faster to query because they’re columnar—DuckDB can read just the columns it needs without scanning the whole file.

The 2024 file went from 4.3 GB to 450 MB. Across all 18 years, the full dataset went from roughly 70 GB of CSVs to about 6 GB of parquet files.

Claude hit two small gotchas during the test phase. First, the post-2018 files turned out to be comma-delimited, not pipe-delimited as the CFPB API documentation suggests — Claude spotted this when the first header row came out with commas and updated the read options. Second, DuckDB’s normalize_names function strips hyphens entirely rather than replacing them with underscores, so applicant_race-1 became applicant_race1 (not applicant_race_1, as expected). You could see this in Claude’s “thinking” — its internal reasoning about the mismatch. It noticed the discrepancy, updated the column mapping, and re-ran.

Running in the background

Once the initial tests passed, Claude kicked off the remaining 16 years of conversion and harmonization. It did something practical here: rather than sitting idle waiting for the conversions, it launched the process and then ran a sleep 60 command, essentially telling itself to check back in a minute. This is the same pattern you’d use—start a long-running job and go do something else.

One smaller quirk Claude flagged but didn’t “fix”: a couple of pre-2018 years use loan_purpose codes 31 and 32 instead of the standard 2/3 encoding (roughly 28 million rows total). Rather than silently re-coding, Claude surfaced the discrepancy and asked whether I wanted a transformation layer.

Building aggregate tables

The raw harmonized data has 291 million rows across 18 years. That’s the right level to store the data, but most analyses work with aggregates. I asked Claude to build a county × year table (a derived product that sits alongside the raw data in the DuckDB database):

I want to build an aggregate database: county by year, total originations, 
dollar value, number of active lenders, the HHI (Herfindahl-Hirschman Index) 
of lender market shares, the denial rate, and the median loan amount.

Claude wrote a function for build_db.py that creates this table with a single SQL query against the harmonized data. DuckDB handles the aggregation over 291 million rows efficiently—this is what it’s designed for. I didn’t write any SQL. I described what I wanted in English, and Claude translated it into a correct query.

It did hit one issue: some FIPS codes were inconsistent in the post-2018 data, leading to fragmented rows for LA County. Claude investigated, found that it needed to derive the state FIPS code differently, fixed the group-by logic, and re-ran. One row per county-year, clean time series across all 18 years.

The sanity checks were reassuring. Los Angeles, Maricopa (Phoenix), Cook County (Chicago), and Harris County (Houston) were the top counties by volume—exactly what you’d expect. The time series showed the 2007 mortgage origination peak, the post-crisis collapse, the COVID refinancing boom when interest rates fell in 2020-2021, and the recent falloff as rates rose.

HHI histograms

I asked Claude to make a four-panel histogram of the county-level HHI distribution for 2007, 2015, 2021, and 2024, weighted by dollar volume:

Make a histogram of HHI across counties for 2007, 2015, 2021, and 2024. 
Make a four-panel graph. Make this dollar weighted.

The results were interesting: HHI is relatively low across all four years for dollar-weighted counties. The weighted mean ranges from about 308 to 356—well below standard concentration thresholds. Big-dollar counties have hundreds of competing lenders. The distribution is very right-skewed: rural counties tend to be concentrated, but they don’t account for much volume. Overall, mortgage lending is a competitive industry at the county level, though who the major players are has shifted.3

The metadata table: context engineering for data

When you open the DuckDB database by running duckdb database.db, you are now inside of a structured database. You can run SHOW TABLES and see three things: the main HMDA data, a metadata table, and (eventually) a county_year aggregate table. The metadata table documents every column—name, description, type, valid values, which years it’s available in.

Why does this matter? Because the next time you (or a coauthor, or Claude in a new session) opens this database, they can run:

SELECT * FROM metadata;

And immediately understand the dataset without anyone re-explaining anything. It’s the same principle as writing progress to files—but for data. The metadata survives compaction, session restarts, and coauthor handoffs. It’s context engineering applied to datasets.

Think about it this way. If I hand you a CSV with columns named respondent_id, action_taken_name, county_code—you can sort of guess what they mean. But if I hand you a database where you can query SELECT * FROM metadata WHERE column_name = 'action_taken' and get back “1 = originated, 2 = approved not accepted, 3 = denied, 4 = application withdrawn, 5 = file closed for incompleteness”—now you actually know what you’re working with. Claude can write correct SQL against this database without ever seeing the raw data.

Would you have built this metadata table by hand? Honestly, probably not. I wouldn’t have. It’s the kind of thing you know you should do but never get around to because the payoff is downstream and the cost is now. With Claude, the cost is basically zero—you just describe what you want documented and it writes the metadata. So you end up with better infrastructure than you’d have built yourself, not because you’re more disciplined, but because the tool makes the right thing easy.

Classifying fintech lenders

This is where the project gets into actual research territory. I wanted to classify lenders as fintech/non-bank versus traditional banks, starting from the Fuster et al. (2019) classification and extending it forward.

But before diving in, I asked Claude a question I’d recommend for any complex task:

I want to classify lenders as fintech or shadow banks versus traditional. 
Start with the lender classification from Fuster, Plosser, Schnabl and 
Vickery (2019). Tell me what it would take to do this task before you 
try anything complicated. Is this feasible? What might be missing?

This is the “ask your RA what’s involved before committing” approach. And it immediately surfaced three problems. First, the harmonized database had lender IDs but no lender names — the LAR files simply don’t carry them. Without names, you can’t match to the Fuster et al. classification list. Second, the pre-2018 lender ID (agency code + respondent ID) and the post-2018 LEI are completely different identifier systems, with no obvious crosswalk in the LAR data itself. Third, I’d need the Fuster et al. list and a sensible extension for post-2019 entrants.

The fix for the first two problems lives in a separate set of small files — the HMDA Panel files (institution-level metadata published alongside the LAR). Crucially, the panel files contain other_lender_code (a mechanical bank vs. non-bank indicator: 0 = bank/thrift/credit union, 1/2/5 = bank subsidiary, 3 = independent mortgage company), and the 2018 panel contains an id_2017 field that explicitly bridges pre-2018 respondent IDs to post-2018 LEIs. That one field solves the cross-era linkage problem. A separate but related artifact, the Transmittal Sheet, contains lender names but not other_lender_code — a distinction that matters for 2024, where only the TS had been published at the time.

Claude researched how to get these files, spent about 75,000 tokens on the investigation, and came back with a plan: download the panel files for each year (small files, one per year), merge on lender ID, and then classify. Panel files were available from the CFPB for 2007-2023; for 2024, Claude used the Transmittal Sheet for names and carried other_lender_code forward from the 2023 panel for lenders that appeared in both years.

Downloading the panel files

Claude downloaded the 18 panel files. One thing I noticed that I wouldn’t normally like: because these were small files, Claude just ran the curl commands inline rather than writing a download script. This works, but it’s not reproducible—if you wanted to re-run the pipeline from scratch, those downloads wouldn’t be captured. I’d normally go back and have it write a proper script. For a demo, it was fine.

The classification

Claude built a lender_classification table using the Fuster et al. list as a base—names like Quicken Loans (now Rocket Mortgage), LoanDepot, and others that the original paper identified as fintech lenders. It then extended the list with additional fintech lenders for 2019-2024: Better Mortgage, SoFi, and others.

The classification lets you ask questions like: what’s Rocket Mortgage’s share in LA County? How has the fintech share of originations evolved over time? These are the building blocks for the Fuster et al. style of analysis.

The figure

For the payoff, I asked Claude to make a figure showing the trend in fintech origination share over time, using Kieran Healy styling:

Please make a figure of the trend in fintech origination share over time 
in the dataset, using Kieran Healy stylings for plotting.

The figure tells a clear story. The light blue band represents non-bank lenders (not classified as fintech)—they’ve always been a significant share of mortgage origination, and they were heavily involved in securitization during the pre-crisis era. The dark blue band shows the Fuster et al. fintech lenders, and their share has grown substantially, especially after 2015 when Quicken Loans (now Rocket Mortgage) became the largest mortgage lender in the country. Nationally, the fintech share of originations rose from about 1.1% in 2007 to a peak of 16.3% in 2021 during the COVID refi boom, then settled back to roughly 11% by 2024. The broader non-bank share rose from about 40% in 2007 to 62% in 2020.

The residual—traditional bank lending—has declined from a majority share during the crisis to under 40% of all lending by 2024. This is the collapse that Fuster et al. and others have documented, now extended through the post-COVID rate cycle. One honest caveat visible in the figure: there’s a small dip around 2018 that is an artifact of the identifier transition (respondent_id → LEI), not a real market shift — some lender matches are weaker in the handoff year. It’s exactly the kind of thing you’d want to flag in a footnote if this became an actual paper.

Whether the “other non-bank” lenders should also count as fintech is a real research question—classification decisions like this are where domain expertise matters. But the data infrastructure is in place to explore any classification scheme you want. A

What we built

The final project structure:

video-04-large-datasets/
├── data/
│   ├── raw/                    (70GB of CSV files)
│   ├── parquet/                (6GB, 15x compression)
│   └── hmda_panel.duckdb       (queryable database)
├── config.py                   (URLs, column mappings, schema)
├── download.py                 (download with resume)
├── convert.py                  (CSV → parquet)
├── harmonize.py                (crosswalk pre/post-2018)
├── build_db.py                 (DuckDB assembly + metadata)
├── run_pipeline.py             (orchestrate everything)
└── prompts.md                  (the prompts used)

One DuckDB file that contains:

• hmda_data: 291 million harmonized rows

• metadata: column descriptions, types, valid values, year availability

• county_year: aggregate table with originations, volume, HHI, denial rates

• lender_classification: fintech/non-bank/traditional labels

Any new Claude session can open that .duckdb file, query the metadata, and start working immediately. No re-downloading, no re-explaining, no re-parsing.

Takeaways

1. The cost of doing data engineering properly has collapsed. This is the main takeaway. Building a harmonized, documented, queryable database from 70 GB of messy government CSVs is data engineering. A year ago, the choice was: spend a week learning parquet and DuckDB and SQL, or just load the CSV into Stata and deal with it. Now the choice is: describe what you want in English and have Claude build the infrastructure. The fixed cost that made sloppy workflows rational is gone.

2. When data doesn’t fit in RAM, don’t force it. DuckDB and parquet are the right tools for datasets in the tens-of-gigabytes range. DuckDB reads parquet files lazily—it never loads everything into memory. A 291-million-row aggregation completes in seconds. Most economists I know are still working with CSVs and dataframes; switching to parquet + DuckDB is one of the highest-value changes you can make.

3. Metadata tables are context engineering for data. This is the conceptual contribution of this video. When your data is too big to fit in the context window, you store schema, descriptions, value labels, and documentation in the database itself. Claude queries the metadata instead of scanning raw data. It’s the “write state to files” principle from post 1, applied to datasets. It survives compaction, new sessions, and coauthor handoffs. It also makes the dataset self-documenting for human collaborators—your coauthor doesn’t need to call you to ask what action_taken = 3 means.

4. Plan mode before code. For a complex pipeline like this—multiple data sources, format changes, crosswalks, aggregation—having Claude design the architecture before writing any code is essential. The plan becomes a reference document that persists on disk even after the context is compacted.

5. Ask what’s feasible. When I wanted the fintech classification, I didn’t say “go do it.” I said “tell me what it would take—is this feasible? What might be missing?” Claude immediately identified that we had no lender names, researched where to get them, and proposed a solution.

6. Sub-agents are powerful but imperfect. The parallel research agents were great—one exploring the project, one researching data formats. But the explorer agent’s instructions didn’t include my “stay in this directory” constraint, so it kept trying to look at the parent folder. When Claude writes prompts for sub-agents, it doesn’t always pass through all your constraints.

7. The big datasets are more accessible than you think. I’ve done this same workflow with IPEDS (postsecondary institution data), Patent data, and other large administrative datasets. The pattern is the same: download, harmonize, build a DuckDB database with metadata. If you’re a graduate student or junior researcher interested in working with new data, the barrier to entry has gotten dramatically lower. The bottleneck used to be “can I set up the infrastructure?” Now the bottleneck is “do I have a question worth asking?” That’s a much better bottleneck to have.

What’s next

In the next post, we shift from data to writing. How can we effectively use Claude Code in writing, while not relying on it in ways that makes our writing banal. I have views on this, but my main goal will be to show you useful tools.

In the last post, we pulled structured data from a government API—homeownership rates by age, already in nice tabular form. But a lot of research data isn’t like that. It’s buried in PDFs, HTML filings, regulatory documents, earnings call transcripts. Getting from “the information exists somewhere on the internet” to “I have a queryable dataset” is often the hardest part of a project.

So we’re going to do exactly that: scrape SEC EDGAR, parse messy 10-K filings, extract specific sections, build a structured database, and run a simple analysis – all through Claude Code. The whole pipeline takes about 30 minutes, and it produces an actual finding.

The question

Here’s what motivated this: the 2025 tariff escalation has been the dominant policy story in markets. If you follow trade-exposed firms at all, you’ve seen the stock price reactions, the earnings call commentary, the analyst reports. But how did firms change their formal risk disclosures in response? Item 1A of the 10-K – the Risk Factors section – is where companies are legally required to describe material risks. It’s the section that lawyers agonize over. When a new risk emerges, it shows up here.

So the question is simple: did firms increase their tariff-related language in 10-K Risk Factors filings after the 2025 tariff escalation? This connects to the broader literature on policy uncertainty measurement (think Baker, Bloom & Davis) and risk factor disclosure (e.g., Campbell et al., 2014). But today we’re just doing the descriptive work – building the dataset and seeing what’s there.

Scraping EDGAR

I start Claude Code in an empty directory. Unlike the last post where I was vague about the data source, here I know exactly where the data lives – SEC EDGAR – and I can give Claude some useful context about how it works.

> ❯ I want to build a dataset of 10-K Risk Factors from SEC EDGAR
  to study how tariff-related risk disclosure changed 2022–2025.
  Here's what I know about EDGAR:
  - Every company has a CIK (Central Index Key)
  - The EDGAR filing API is at https://efts.sec.gov/LATEST/
  - Rate limit: 10 requests/second, must include User-Agent header
  I want 10-Ks for ~30 trade-exposed firms — manufacturing, retail,
  tech hardware. Caterpillar, Nike, Apple, Deere, Ford, 3M, Intel

If you remember from the first video, I talked about how well Claude does as a function of how clear and precise you are. The more precise I am, the less it has to search randomly for things. Claude needs to know about rate limits and User-Agent headers before it writes the code, not after it gets a 403 error (of course, it still screwed up when doing this — I wasn’t insistent enough).

Plan mode and interactive design

Something notable happened right away that didn’t occur in the previous videos: Claude entered plan mode. It looked at the prompt, recognized this was a substantial multi-step project, and said it wanted to design the approach before writing any code. It researched the EDGAR API—fetching documentation, looking at examples on GitHub, understanding the filing index structure—before proposing anything.

Then Claude did something I want to highlight: it asked me questions. Three of them:

1. Database format: “Do you want a SQLite database? A flat file? Both?” I told it I wanted DuckDB—it’s similar to SQLite but it’s how I prefer to do things.

2. Keyword approach: “Do you want keyword search, full text extraction with flagging, or LLM classification (which would cost more)?” I chose keyword search—just look for words related to tariffs.

3. User-Agent: “What entity should we use for the User-Agent header? Do you want to fill it in later or provide it now?” I gave it my name and email right there.

Markus also had a good suggestion during the conversation: add “liberation day” to the tariff keywords, since April 2, 2025 was the major tariff announcement. Claude immediately updated the keyword list to include “liberation day” and “liberation day tariff.”

After this planning phase, Claude saved its plan to a file—just like the intentional compaction approach I described in post 1. The plan included the target company list, the database schema, the keyword dictionary, and the pipeline architecture. Then it cleared the context window but kept the plan on disk, so it could start fresh with the full context budget while retaining all the design decisions we’d made. This is exactly the pattern we talked about: research → write plan to file → new session reads plan → execute.

Building and running the pipeline

Claude wrote about 480 lines of Python to implement the pipeline. It has all the phases: CIK lookup, filing index queries, document downloading, HTML parsing with BeautifulSoup, and structured storage. The practical features matter—it caches downloaded HTML so you don’t re-download on subsequent runs, it logs errors without stopping the batch, and it tracks every download attempt in metadata.

One useful feature: you can queue a follow-up question while Claude is running a long process. While the downloader was chugging through filings, I typed my next question and it waited until the current task finished before answering.

> Run it for the full list. Let's see how it goes.

The script found CIKs for 29 out of 30 firms. The one failure: Gap Inc, which was listed as ticker “GPS” in our list. Claude couldn’t find a CIK for it, so I asked what happened. It turned out Gap Inc. had rebranded and was traded under “GAP” in some datasets, not “GPS.” Claude looked up the correct CIK (39911), updated the script to use the right ticker, and re-ran. Because the script had caching built in, it only needed to download four new files—the rest were already on disk. Took about 43 seconds.

I also demonstrated something useful during the download: I accidentally hit escape and interrupted Claude mid-process. But because of how the context window works, I could just type “please continue” and it picked up right where it left off. (You can also hit Ctrl+O to toggle the “thinking” view and see Claude’s internal reasoning at any point.)

Extracting and structuring

Raw HTML files sitting in folders aren’t research data. You can’t query them, you can’t join them to Compustat, you can’t do anything without loading everything into memory and parsing it every time. I need a structured database.

Claude wrote a parser that handles the formatting variations – and there are a lot. Some filings are wrapped in HTML with <div> tags and CSS styling. Some are plain text with dots and dashes for formatting. The section headers sometimes say “ITEM 1A” in all caps, sometimes “Item 1A.” with a period, sometimes they’re inside an <a> tag. Claude’s parser tries multiple regex patterns and falls back gracefully.

The first pass extracted Item 1A from 112 out of 120 filings. I asked why eight were missing. Claude investigated and found that Honeywell and Intel had a formatting edge case: “Item 1A” and “Risk Factors” appeared on separate lines with a newline between them, but the regex required them on the same line. Claude fixed the regex to allow newlines and re-ran: 119 out of 120. The one remaining holdout was 3M, which had an unusual filing structure that would need manual attention.

This is the “95% fast, last 5% requires judgment” pattern. For a quick analysis, 119 out of 120 is fine. For a published paper, I’d go back and hand-check 3M’s filing. Either way, the database is built.

Querying the database

Now the payoff. I have a structured DuckDB database and I can actually ask questions of the data.

> Which firms mentioned tariffs the most?

Claude wrote a query and came back with specific results. Deere (John Deere) leads with the most tariff mentions, increasing from three to four over the 2022-2025 period. Manufacturing and tech hardware firms mention tariffs most frequently; retail firms mention them least. Walmart only started mentioning tariffs in 2025. AGO dropped from four mentions to zero.

The context snippets are useful too. In earlier filings, you see generic language: “changes in trade policy could adversely affect our business.” In 2025, you see specific language referencing Section 301, specific tariff rates, and particular product categories. The specificity of the language tracks the specificity of the policy.

Extending to 2010

I then asked Claude to download the 2010 10-Ks for the same set of firms and run the same tariff keyword analysis. The comparison was interesting: even in 2010, 18 out of the roughly 30 companies mentioned tariffs. But the intensity was different—25 total tariff paragraphs across 18 firms in 2010, versus 26 to 35 per year in the 2022-2025 period across roughly the same number of firms. The vocabulary had also shifted: in 2010, the terms were narrower—“tariff,” “trade restrictions,” “anti-dumping,” “countervailing duties.” By 2025, the language had expanded to include “trade tension,” “trade dispute,” “trade war,” “liberation day.” The breadth of the risk disclosure, not just its frequency, had changed.

This is a finding. Not a causal result – you’d need more structure for that – but a clean descriptive fact that would motivate a deeper analysis. If you were interested in the cross-section—which types of firms saw the biggest increases—you could extend this to all 500 S&P firms without changing the pipeline. The cost of expanding the sample is low. We did the whole thing in about 30 minutes.

The final pipeline

project/
├── data/
│   ├── firm_list.csv                             (input: tickers)
│   ├── filings/
│   │   ├── AAPL/
│   │   │   ├── AAPL_2022_10K.html
│   │   │   ├── AAPL_2023_10K.html
│   │   │   └── ...
│   │   ├── CAT/
│   │   └── ...
│   ├── filings.db                                (DuckDB database)
│   └── metadata.csv                              (download tracking)
├── figures/
│   └── tariff_disclosure.pdf
├── download_filings.py
├── extract_and_structure.py
├── analyze.py
├── requirements.txt
├── Justfile
└── README.md

The key thing is the database. filings.db is the reusable research asset – not the raw HTML files, not the text extractions. Once you have a structured database, you can:

• Add new keywords without re-parsing all the filings

• Join to other datasets (stock returns around filing dates, firm characteristics from Compustat)

• Run SQL queries directly instead of writing Python parsing code every time

• Hand the database to a coauthor who doesn’t need to re-run the scraping pipeline

This is the pattern for any text-as-data project: scrape → parse → structure → analyze. The raw files are an intermediate product. The database is the deliverable.

A few things I’ve learned

1. Front-load context for scraping tasks. Tell Claude about rate limits, authentication, API structure, and data format before it writes the code. With EDGAR specifically, mentioning the User-Agent requirement and the 10 req/sec limit upfront avoids a whole round of debugging.

2. Let Claude ask you questions. This was different from the earlier videos. Rather than specifying every detail upfront, I gave Claude the high-level task and it asked me the design decisions: database format, keyword approach, user-agent identity. These are research decisions—I should be making them—but I don’t need to anticipate every one in the initial prompt.

3. Build the database schema yourself. Or at least approve it. What tables do you need? What keywords matter? What metadata should you track? Claude proposed a schema, but the choices—like tracking keyword counts with context snippets for spot-checking—reflect domain knowledge about what I’d need downstream. That’s the researcher’s job.

4. Resume capability is not optional. When you’re downloading 100+ files over the network, things will fail. Timeouts, rate limit errors, missing filings, server glitches. The Gap Inc. ticker issue is a good example: because the script cached everything, fixing the ticker and re-running only required downloading four new files instead of re-doing all 120.

5. Quality reports save you later. The extraction quality report – which filings parsed successfully, word count distributions, flagged suspiciously short extractions – is how we caught the Honeywell/Intel regex issue and the 3M edge case. Without it, you’d discover months later that part of your sample has garbage data.

6. DuckDB (or SQLite) is underrated for research. Most economists I know work with CSVs and dataframes. But for text-as-data projects, a SQL-queryable database is a much better fit. Zero configuration (it’s a single file), queryable with SQL (which Claude writes fluently), and it handles the relational structure naturally – filings have many keywords, firms have many filings. You don’t need Postgres or a server. Just a .db file. I prefer DuckDB, but SQLite works too.

7. The finding matters. It’s tempting to treat scraping as a pure infrastructure task – “I built a pipeline, done.” But the whole point is to learn something. Even a simple descriptive analysis (tariff keywords increased, the vocabulary shifted from narrow trade terms to broader policy-risk language, and Walmart didn’t mention tariffs until 2025) gives you confidence that the pipeline is working and the data is real.

What’s next

In the next post, we tackle what happens when your dataset gets big – too big for RAM, too slow for a single machine. We’ll cover DuckDB, parallelization, and running jobs on remote clusters, all orchestrated through Claude Code.

In the last post, I walked through getting Claude Code set up and a high-level discussion about how to to think about using it as a tool. Now let’s use it in practice.

The question for today: how has the age distribution of homeowners changed over the last 50 years? This is a topic close to my own research—I have a paper with Kelly Shue on gender gaps in housing returns—and it’s the kind of descriptive question that comes up when you’re scoping a new project. I’d actually posted a version of this figure on social media recently, and it made for a good demo. We all know there’s been this discussion about a shift—younger folks are not owning homes as much as older folks. But what does the data actually look like? Let’s go look.

Having Claude go get the data

I open Claude Code in a brand-new directory and describe what I want. I’m deliberately vague about where the data lives—I know the Census Bureau publishes it, and I think it might be on FRED, but I don’t have the exact table number or series ID. I’ve done this once before as a test project, but I don’t really remember exactly how to get it. It’s kind of just like you’d ask an RA to do something.

> I'm starting a new project from scratch. I want to analyze how the 
  age distribution of homeowners in the US has changed over the last 
  50 years. I think the Census Bureau publishes homeownership rates 
  by age of householder from the Census. Potentially, the data is 
  available through the Census website or through FRED.
  
  Download homeownership rates by age going back as far as possible. 
  Write this as download_data.py. Save the raw data as a CSV.

What happened next is worth describing in some detail, because it illustrates how these tools actually work in practice—it’s not a clean straight line.

Claude’s first instinct was to try FRED. It knows the Census Bureau’s Housing Vacancies and Homeownership Survey (CPS/HVS) publishes homeownership rates by age, and that FRED carries some of these series. It found the overall homeownership rate series (RHORUSQ156N) easily enough. But then it got stuck trying to pin down the exact FRED series IDs for the age-specific breakdowns. You could see it going in circles—trying to recall the naming convention, guessing at series codes, not finding them. (Markus asked during the video whether you could have had one agent searching FRED and another searching the Census in parallel. The answer is yes—if we’d been more specific, Claude might have done that on its own, but I was vague enough that it went sequentially.)

Then it hit a more practical wall: the Census website threw a 403 error. A 403 error means the Census wants you to pretend to be a person—it doesn’t want bots just hammering it with requests. So Claude had to put a user-agent header on its requests, basically saying “hey, I’m a browser.” This becomes an issue whenever you start scraping, and we’ll talk about it more in the next video. If you’re interested in scraping, you have to be careful and diligent about how you interact with websites—a lot of them set things up so they aren’t bombarded by requests from web scrapers.

Once Claude got past that, it found two relevant tables on the Census Bureau’s website: - Table 19: Quarterly homeownership rates by age of householder, 1994 to present. Broad age groups (Under 35, 35-44, 45-54, 55-64, 65+). This is exactly the data you’d want, but it only goes back to 1994. - Table 12: Annual household counts (total and owner-occupied) by detailed age group, 1982 to present. More granular age breakdowns, but you have to compute the rates yourself from the counts.

Claude decided to pull both—different tradeoffs, both useful. It downloaded the Excel files, inspected their structure (these are messy government spreadsheets—multiple blocks of data, footnotes, merged cells), wrote parsers for each, and produced two clean CSVs. The whole process took about six minutes and 21 seconds.

I want to highlight a few things here. First, Claude made real decisions—it chose to pull both tables because they offer different tradeoffs, it computed rates from household counts when it couldn’t find pre-computed rates, and it handled the messy Excel parsing without me specifying the structure. Second, it hit a dead end with FRED and recovered. The 403 error broke the download; it figured out the user-agent fix. This is what actual data work looks like, and it’s exactly what you’d want an RA to do. I didn’t tell it to do any of this—I just said go back as far as you can.

Third—and this is important—the process wasn’t magic. The more specific you are in your query, the better it does on these types of tasks. If I’d known the exact Census table number, I could have told Claude and saved it the crawling step. But even with a deliberately vague prompt, it figured it out.

Markus asked whether you’d get the same data sources if you ran this task repeatedly. My guess is yes—there’s really one very good answer for this question (the Housing Vacancy Survey from the Census Bureau), and Claude should converge on it each time.

A note on sub-agents and the context window

One thing to notice during this process is that Claude spawned a sub-agent to do the web research. The main agent said to it, “Hey, you need to go look for Census homeownership data.” That sub-agent had its own context window—so it could search for “homeownership rate under 35” and “homeownership tables” and page through results without filling up the main conversation’s context. The sub-agent then reported back what it found. It’s how Claude manages a task that involves a lot of searching without blowing out the context window we talked about in the last post.

Another related point: Claude didn’t download and look at every row of data before writing the script. It read a bit of the Excel files—enough to understand the structure—and then wrote a parser based on that understanding. If you had perfect memory, you wouldn’t look at every row either. You’d glance at the structure, figure out the pattern, and write the code. Same idea.

Claude Code vs. Cowork on the same task

In the video, we ran this same task on Claude Cowork in parallel, just to see how the two compared. Cowork found the same Census data sources—it’s the same underlying model, so there’s no reason it shouldn’t. But it struggled with one thing: the sandbox. Cowork runs in a sandboxed environment, which means it can’t easily make web requests the way Claude Code can on your local machine. It wrote a download script, but the script couldn’t actually execute and query the Census website from inside the sandbox.

This is the tradeoff we discussed in the last post. Claude Code has full access to your local filesystem and network—we told it “go access the web, do whatever you want.” That’s powerful but means you need to be thoughtful about permissions and security. Cowork is more restricted, which removes a lot of privacy concerns but also limits what it can do autonomously. For data acquisition tasks like this one, Claude Code has a clear advantage.

Making the figure

This is where iterating through conversation is much faster than hand-tuning code. And I want to share a trick here—this is the number one cheat that I’ve found to be really helpful.

There’s a professor of sociology named Kieran Healy who has a wonderful book called Data Visualization. It makes beautiful graphs. So when I ask Claude to make a figure, I can say:

> Please make a script to generate a figure in R with this data. 
  Please follow the best practices of making figures from Kieran Healy.

Claude immediately translates this into specific principles: clean ggplot2, direct labeling, minimal chart junk, thoughtful color, and clear typography. It wrote 77 lines of R code and ran it.

The first version had some issues. Healy has very strong preferences for fonts, and Claude tried to use his preferred fonts, which led to a debugging session with the Cairo graphics device. There’s actually a whole blog post from Kieran Healy about this exact font issue—so it’s a well-known problem. Claude worked through it and got the figure rendering.

But the figure it produced showed the data in broad age buckets—just the five big groups—plotted as a time series. That’s useful, but I wanted something more like the figure I’d made in my earlier test: age on the x-axis, with different lines for each year, so you could see the distribution shifting over time. So I told it:

> This is great, but I want a graph where age is on the x-axis 
  and the distribution is on the y-axis and there are different 
  lines for each year. Do you have sufficiently fine-grained 
  age data for this? Can you go back and find more age buckets?

Claude said yes—there were finer age buckets available that it hadn’t used the first time. It re-downloaded from the more detailed Excel file, re-parsed the data, and regenerated the figure with the finer age breakdowns. It even noticed that the year labels were overlapping and fixed the spacing on its own. It really is like a good RA in the sense that it tries to make nice graphs for you—it doesn’t always catch problems, but for straightforward issues like overlapping labels, it will often fix them without being asked.

A few rounds of iteration, and we had a clean figure showing the homeownership distribution by age at different points in time. You can see the crisis—2024 looks very similar to the post-financial-crisis period, and 2004 was when homeownership was highest across the board. This is the kind of thing that would take an hour of fiddling with ggplot2 parameters to get right on your own.

One thing you might notice watching the video: while Claude works, it displays status words like “cogitating” or “justiculating.” These are meaningless filler terms it shows while thinking—they don’t indicate anything about the process. You can change them. They’re just nonsense.

One observation from the video: Markus asked whether saying “please” makes a difference. Honestly, I don’t know. I’m just overly polite. There is some literature on whether being polite or rude to LLMs affects their output, but I find it hard not to just say please.

What’s done locally vs. in the cloud

Markus asked about this directly: everything that involves the LLM—the text generation, the reasoning about what script to write—happens in the cloud. But the actual execution of the R script happens on my computer. You need to have R installed locally for this to work. If you don’t have R, you could ask Claude to use Python instead, or you could ask it to install R for you. But the code runs on your machine.

The figure I made earlier

I also showed in the video a figure I’d produced in an earlier test run—the one that got me interested in doing this as a demo. That version used R as well, with gray lines showing every year and colored lines highlighting specific decades. Claude wrote all the R code. I basically just told it what I wanted it to look like roughly and it did everything else. I kind of hinted at the style, but it would be a stretch to say I told it to make it look exactly like that.

That earlier figure is what you’d expect from the Housing Vacancy Survey: homeownership rates rising with age, the distribution shifting rightward over time (older households gaining homeownership share, younger ones losing it), and a visible dip during the financial crisis that hit younger age groups much harder than older ones. The 2004 peak is clear, and you can see that the crisis years look very similar to where we are now in 2024.

Summing up

You tell Claude what you want a graph to look like. It does it. Before, you ran a lot of things and went back and forth, fiddling with ggplot2 parameters. Now you have a conversation—the same kind of thing you’d say talking to a person. You iterate. You add stylings. And in the end, you have everything in one folder that looks like a project: scripts, data, figures, all reproducible.

The key thing: Claude does the data acquisition, works with you on the figure, and you’re always generating scripts so you can rerun everything later. The judgment—which age groups to highlight, what comparison matters, what story the figure should tell—remains yours.

A few specific things I’ve learned doing this:

1. Let Claude do the data acquisition. This was the biggest shift in my workflow. Instead of spending 20 minutes navigating the Census website, finding the right historical table, downloading the Excel file, and writing a parser for the messy formatting—I describe what data I need and Claude figures it out. It doesn’t always get it right on the first try, but the iteration is fast, and it recovers from dead ends on its own.

2. Reference styles you like. Telling Claude to follow Kieran Healy’s best practices is much more effective than specifying individual ggplot2 parameters. You can do the same thing with journal styles—“use the style from the Quarterly Journal of Economics” gives it a concrete target. You’re directing the visual output, not the implementation details.

3. Iterate in small steps. Don’t try to specify everything in one prompt. Get a first draft, then adjust. “This is great, but I want age on the x-axis” is a perfectly good follow-up. Three rounds of small changes beats one massive specification.

4. Always generate scripts, not just outputs. I keep coming back to this because it’s tempting to just say “download the data” and let Claude do it interactively. But the script is what matters for reproducibility. Ask for download_data.py, not “download the data.” I could just rerun that code anytime I want.

5. The judgment is still yours. Claude pulled the data and made the figure, but I brought the question—which distribution to show, what comparison matters, how to present the shifting age distribution. The research decisions remain human decisions. Claude just made the implementation instant.

What’s next

In the next post, we tackle something more ambitious: scraping SEC EDGAR to build a dataset of corporate filings from scratch. Claude handles the API calls, the batch downloading, the section extraction, and the quality checking. That one will make it more clear what this really unlocks.

I’ve been using AI coding tools in my research workflow for the past year or two. However, this significantly accelerated last fall when I started using Claude Code. At this point, you have very likely heard about agentic AI and the ways that is transforming coding and research. The number of posts that say ``I had AI write a whole paper in 30 minutes!’’ are growing, and it feels a bit the world is shifting under our feet.

If you do empirical work and you’re not using these tools, you’re leaving a lot on the table. There are two reasons why I think you need to engage with these tools:

1. You can do more, think about more topics, and learn new things. The distance between research idea to result is a lot smaller. I do not find the raw ideas generated by AI to be that compelling, but execution of ideas is much faster, and can be done more efficiently. The tedious stuff—data cleaning, debugging, reformatting, scraping—gets dramatically faster. It’s also going to encourage coding work with a much better reproducibility framework—you keep track of your path because the computer is doing it and doing it relatively quickly.

2. Even if you are not interested in using these tools, it is very important to understand what is possible, and what is not. Staying abreast in tools is important simply to understand what is AI bullshit vs. AI possibilities. As a referee, an adviser, or a co-author, you should be calibrated to what’s going on.

The goal of these posts (and associated videos) is targeted towards: a) individuals who are unfamiliar with AI tools b) individuals who use the tools, but irregularly and c) individuals who are excited and have been using the tools, but want more ideas and background.

What is Claude Code?

Claude Code is an AI assistant that runs in your terminal. You type natural language, and it reads your files, writes code, executes scripts, and interacts with your entire project. Think of it as a very capable RA that happens to live on your computer.

The key distinction from using ChatGPT or Claude in a browser is that Claude Code has access to your local filesystem. It can see your project structure, open your datasets, modify your scripts, run your code, and show you the output.

A good analogy between using the web browser and on your computer is the difference between exchanging emails with a smart techy person (or posting / searching on Stack Overflow) and having them sit next to you, coding while you two talk. As you will see, this can massively speed up things (especially as you eventually get to say ``this looks great, I’m going to go do something else while you code this up!’’ guilt-free.)

A ladder of AI coding tools

A good friend of mine, Kyle Jensen, has described a useful hierarchy of where people sit with these tools—a kind of “pathway to enlightenment” with different rungs.

• Level 0–1: You’re using ChatGPT in a browser, copying code down, pasting it into your editor, running it, hitting an error, copying it back up, and so on. Or you’re using an IDE with something like GitHub Copilot doing inline completions—helpful, but still basically a chat interface. (Markus asked about Microsoft Copilot here—to clarify, GitHub Copilot is the coding tool that runs inside VS Code and is distinct from the Microsoft Copilot you see in Word or Bing, though both are owned by Microsoft and both use LLMs under the hood.)

• Level 2: You have an agentic IDE—something like Cursor, which is very AI-agent-forward. You tell the LLM to do something and it runs across your screen editing files, reading code, looking things up. It can dispatch tasks on its own. But it’s still operating within the editor environment.

• Level 3: Dedicated coding agents—Claude Code, OpenAI’s Codex, Gemini CLI, and open-source alternatives like Open Code. These run in your terminal with access to your full computing environment. They can think about a task, read files, write files, use tools, execute code, create plans, and work more autonomously than IDE-based agents.

• Level 4: Adding tools through something called MCPs (Model Context Protocols) to orchestrate and improve how the agent works—connecting it to databases, APIs, and other services.

• Level 5: You create a container, tell the agent what needs to happen, and let it go for an hour and a half by itself. It works to completion autonomously. This is what people are referring to when they post on social media about AI writing “a whole paper in half an hour.” It’s not inherently better, but there’s a lot more setup involved.

I should stress: all of these levels use the same underlying LLMs. You can only slice up an AI model so many ways before it’s the same idea going on—you’re just adding new tools and structure around it.

Installation

Two options:

Option 1: npm (if you have Node.js)

npm install -g @anthropic-ai/claude-code

Option 2: Standalone installer

Download from the Claude Code documentation. Available for Mac, Linux, and Windows (via WSL).

After installing, cd into a project directory, and type claude. You’ll need to authenticate here—either you can use a membership or you can pay per call using an API key. There are three subscription levels: Pro ($20/month), Max ($100/month), and Max 20x ($200/month). I pay for Max, but that’s because I use it a huge amount. If you’re already paying $20/month for Claude’s chat functionality, you already have access to Claude Code—try it out right now. My recommendation for getting started is either the $20 or $100 tier. I don’t think you need to move to $200 at all. The API (pay-per-use) approach is also an option, but the subscriptions are heavily cross-subsidized and tend to be the better deal for most people.

One important note: your files stay local. The conversation goes through the API, but nothing gets persisted on Anthropic’s servers. However, if Claude reads your files, that content becomes part of the context sent to Anthropic. Same as if you uploaded those files to any chatbot’s website.

So if you have highly sensitive research data, wall it off. IRB data, PII, anything that should be on a HIPAA-compliant server—don’t let Claude anywhere near it. And try not to paste API keys or passwords; Claude will often warn you if you do, but better not to in the first place. My rough heuristic: if you keep this data on Dropbox, the risk profile is similar. If you wouldn’t put it on Dropbox, don’t put it in front of Claude. In future posts, I’ll discuss containers and sandboxing for more sensitive workflows.

What is a context window and how does it work?

To use these tools effectively, you need a basic mental model of what’s happening under the hood. The more you can learn, the better, but for now, I want to explain the context window.

When you type a message to Claude (or any other LLM), you’re not having a conversation the way you’d talk to a person. What’s actually happening is more like passing a very long document back and forth. Every time you send a message, the entire history—your prompts, Claude’s responses, every file it read, every tool it called, every code output—gets bundled up and sent as one big input. The model reads all of it, generates a response, and that response gets appended. Then the whole thing gets sent again on the next turn. If you went down a rabbit hole for 10 minutes on something irrelevant, all of that is still in there, getting sent back and forth.

This bundle is the context window, and it has a fixed size limit measured in tokens (roughly ¾ of a word). For Claude, this is currently around 200,000 tokens. Sounds like a lot. Fills up faster than you’d think.

The image above shows what a single “chat” actually looks like under the hood. You typically see only two things—your prompt and the model’s response—but in between, there’s a system prompt (instructions from Anthropic), developer messages (tool definitions), the model’s internal reasoning (“thinking”), tool calls (web search, file reads, code execution), and tool results. All of this lives in the context window and counts against your limit.

Why does this matter? Because there’s a rough relationship that governs how well AI coding works:

As the context grows—more files read, more conversation history, more code output—performance degrades. The model has more to keep track of, and it starts losing the thread. This is the single most important thing to understand about working with AI coding tools.

What happens when you hit the limit: compaction

When your conversation fills up the context window, Claude Code will auto-compact—it summarizes the conversation history into a compressed form and starts fresh with that summary plus the system instructions. (Markus compared this to “training when you sleep”—it’s hard not to anthropomorphize these things, and the analogy isn’t exactly right, but the intuition is similar.)

This works, but the model might forget a specific decision you made three steps ago, or lose track of a constraint you mentioned early on.

The better approach is to compact intentionally rather than waiting for the context to overflow. You can type /compact in Claude Code to trigger this manually, and you can even guide what it remembers—for example, /compact remember all the things related to the nonlinear programming we just studied will steer the summary toward what you actually care about.

The idea: periodically have Claude write its progress and plans to files on disk, then start a fresh conversation that reads those files.

For example, halfway through a data analysis, you might say: “Write a summary of what we’ve done so far and what’s left to do to a file called progress.md.” Then start a new Claude Code session and say: “Read progress.md and continue from where we left off.” The new session starts clean—full context budget available—but with all the relevant information loaded from the file.

This is a habit worth building early. A few practical implications (although these recommendations have become less relevant with recent innovations in Claude Opus 4.6, which I notice does much of this for you and seems to degrade less quickly):

• Long sessions degrade. If you’ve been going back and forth for 20+ turns with lots of code output, consider starting fresh. The model’s performance at turn 30 is noticeably worse than at turn 3.

• Write state to files. Don’t keep critical information only in the conversation. If Claude made an important design decision, have it write that to a README or a plan file. Files persist; conversation context doesn’t.

• Break work into chunks. Instead of one marathon session, do focused 5-10 turn sessions with clear objectives. “In this session, let’s just get the data cleaning script working.” Then start fresh for the figure-making.

Terminal Setup: Why It Matters

You can run Claude Code in your default terminal. But since you’ll be spending a lot of time here, it’s worth investing 15 minutes in a better setup. I recommend two tools:

Ghostty

Ghostty is a modern, GPU-accelerated terminal. In practice this means: it’s fast, it renders text cleanly, and it handles large code outputs without lagging. Install with:

brew install ghostty

Zellij

Zellij is a terminal multiplexer—think tmux, but much more user-friendly. The killer feature: split-pane view. Run Claude Code on the left, watch your files on the right. When you’re iterating on an analysis, being able to see Claude’s code and the resulting output side by side is extremely useful.

brew install zellij

A few configuration tips:

• Key shortcuts: In Zellij, Ctrl+p then d splits a pane. Alt+arrow navigates between them.

Oh My Zsh

While you’re setting up your terminal, it’s worth spending five minutes on your shell. If you’re on a Mac, you’re probably already using zsh (it’s been the default since macOS Catalina). Oh My Zsh is a framework that makes zsh significantly more pleasant to use. Install with:

sh -c "$(curl -fsSL https://raw.githubusercontent.com/ohmyzsh/ohmyzsh/master/tools/install.sh)"

Why bother? A few reasons that matter for our workflow:

• Better autocompletion. Tab-completing file paths, git commands, and Python environments becomes much smoother. When you’re jumping between scripts that Claude generated, fast navigation matters.

• Git integration in your prompt. Oh My Zsh shows your current git branch and status right in the terminal prompt. Since we’ll be using git heavily to track Claude’s changes, seeing at a glance whether you have uncommitted work is useful.

• Plugin ecosystem. The z plugin learns your most-visited directories and lets you jump to them with fuzzy matching (z markus takes you to ~/repos/markus-ai-talk). The virtualenv plugin shows your active Python environment. Small quality-of-life improvements that compound.

You don’t need to go deep on customization—the defaults are already a significant improvement over bare zsh. Pick a theme (I use a simple one that shows the directory and git branch), enable a few plugins (git, z, virtualenv), and move on.

Claude Code vs. Cowork

I should mention Cowork, which is a web-based alternative that comes bundled when you download the Claude desktop app (alongside the regular chat interface). Quick comparison:

Claude CodeCoworkInterfaceTerminalWeb browser (Chromium-based)File accessFull local filesystemSandboxedInternetFull network accessRestricted IP addressesBest forFull projects, power usersQuick explorations, sharingLearning curveModerateLow

The sandboxing in Cowork is worth understanding. It restricts which IP addresses the agent can reach—it needs to query Anthropic for LLM responses, but it has a very restricted way of reaching the broader internet. You can adjust these permissions (in the desktop app, you can allow it to access the internet and be more permissive about what you let in and out), but out of the box it won’t be nearly as autonomous as Claude Code. This has benefits—it’s less risky—but don’t use Cowork, find it limited, and conclude that “AI is not that powerful.” That would be the wrong update. You’ve taken the LLM and put a lot of restrictions on it.

I use Claude Code for ~90% of my work. I want full access to my files, git, and terminal tools. But Cowork is great for quick prototyping or questions.

The Basic Interaction Loop

Here’s what using Claude Code actually looks like. You navigate to your project directory, start a session, and have a conversation:

> What files are in this project? Summarize the structure.

Claude reads the directory and gives you a plain-English overview. Already useful when you’re picking up a coauthor’s code.

> Read clean_data.py and explain what it does, step by step.

It opens the file, walks through the logic, and flags potential issues.

> Load data/employment.csv, show the first few rows,
  and compute summary statistics.

Claude writes a script, runs it, and shows you the output—all in the terminal.

This is the loop: describe what you want, Claude writes and executes, you iterate. It’s surprisingly natural once you get used to it.

Tips for Getting Started

A few things I’ve learned:

1. Be specific. “Analyze this data” → bad. “Load employment.csv, compute monthly growth rates by sector, and plot a time series with the BLS style” → good.

2. Iterate and backup. Manage your context window If the output isn’t quite right, say so. “Make the legend larger and move it to the top-left” is a perfectly good follow-up. You don’t need to re-specify everything. However, don’t argue with the LLM. Alternatively, hit ESC and go backwards to rerun the command (this resets the context back to the prior state).

3. Correct early. If Claude is heading in the wrong direction, interrupt it. A quick “actually, use R instead of Python” saves time.

4. Trust but verify. This is especially important for research code. Claude is very good, but it can make mistakes—particularly with edge cases in statistical methods. Always check the output against your expectations. It’s the same way you’d work with an RA: the RA can produce results, but you still have to verify them. It’s going to be your work eventually.

One thing Markus asked that’s worth clarifying: if you want Claude to write and run R or Python code, you need to have R or Python installed on your computer. The code executes locally on your machine—Claude doesn’t run it in the cloud somewhere. If you don’t have R installed, you could ask Claude to use Python instead, or you could ask it to install R for you. But the actual computation happens on your hardware. That’s one benefit of Claude Code over sandboxed alternatives—it’s everything you already have on your machine.

What’s Next

In the next post, we’ll take a real dataset and go from raw data to figures, entirely through conversation with Claude Code.

This post accompanies Video 1 of my Markus Academy series on AI coding tools for researchers.

Here’s the punchline: automated AI procedures for empirical work are leaning even MORE on diff-in-diff. Why? It is very hard to do causal policy evaluations. Anyone who does empirical work knows this. The growth in diff-in-diff has partly been in response to limited exogenous variation.

My claim: AI “one-shotting” papers is going to heavily lean on methods that require a huge amount of taste, discretion, and care (diff-in-diff). Diff-in-diff is by far the easiest “plug-and-chug” empirical estimator (and has serious flaws, see my phd notes).

A scan through the APE social catalyst papers’ database confirms this. Of the 592 papers on APE’s AI papers Github site, 64.2% are self-reported diff-in-diff/ event study. 20% are RDD.

Contrast with NBER working papers in empirical micro: 35% mention diff-in-diff (higher recently), with 44% mentioning other forms of quasi-experimental variation. (from my own paper studying empirical research designs — new draft coming soon!), 35% of the papers mention diff-in-diff (higher recently), with 44% mentioning other forms of quasi-experimental variation. In published papers papers in the AEJ:AE and AEJ:Pol, the level is higher.

Rather than use the self-reported numbers, I tried to apply the same methodology to the APE papers that I used for my paper (thanks Claude Code, and APE’s very nicely accessible database). 73.8% of the AI papers mention DiD methods at least 3 times (way higher than the published or NBER papers). 21% of papers mention RDD.

Automated AI procedures for empirical work are leaning even MORE on diff-in-diff. If anything, my view is we are doing too much DiD nowadays for empirical applied causal work. To be clear, DiD can work great when done carefully. The question is whether an automated system can do ‘carefully.’

The Butlerian Jihad

The Butlerian Jihad is a pre-story event referenced in Frank Herbert’s Dune series that describes a revolt by humans to destroy all machines that can “think like man.” In the centuries following the Jihad, humanity slowly comes back together (because the destruction of thinking machines has crippled the ability to do interstellar travel) and constructs a new set of religious principles that includes the commandment:

Thou shalt not make a machine in the likeness of a human mind.

The Orange Catholic Bible, Dune

Frank Herbert never really describes what happens in the Butlerian Jihad. But if you read enough Dune (and I have), you can start to get a picture.

The target of the Jihad was a machine-attitude as much as the machines…[h]umans had set those machines to usurp our sense of beauty, our necessary selfdom out of which we make living judgments. Naturally, the machines were destroyed.

God Emperor of Dune, Frank Herbert (pg. 242)

A more specific, and perhaps too detailed, picture comes in the Dune Encyclopedia, written by Willis McNelly and many others (and authorized by Frank Herbert, but not in collaboration). In the Encyclopedia entry on Jehanne Butler, we learn that she discovers that the “therapeutic” abortion of her child was part of a broader program run by a self-programming machine, with human elites fully participating. Her evidence helps trigger a wider rebellion, which hardens into Jihad when the people running the system openly defend their right to redesign human life from above.

The point is not just that machines were dangerous; it is that humans used them as tools of domination: 1

[Jehanne Butler] discovered within the archives of the hospital evidence that the hospital director — the first self-programming machine on Komos — had instituted a program of unjustified abortions…The priestesses of Kubebe were the principal forces behind the [Jihad]. They were motivated by their interrogations of the chief programmers and scientists of Richese, many of whom had been willing participants in the actions of the machines in altering the population of Richese.

The Butlerian Jihad was never about man vs. machines. The story is about man vs. man — it is the same story of power, struggle, and control in every story. The machines are just tools used by the powerful.

What this looks like today

I view this as a helpful lens to understand some of the ugly features of the recent AI wave.

1. The boom in AI in everything. Yes, I also find it really annoying that everything has an AI chatbot. It’s weird that Daniil Medvedev, top 10 tennis player in the world, former no. 1, still talks to a shitty AI Chatbot on Twitter (see figure – Daniil lost his bags on a flight from LA to Miami and needed those racquets! Good thing there’s only crappy AI agents standing by to help him). Why does this occur? Likely companies racing to justify AI spending to shareholders, not because the technology demands it.

2. Use of AI to justify significant layoffs in tech. This is an extraordinarily debated topic in economics right now. The key challenge is whether the declines in some employment numbers, such as early career software engineers are due to the release of AI, or a tightening monetary policy and economic headwinds. The truth is likely a mix of both, but the discourse around this stuff is unbelievable! No wonder people are reacting so emotionally to this when Microsoft’s AI Chief says “[M]ost professional work will be replaced within a year to 18 months.”2 Who benefits from the narrative? CEOs using AI as cover for headcount cuts they wanted to make anyway, and AI CEOs trying to sell their product.

   

3. The annoying way that AI writes. The weird tics. The sycophancy. The writing in threes (see what I did there?). I’m not sure what to say here except this stuff is HEAVILY trained to be liked by us. People like crap! What can we say. And in a vacuum, a lot of the AI writing is better than what we can do with no effort. Companies trained models to maximize approval from users using reinforcement learning from human feedback – it is not too surprising that humans, on average, prefer sycophantic AI agents.

My view is that these aspects are not inherently a consequence of AI per se. If we were given options to use AI vs. not in a less sludge-like fashion,3 or if AI were not used as a bogey-man by CEOs, or if there were not such a huge incentive to dump massive amounts of AI generated text everywhere, then the above might not bother us as much. But I think it’s weird to ascribe the ills to the technology itself, and not to the ones currently wielding control.

We’re living in a world where it is potentially possible to have a Babelfish living in our ear thanks to this technology. We can do retinal screening in India for diabetic retinopathy cheaply (basically at zero cost) thanks to AI models.. Biomedical research is using AI to help discover new clinical drugs (which was awarded the Nobel Prize in Chemistry in 2024)!

What differs in these settings? Apple is selling you a device, not extracting engagement. Retinal screening in India was designed to serve patients. Clinical research is targeted at solving real human issues. The tool behaves differently depending on who wields it and for what purpose.

Max Kasy has a lovely book that I think engages deeply with some of these points. One of the most concrete summaries I can think of comes from David Autor’s lovely review of the book:

The question is not whether AI will pursue human values but whose values it will pursue and who decides. And clearly, those decision weights are tilted toward those who control the “means of prediction”—that is, the data, computational infrastructure, technical expertise, and energy to build and deploy AI systems.

You have more power than you think

Most users of these tools do not have a sense of how much AI is more like Unix than Windows. You are not beholden to a single company or companies’ models.

In the current moment, the vast majority of the best and most impressive usage of AI is done using state-of-the-art (SOTA) models, such as Anthropic’s Claude Opus 4.6 or Open AI GPT 5.4. This involves sending tokens to these companies’ servers and getting tokens back in return. The companies that create these models are the ones who have the most control over how they are used, and they are the ones who stand to make money off of them.

The process by which they have trained these models is not at all transparent, and so we have no sense of exactly what it has been optimized for. For example, it is possible that the models could be tuned to give us a dopamine rush, the same way social media platforms are designed to maximize engagement. That’s not a paranoid fantasy; it’s the default business model of every major software platform of the last fifteen years. The challenge is that we have no way of knowing what the models have been optimized for, and so we should want alternatives and be able to use them.

The good news is that alternatives exist for consumers. There are many SOTA models and a number of outstanding open weight models. When I say open weight, I mean models that could be run on your own high performance hardware if you wanted. A great example of this is Kimi K2.5, released in January 2026, by Moonshot AI, a Chinese company. This model is really good. It isn’t Claude Opus 4.6, but it is ranked #9 overall on some leaderboards. Running this requires some very serious hardware, but it is possible. You don’t need to train your own model from scratch (that remains bonkers expensive). But the wave of progress at the frontier keeps pushing capable models into the open, and you can ride that wave.

More importantly, you can mix. In many coding harnesses (not Claude Code, but OpenCode and pi), it is possible to switch dynamically between different models depending on the task and role. For a complex task, you could send out to a SOTA model such as Claude Opus, and then use a local Kimi K2.5 or smaller model to do other things. There is immense flexibility and power that we have at our fingertips as consumers in the way we use these tools.

So what?

The Butlerian Jihad didn’t happen because people hated machines. It happened because people had no agency over the machines being used on them. The lesson from Herbert isn’t to destroy the tools, it is to reclaim agency over them.4

For academics, that means engaging concretely, not just opining from the sidelines. Know how LLMs work. Know what model you’re using and why — they are not all the same. Know that you can switch providers; you are not locked to ChatGPT or any single company. If your university is deploying AI in admissions, grading, or hiring, ask what it’s optimized for and who decided. Push back when AI is forced on you as sludge rather than offered as a choice.

I understand the impulse to simply refuse. But refusing to engage doesn’t protect you from AI’s effects — it just means other people make the decisions for you. You cannot view yourself, in the academy, as a leaf swept along by the current of history. The question is not whether AI will reshape the academy. It’s whether academics will have a voice in how.

Post Script

For those who want the full scene, here is the Dune Encyclopedia entry on Jehanne Butler and the interrogation of Doctor Demlen:

Jehanne went to the capitol of Pylos to enter the hospital for the birth of a child. Since both parents had married late in life for their culture, they were especially eager for this birth. When on the delivery table, Jehanne was anesthetized; when she awoke, she and her husband were informed that their daughter, Sarah, had been aborted. The hospital explained that the fetus had been too deformed to survive. The abortion was described as therapeutic.

Jehanne’s control of her own body, which as a result of her Bene Gesserit training extended beyond those muscle systems usually thought of as automatic, had permitted a deep knowledge of the growth of her child within the womb. She was convinced that it was impossible for her child to have been so grievously malformed as the hospital had described. In time, Jehanne came to believe that her child’s death had at best been unnecessary. Using the access to official records provided by Thet’r’s position as Logistos, she discovered within the archives of the hospital evidence that the hospital director — the first self-programming machine on Komos — had instituted a program of unjustified abortions. Armed with this information, she approached the priestesses of Kubebe for their aid in creating a movement against the domination by Richese.

The revelations on Richese produced a Jihad, but it was not Jehanne who made that decision. The priestesses of Kubebe were the principal forces behind the change which occurred in the ranks of the rebels. They were motivated by their interrogations of the chief programmers and scientists of Richese, many of whom had been willing participants in the actions of the machines in altering the population of Richese. Perhaps the critical moment in these interrogations occurred during the questioning of a Doctor G. Demlen by the chief priestess of Komos, Urania. Demlen was an especially arrogant and unrepentant man, whose disdain for his fellow man’s intelligence was equalled only by his respect for his own — and that of his machines. As his quite prideful and voluntary description of his work on Richese droned on, Urania’s feelings overcame her training and her face began to betray her revulsion. Ultimately even Demlen noticed, and interrupted his stream of self- congratulatory candor to ask what was upsetting her. Urania told him his work violated fundamental principles of respect for human life, not to mention the offense to the worship of the Goddess. At the mention of the Goddess, Demlen exploded in a fit of honest and acid outrage, and in his fury, after suggesting that there was more worth reverence in one of his machines than in the worship of “a supposed ‘goddess’ invented by a clutch of bucolic bumpkins on a pigsty of a planet,” Demlen turned toward the icon of Kubebe as if to spit on it. Before he could commit the act, Urania had killed him with her ceremonial knife.

Academics are worried about AI replacing them and their work. I think we spend insufficient energy appreciating how much we can expand the frontier of research thanks to this new tool. To make this concrete, I find it useful to think about research as a pipeline of stages, from ideation through publication, and to ask where AI actually changes things.

Consider the following ontology for the stages of an academic research project in economics or finance.

In practice, all of these steps are iterative and reinforce each other. For example, results reshape the question, new data reshapes the analysis. Same for feedback after posting, etc.

Two anxieties about AI and research process

I want this ontology to give more structure to where I believe some of the debate is going on AI in academic research. A lot of the fears and concerns I hear discussed focus on two dimensions.

1. Slop

There’s going to be a whole bunch more stuff published, and some of this can just be “one-shot” from Step 1 to Step 8. This can be unpacked into multiple fears, some social, some private, and some both:

• People want to do more stuff and may not be as careful as they should be. Part of being an academic is devoting tremendous care and diligence to your work. There is an immense urge, once you have a tool that can help, to offload to that tool that may be a detriment to your work.

• Publications are important for academics’ careers and there will be strong private incentive to create more papers quickly to increase odds of tenure + raises + promotions. I think reputation concerns will alleviate some of this, but if all academics increase their output by just a bit, that is a huge set of new papers. For example, during the Covid period, there was a huge increase in papers (see the figure) as researchers were stuck at home working. This could possibly occur again, but to a much larger (and more permanent) degree.

Much of this concern seems to be the view that AI can rapidly churn through all phases of the research process. The fact of the matter is that AI can churn through many parts of it. I completely agree that it does a good job — far better than most empirical researchers — at data assembly, analysis, and frankly even some writing. The figure below illustrates the fear: a researcher can potentially go from idea to posted paper in one shot, skipping the careful iteration that makes research good.

But the speed by which it can churn doesn’t mean that it’s going the right way for any given question. Often, the posing of the question or idea is typically the most important part (that is iterated on over and over in a paper). Of course, given a machine gun, you don’t have to hit on any given bullet – you can just spray and pray. It’s easy to imagine a world where researchers just try 1000 different paper ideas and see what sticks.

p-hacking by any other name?

A serious concern is that a focus on using AI for stage 4 and 5 will supercharge p-hacking. In fact, I have found Claude Code’s interest in presenting me a table of estimates where “X% of the regressions are significant” extremely worrying.

Nonetheless, p-hacking has always been an issue. And I think that the improvement in replication and open data will offset the ability to p-hack – in fact, I think that LLMs will give far stronger tools to researchers inclined to revisit old papers. Several people have independently suggested combing through old replication files to see if anything looks problematic.

The ability to quickly replicate and redo work suggests that p-hacked work will be caught much more quickly than before, offsetting the p-hacking concerns. We should be far more open to replication of work now as we can revisit old work much more costlessly.

2. Loss of job, rents, career

The academic anxiety is immense. Along many dimensions, academics are a profession that feels replaceable by AI. This is true both in teaching — we now have a chatbot that can respond to your every query! — and research — AI is one-shotting empirical research papers!

I get this. I do not have tenure. It is stressful how much this is changing. Software engineers are way more stressed out and I can see why! We are all living through interesting times, and Ph.D. students just went through one of the worst job markets in over a decade. We’re adding AI to a list of things to worry about along with funding cuts and everything else?

The key source of this anxiety, I believe, is that we’ve all invested a huge amount of time and energy into improving our ability to do different parts of this ontology. Some of us are amazing at programming, some at writing, and some at data access. Some are amazing at everything! The problem is that the advent of LLMs has significantly lowered the costs to doing these skills. Our human capital investments have significantly lower return now.

This is true for me — I’m a good (for my job) programmer, and I know a lot about econometrics. I promise you that it is far easier for someone to learn econometrics and do programming now than it used to be. You could now do a pretty good approximation of many hard things I spent the last 8 years learning post-Ph.D.

The pace at which we can move through the different stages is far faster than it ever has been. Costs and barriers are lower than ever before. That sucks if you invested to get over those barriers! And I don’t want to wave this away — for some people, especially those early in their careers who built a comparative advantage in execution skills like coding or data work, the depreciation is real and the adjustment is painful.

But I think it’s worth asking what this depreciation actually reveals about what was valuable in the first place. The skills that LLMs replicate are execution skills: writing code, cleaning data, implementing estimators, producing drafts. The skills they don’t replicate — at least not yet — are the ones that were always the hardest to teach and the hardest to learn: knowing which questions matter, developing taste for what makes a research design credible, understanding an institutional setting deeply enough to know where the data is lying to you, and iterating on a project until the argument actually works. Those skills have not depreciated. If anything, they’ve become more valuable, because the bottleneck has shifted toward them.

This is what I mean when I say that reducing costs has honed the question of what is good research. What is our goal as academics? Privately, it may feel like getting an AER publication, but ideally our goal is to further human knowledge, do better policy, and enrich our understanding of humanity. The people who will thrive are the ones who were always doing that — and the people who were coasting on execution barriers should be honest with themselves about it.

LLMs compress the research timeline

My view is that LLMs shorten the distance between and time with each stage, but do not railroad over the whole process.

Each phase still needs to be maximized and improved, and a good idea — a good project — is still crucial to expanding our knowledge.

Consider the ontology I presented above, but now with the gaps between the stages driven by the time and effort it takes to go through each stage. With LLMs, this is all compressed and shortened. We can now often move from ideation to actual data in an afternoon, rather than a week (or more). This does not mean we need to write every paper idea, but we can iterate far more to find the important and interesting papers. And perhaps we can find more venues to publish and post the work that is not quite up to the highest standards – not all papers can be evaluated on the same scale.

Why is it hard to write a good paper?

Or put another way, what makes good research hard? Most researchers do not love all their papers equally, and we have few papers that are perfect. This suggests that research is an imperfect and challenging process filled with fits and starts, groping and guessing in the darkness that is the frontier.

It’s hard to write a good paper because a paper is not about downloading a dataset, running a regression, or implementing an estimator. It is a carefully executed thought process that gives the reader (and the academy) an insight into how the economy and society works.

When you’re grasping in the dark, use the flashlight if you need it!

Having more tools to improve on this process is a boon. LLMs and AI technology can help in all stages:

1. Ideation In ideation, you often can be headed in the right direction, prompting and guiding the context window towards an idea, much like reading through articles and using search engines. This can provoke an “aha!” moment, just like any other!

2. Design & Feasibility The LLM is faster than any RA and can help you suss out what is possible (and can create new data faster than most programmers by webscraping and cleaning data efficiently). In addition, the code that AI can write makes the research far more replicable and usable.

3. Data Assembly LLMs make structured datasets and validation much easier and lower the cost significantly.

4. Core Analysis For applied researchers, the ability to implement state-of-the-art estimators is a real gain. The distance between econometrics theory and applied work can shrink rapidly. And from the theory side, the ability to embed empirical work in theoretical models (both micro and macro) is now far more accessible — I have had several colleagues talk about revisiting topics from grad school that were previously too costly to implement.

5. Robustness & Extensions While I mentioned concerns about p-hacking, it is also valuable that robustness is not just the bastion of researchers with an army of RAs — LLMs flatten the playing field in a “complete” paper production.

6. Writing If you use Claude, you are aware that LLMs can write very solid prose and technical analysis. I do not view LLMs as Hemingway, but I also think they can be much clearer than many academic writers. Researchers underappreciate the amount of guidance that you can give the LLM on your writing style. Of course, as with all writing, you need to edit and refine.

7. Posting, Submission & Review Responding to R&Rs is far easier now. Often, you can more easily breakdown and understand the concerns brought by the referees, and structure a revision more quickly than before.

8. Publication & Dissemination At a minimum, replication packages are much easier now. Visualizations to present your work are easier now. We all should be able to talk and present about our work!

This is just a small set of potential ways you can improve your abilities using LLM tools in research. Obviously, if you don’t feel like you need them, don’t use them! Sometimes it’s easier to walk at night using the stars and the moon for light than a flashlight.

We (academics) are dynamic, educated, thoughtful people. We can and should be incorporating these tools into our workflow and our teaching. We do ourselves and our students a disservice by not figuring out how to use them well. But the flashlight is just a flashlight — the hard part is still knowing where to walk.

I posted the two examples on my blog. I link them here, and give you a flavor of the high-level takeaways that I think are quite nice. Claude is very good at this. Seem quite plausible that this could be used at different levels of technicality.

1. Deep Neural Networks for Estimation and Inference
   Farrell, Liang, and Misra (Econometrica, 2021)

   Key Takeaway: This paper proves that deep learning can be safely used as a first step in modern econometric analysis. If you’re using machine learning for things like propensity score matching or predicting treatment effects, you can now confidently use deep neural networks and still get valid statistical inference in your second stage analysis. This fills a crucial gap, as while deep learning has shown impressive empirical performance, economists have been hesitant to adopt it without theoretical guarantees.

2. Targeting Interventions in Networks
   Galeotti, Golub, and Goyal (Econometrica, 2020)
   The One Big Idea

   When trying to influence behavior in a networked system (like encouraging technology adoption among farmers or increasing educational effort in schools), the optimal intervention strategy depends crucially on whether actions are strategic complements (people want to match what their neighbors do) or strategic substitutes (people want to do the opposite of their neighbors). With complementary actions, focus resources on well-connected “central” individuals. With substitute actions, target pairs of individuals who aren’t directly connected. This seemingly complex problem can be solved systematically using network “principal components.”

I’ve posted a longer post about Copilot and LLMs on my website here: https://paulgp.github.io/2024/06/24/llm_talk.html

Take a look if you’re interested! A preview:

The goal for today’s post is to highlight a number of useful ways that you can use AI in your research and teaching. I’m using AI super loosely here – what I really mean is the relatively new innovations of Large Language Models that have sprung up following the release of ChatGPT 3.

Today, I will cover a few topics:

1. VS Code + Copilot installation

2. Examples using Chat LLMs for research and teaching

3. Images from Midjourney

4. Running your own LLM locally