Why does it feel like AI assistants can suddenly do... everything right now?

Circa 2023 with the first launch of ChatGPT, they were simple chatbots that could read messages and nicely respond back. Now they’re sending emails, building entire websites and videogames, coding up apps overnight, hacking public websites left and right -- and, yeah, even serving as my semi-automated video editor. If any of these recent developments around AI capabilities have felt hard to follow or completely unintuitive, you’re definitely not alone.

Lesson 3 of the fully-free Open Augments AI Academy is out today, and it’s the one I think will probably be the most useful to people just trying to make sense of AI today: What Is an AI Agent, Actually? From Chatting to Doing. This is where all of the biggest buzzwords you keep hearing thrown around -- agents, tools, harnesses, context engineering -- finally get demystified in plain English, in 18 minutes flat.

Once you really understand these core building blocks, everything you’ve been seeing in AI in 2026 stops looking like magic (or snake oil!) and starts looking like something you can actually wrap your head around -- and carefully, responsibly start using for yourself. In this one lesson, you’ll learn:

• What a “Tool” actually is, and why this concept is behind basically every interesting thing you’ve seen AI pull off lately (spoiler: they’re apps for AI, and they make them INFINITELY more capable and flexible)

• What an “Agent” really means (spoiler: Any AI assistant with access to Tools)

• Harnesses vs. models (spoiler: You’re never actually talking to a model like Claude Opus 5 or GPT-5.6, you’re always talking to a “Harness” app that translates and facilitates on your behalf... often in very opinionated and unintuitive ways!)

• Context Engineering, or: how you teach your AI to gather its own grounding references and research with Tools to automate the tedious parts of coaching your AI assistants

Plus, as usual: I’ve added a bunch of new hands-on Context Gym exercises for this lesson to tackle (also for free!) that should help you really build the intuition for all of those concepts in more detail.

I’d honestly call this lesson the highest leverage lesson of the course, because these building blocks power nearly every major product you’re seeing in AI right now: Claude for Science, Claude Design, Claude for Teachers, ChatGPT Health, ChatGPT Finances, etc. etc. etc.

If you’ve been following along, I mentioned that this course is really intended to get people caught up at light-speed on AI developments from 2023 to now. Lesson 1 got you to ~2024, Lesson 2 (prompt engineering) got you to early 2025, and this lesson gets you caught up to the top of 2026 in concept. From here, we’ll get more into how to ACTUALLY start using these frontier techniques *safely* and *responsibly* in a principled and iterative way. This is as important as ever, because all these new capabilities mean AI assistants with access to your files, your email, and the whole internet can do VERY real damage when you’re not paying attention. The point of this whole series is to help you take advantage of these capabilities *without* getting burned.

New here? You should be able to get some of the core ideas still, but I recommend starting with Lesson 1 and running through the Context Gym as you go -- this one builds directly on everything before it in many ways.

We are still just getting started, and I genuinely don’t think there’s a better way to get properly caught up on AI out there (that’s why I’m making it: free, forever, for everyone!). I hope you’ll dive in. Links in the comments below!

Lesson 2 of the fully-free Open Augments AI Academy is out today: “How to Make AI Less Bad: Context Is Everything”

First, a big thank you to folks for sharing and digging in: the AI Academy launched just a week ago, and we’re on track break a thousand subscribers on YouTube today or tomorrow. W I L D. These videos are a LOT to put together at the level of production quality and accessibility I’m shooting for, and I’m so grateful it’s resonating with folks (things to remember when I’m frantically editing videos with Claude past midnight!!).

Lesson 1 covered how these large language model AI systems actually work under the hood (i.e., Autocomplete with an extremely fancy hat on). While many see this as a fundamental and fatal flaw (and it is!), this lesson is where we learn how to start making that autocomplete engine work *for* you as its greatest strength, as well: endless flexibility and adaptability. In an 18 minute lesson, you’ll learn:

• The most practically useful mental model in the entire course: thinking of your AI assistant as having two very different kinds of memory (one fuzzy and unreliable “long-term” memory, one sharp and steerable “short-term” memory)

• How to start “loading the dice” on these autocomplete engines to get dramatically more useful and reliable responses back (what people call prompt engineering)

• Why more context is NOT always better: two extremely common failure modes called context rot and auto-compaction that explain an enormous amount of everyday “why did my AI suddenly get dumb?” frustration -- and one simple habit you can start practicing that prevents them outright

If you’re just hearing about the AI Academy: I strongly recommend starting with Lesson 1 and running through its practice exercises in the interactive Context Gym first -- this lesson builds directly on both (with a lot of persistent mental models and visual diagrams you’ll want to get familiar with). And there are eight brand-new hands-on Gym exercises accompanying today’s lesson (auto-compaction in action, memory files, custom system prompts, and more). Free and in your browser.

And we are still just getting started. One way to think about it: My goal is that this series is a crash course to catch people up on these utterly crazy past two years in AI. Lesson 1 catches you up to where the AI world stood in ~2024. Today’s lesson on prompt engineering gets you to the beginning of 2025. Next Wednesday’s lesson -- on tools and agents and context engineering -- catches you up to roughly the end of 2025. From there, we’ll get everything practiced and really digested through the end of Level 1, and Level 2 is where we start getting through the advances of 2026.

I’m trying to make this the smoothest yet *accelerated* on-ramp I can for anyone who wants to genuinely understand this moment, not just skim the headlines. I genuinely don’t think there’s a better way to get caught up (that’s why I’m making this: for free, forever, for everyone!), and I hope you’ll take the time to dive in!

Over the past six months, I've been teaching teams at places like Stanford, Penn, Northwestern, and many more how to start using AI responsibly and effectively in their work. Today, I'm starting to release my entire curriculum: for free, forever, for everyone!

I'm calling it the Open Augments AI Academy. It's built for anyone out there who's seen all the crazy hype and discourse around modern AI and is just looking for a guided, grounded, and sane way to move forward learning how to approach these tools for themselves. As someone who's been using these tools and their predecessors for my research since ~2019, I'm trying to provide everyone the intuition and critical awareness they need to get started at this very confusing and pivotal time (my north-star audience is my mom and dad!).

The first lesson starts with one foundational idea that most people miss when they get started with AI: that modern AI is much less like a hyper-intelligent database or brain, and much more like autocomplete with an extremely fancy hat on. That's its single greatest flaw AND its single greatest strength, at the same time. When you really understand what's happening under the hood and how it works (no math or stats required!) a lot of confusing AI behavior suddenly clicks: why it hallucinates, why it's sometimes confidently wrong, and why it can now do way, WAY more than just write words on a page. From there, we're going to learn about all the crazy buzzwords (context engineering, harness engineering, and Agents, oh my!) and advanced techniques, with much more to come!

No jargon, no experience required, and all taught with the care of a former high school English teacher so that you, your coworkers, your friends, and your mom can follow along. I pair these videos with hands-on demos and interactive activities in the Context Gym: my way of giving you a safe and guided place to practice some of the core principles that should deepen your intuition as we go.

If any of this strikes a chord with you, the full 10min course overview and the first lesson (16min) are live right now on the Open Augments AI Academy page. Start there, and if you find it helpful, please do subscribe here to get email updates on new course videos via Substack below, or on YouTube, and share with friends as I release lessons weekly! It’s a really wild time, and this is my best shot at trying to help others navigate things more capably as the tech shifts and grows rapidly from here.

I've run intensive workshops on best practices for responsible and rigorous AI adoption for well over 500 researchers, colleagues, academics, and non-profit leaders over the past few months. As a former teacher, the Q&A after my workshops are some of the most fun and productive conversations I've had, so I figure: let's make this its own regular thing!

With that in mind, I'm excited to launch a nice little community service, free and open to the public, for as long as it continues to feel useful to folks: Every other Thursday from 2-3pm ET, I'm hosting free AI Office Hours: an open Google Meet session for anyone navigating AI in research, education, teaching, policy, or nonprofit work.

Bring your questions, your half-working pipelines, your skepticism, your worries, and your ethical quandaries. No question too basic, too weird, or too heady/philosophical. I can't promise satisfying answers to *everything*, but I can promise helpful thinking and opportunities to dig in deeper to learn more together. Drop-in, drop-out, attend one, attend all, talk a lot, or just listen in off-camera. Taking all kinds, because there's always more to learn from others on this crazy and wild frontier right now!

A few things I want to emphasize:

• It's genuinely free, and always will be, with no catches or pitches. I am intentionally dedicating 40% of my time to explicit pro bono work via Open Augments, and I love teaching. I want it accessible to anyone who wants to think more critically and carefully about AI given all the crazy hype and misinformation floating about.

• This is for kind of everyone! Most of my experience is supporting researchers, academics, and non-profits, but if you're interested and in another field or role, no harm in stopping by and joining the conversation.

• Every level of experience is welcome. The "embarrassing" beginner questions are always worth discussing in more detail, and I can guarantee you're not alone.

• You can submit questions/topics ahead of time when you register, so I can come prepared with real answers (and real demos, where useful). But totally optional -- registration takes two minutes just to grab contact info (which will always be kept private and protected).
  

Whether you've been to one of my workshops or webinars and left with a question you never got to ask, or you're deep in the weeds on agentic orchestration and want to learn some pro-tips, or you've just been seeing my posts and wondering what the heck all this AI stuff is about: this is for you!

First session is this Thursday (July 30th) from 2-3pm ET and continues biweekly from there; register here. Excited to see some of you there!

TLDR: DAAF v3.0.0 is out today, and it brings an ENORMOUS set of long-awaited features and valuable flexibility. Full R support, full OpenAI API and subscription support (use GPT-5.6 Sol in Claude Code!!), more battle-tested OpenRouter API support (use Kimi K3 and GLM 5.2!), far better and more intuitive UX and Docker management, privacy-preserving data analysis options, newer Claude Code version with access to Fable, and MUCH more!

If DAAF v1.0.0 was about initially sharing the concept of DAAF and AI-powered quantitative data analysis assistance, v2.0.0 (Gaining Altitude) was about rebuilding DAAF’s core framework engine for far greater flexibility and robustness, and v2.1.0 (The Frictionless Update) was about making it genuinely easy to install and run, then v3.0.0 is about something more fundamental to the mission of DAAF: who gets to use DAAF at all. This is an update designed to better meet researchers where they are. Every earlier version was geared towards a particular kind of user: someone who’s comfortable enough in Python, who has an Anthropic subscription/API budget to spend, whose data is non-proprietary and/or public enough to be safe to share with AI assistants, and who just wants to run a single project on a single machine. Each of those assumptions was a barrier to entry and a limitation of what DAAF could offer to researchers across the sciences.

v3.0.0 takes almost all of these most impactful walls down, and I’m hoping that if you’re reading this, you’ll find DAAF is now for you.

• The language wall falls as R becomes a first-class execution lane: enormous for social science, education research, epidemiology, and policy analysis, where R is often the native tongue. (Coming from Stata? Tell DAAF and it’ll translate any R or Python code it writes for you in the comments!)

• The provider and cost wall falls because DAAF can now run AI models from all major contenders: use an Anthropic API key or Claude subscription, use an OpenRouter account and API key (for any and all models available, inclusive of Claude/OpenAI/open-source), and/or use an OpenAI API key or ChatGPT subscription. Claude Code remains the driver of the system, but it can now “speak” with pretty much any service you have access to (which is especially valuable and timely given the currently intense cost-competition happening across the industry!).

• The sensitive data wall gets softened for researchers whose data can never leave a secure environment (HIPAA- or FERPA-governed, enclave-bound, or otherwise restricted): DAAF now comes bundled with a privacy-preserving synthetic-data generation and analysis workflow. In other words, DAAF helps you summarize/profile an existing dataset with a helper script from afar (with as much or as little detail as you can provide under your data protection agreements) and then uses the summary statistical output to construct a fully synthetic “twin” dataset it can operate on and analyze to help you rapidly prototype code without ever exposing it to your real data.

• The operations wall falls with a consolidated and much more user-friendly DAAF Control Panel interface on every operating system (no more remembering “run_daaf.sh” or “backup_daaf.sh” -- it’s just “daaf.sh” for everything now!), more advanced configuration options for any type of setup, a self-healing updater, better and more sophisticated backup processes, and security hardening throughout. We’ve updated the Claude Code version in the container, bringing a ton of nice quality-of-life updates there (much better subagent viewability) plus access to Fable, and do a better job of saving your Claude Code configurations (so rebuilding is now truly painless! No more logging in over and over).

• And the scale wall falls as DAAF can now support installing arbitrarily many versions of DAAF with different configurations, environments, and setups: run multiple projects side by side, each in its own container, on its own provider or endpoint, even its own DAAF version. For example, I now have my Anthropic subscription working with Claude in one container, while my OpenAI subscription works with GPT-5.6 Sol in another container! Advanced users can even point them to the same research work and have them duke it out.

All of this comes while actually enhancing the safety guardrails in the same sweep. This release expands DAAF’s defense-in-depth safety guards, adds more pre-flight system checks, hardens and enhances the development environment experience and workflow for contributors, fixes several research-integrity gaps, and is backed by extensive model-performance testing through our new DAAFBench analyses (Kimi K3 and GPT model benchmarks to be added shortly!).

To go with this release, I’ve also finally put together a full DAAF installation tutorial and walkthrough: Claude Code for Social Scientists: Get Started with DAAF in 30 Minutes. It takes you through the entire installation process and gets you oriented to the whole environment for people entirely new to Claude Code, Docker, and the Terminal, while also explaining in detail everything that’s going on to give you a good intuition of how it all works and how to go forward in using and learning these tools safely and responsibly from here. I’ve now got a much more robust video recording and editing pipeline figured out, so look forward to a much faster release cadence of these styles of tutorials in the future!

Phew. Lot more in the update (and many invisible, hard-won software development wins for the user!) than is worth getting into here, but you can see a fuller changelog in the GitHub releases page.

Usual caveats: I’ve worked really hard to get all of these various systems up and running and robust enough for reliable daily use. That being said, everything requires testing and feedback from users to improve so we know how things go under real-world conditions! Whether you’re one of the ~2,000 unique installers of DAAF so far or just getting started today, I’d love to hear from you!! If you have any feedback, ideas, suggestions, or just want to let me know how it’s going, I’d love to hear from you at our Discord community or at support@openaugments.org. For some more guidance on updating to v3.0.0 if you’re a current user, please see the addendum at the end of this post.

Everything about DAAF is free and open-source, and always will be. I remain more firmly convinced than ever that these tools have an immense transformative potential for the scientific community, and our society writ large by proxy… but if and only if we as trained researchers and experts are critically aware and informed consumers and contributors to these efforts. We must take an active role in shaping what rigorous, responsible research looks like in this new paradigm. We cannot afford to cede the territory to people more than happy to fill the void with “research-shaped objects” and slop.

The goal with today’s update is to make DAAF accessible and relevant to more researchers, in more circumstances, as carefully and responsibly as ever. With v3.0.0, it is hopefully easier for peer researchers and colleagues to get started learning how to harness AI responsibly and rigorously than ever before, and today’s as good a day as any to finally get started! I can’t wait to hear how it goes and what you all do with it. Want to learn more? Visit the full website for a nice breakdown of what DAAF is and what it offers.

Til next time!

Important Update Notes for Existing Users

If you’re coming from an existing installation, v3.0.0 is a larger release than its recent predecessors, and a few things will need your attention to update smoothly.

• Back up your work. For those coming from v2.1.0 (anyone who has the update_daaf.sh/ps1 script), the updater automatically asks you to back up your work. You should do this just in case! Otherwise, download your /research folder from the Docker Volume just in case for safe keeping. We’ve developed some really careful systems to not do anything harmful while updating, but better safe than sorry.

• Run your updater script TWICE. Because of a limitation in the older updater, you’ll just need to run your updater script twice in a row. The first run applies most everything, as well as some helpful updates to the updater itself. The second run uses the freshly upgraded updater to deliver the new daaf.sh/daaf.ps1 control panel for you to start using instead of the older helper scripts (keep them there! daaf.sh/ps1 uses them behind the scenes). If everything is already current, it simply reports “Already up to date!”, and it is always safe to re-run.

• If coming from a pre-v2.1.0 installation (any install without the built-in update_daaf script in your installation folder), use the migrate_daaf path instead: see the migration guide in the Installation and Quickstart guide.

• When you’ve started it up again, confirm your execution language. Python remains the default, set in CLAUDE.md for all agents to respect, but R is now a fully supported alternative. If you want to change the default, just ask Claude. You can also set it on a per-project basis; just let Claude know when you start a project or revisit an older project.

• New safety/reproducibility/auditability rules your custom scripts and workflows must respect. A handful of constraints are now enforced to protect reproducibility and the core audit trail: runtime package installs are blocked for both Python and R (ask Claude to instead add packages to the Dockerfile’s user-additions block and rebuild) to ensure actual reproducibility and portability; scratch and intermediate files must be written inside the project folder, never in the Docker container’s ephemeral /tmp folder (so you can keep them in the Docker Volume and via backups, otherwise they just disappear on rebuild!); shell commands must run one per call (no &&, ;, or || chaining); and agents no longer make git commits by default. That last item is a new opt-in ”Git commit management” preference - leave it off and every script version is still preserved in the working tree as your complete audit trail; turn it on and DAAF may offer Git version commit tracking suggestions at natural milestones for you to approve.

• Running more than one DAAF install. You can now run as many distinct installs of DAAF as you’d like! You’ll need to set up some command line arguments before installing to do so: Give each simultaneous instance a distinct DAAF_PROJECT_NAME and non-conflicting DAAF_PORT_* values in its environment_settings.txt so their containers and browser ports don’t collide. See the Advanced Configuration section of the Installation and Quickstart guide

Again, if you run into any issues, please don’t hesitate to create a GitHub Issue, reach us on our Discord community or at support@openaugments.org.

After a wild two months of basically touring the country to give AI workshops and trainings for conferences, academic departments, non-profit leaders, and research teams, the same question has come up in every single room: So what are we going to do about teaching and learning in the era of AI?

This is a super complex issue that has no easy or blanket answers, but what I’ll aim to do in this shorter (setting a self-intention here) post is suggest a few lines of challenging thinking to get people stirring around the possibilities. I think there are two really nice analogues/parables I want to surface that should get us moving in a productive direction: the first comes from my days as a public high school English teacher battling students about SparkNotes summaries, and the second comes from how we’ve come to measure progress and “benchmark” the capabilities of frontier AI models against one another. And note that I’m going to focus on the college/university teaching context rather than the K-12 context, at least in part because I think there are a LOT more complications when you throw younger minds and cognitive/emotional development into the mix.

So first, let’s lay out a basic mental model. The research on this frontier is quite messy and still coming to grips with what AI is actually capable of as of right now (as is the case with anything related to researching AI’s impacts and consequences, frankly), but my distillation of the current consensus is roughly:

• Learning new things is, and should be, hard (i.e., “productive struggle” or “zone of proximal development”). Obviously, arbitrary difficulty is not desirable/useful at all -- but short-circuiting the sort of necessary engagement and effort inherent in really grappling with learning a new thing (whether with AI, human tutors, explicit cheating, overly involved parents, etc.) likely leads to less durable and sustained learning over time (see, e.g., Rismanchian et. al, 2026).

• The growing capabilities of AI assistants makes this short-circuiting of difficulty while learning both easier than it has ever been, and more immediate. When a student can avoid the difficulty of grappling with a hard reading or a complex concept by asking the ever-present ChatGPT to summarize it for them, very little learning is happening, even if they might go on to perform fine according to current assessments/assignments (but very likely not!). Inappropriate use, and/or extended use, in this direction seems to cause reductions in general understanding, task performance, and skill acquisition (i.e., “cognitive surrender” and “cognitive offloading”).

These two dynamics together are a very bad time for anyone trying to teach right now, because it means that students may be sailing through assignments better than they ever have, on average, at the same time as they learn less than before.

So what do we do about it?

There are a few ways to think about this dynamic when it comes to assessing students’ learning. Going with a simple visual model, imagine we have exactly two student types: a higher-achieving/engaged/motivated/effortful/focused student, and a lower-achieving/disengaged/unmotivated/lazy/distracted student. Use whatever words you want, you get the idea; I’ll call them “High interest” and “Low interest” for short. In an ideal state, we measure how students are doing with an assessment and the results look something like this:

Now, in a pre-AI world, this is all fine and dandy. But the unfortunate reality is that things are starting to look a lot more like this:

In other words, Low Interest students can start to look very, very similar on assessments and assignments as High Interest students when they leverage AI.

Given these circumstances, instructors seem to be struggling with two very bad options in front of them to ameliorate this. The first option is to try and heavily police student AI use to make this plot look more like the pre-AI plot by making it difficult or impossible to use AI for assignments where possible. We’re talking proctored exams, written exams, oral exams, and so on. The second option is to perhaps surrender to the use of AI by some students, but do more to try and distinguish the students from one another by becoming more particular and specific about what “success” actually looks like. Roughly speaking, that looks something like this:

Based on what I’ve heard from colleagues via workshops and social media, neither of these tacks seem to be particularly successful or enjoyable by anyone involved.

So let’s talk about a third option, which importantly may not be available and/or tolerable to everyone given constraints on institutional policies, field- and topic-specific alignment, and your own disposition towards the proliferation of AI in society. But let’s talk about it.

First: Raise the floor

The modified assessment range pictured immediately above is maybe directionally correct, but how we adjust the assessment range and why matters a lot.

When I was an English teacher, I taught 9th and 11th graders up in Maine at a public high school. One thing that was always a struggle at the time was students’ use of SparkNotes. For those who don’t remember or aren’t aware, SparkNotes is basically a study guide platform that served up chapter-by-chapter summaries, essay topic ideas, reading guides, etc., for basically all the major books English classes would cover. The problem was that a student could choose not to do any of the actual reading assigned for class, quickly read the SparkNotes summaries for the assigned reading instead, and show up to class discussion/assignments conversant enough in the material to pass by undetected.

This meant low interest students were able to completely short-circuit the important parts of both having actually read and engaged with the material, as well as synthesized their own thoughts and ideas from it. The summaries they read were often hyper-generalist and ended up directing students into a sort of mediocre-but-not-wrong mindset about the main themes and ideas of a given piece of literature. Sounding familiar?

My imperfect, but I think useful, strategy at the time was simple: Some students will take the path of least resistance. Embracing this aspect of human nature and understandable student circumstances, how can we make a path of least resistance that is still useful for their learning and engagement? It’s not super important to dig into the finer design/details, but I started crafting something I called “Reader’s Guides.” The basic goal of this assignment structure was to:

• Literally hand students the exact passages that I felt were most important to understand for upcoming class discussion and more involved textual interpretation down the road (e.g., summative essays, debates, projects, etc.)

• Ask directed but open-ended questions that low interest students could answer using the selected passages alone, and that high interest students could enhance substantially with evidence from passages beyond them

It was a version of, “If you can’t beat them, join them and do it better.” I wanted to hand students a better shortcut, basically telling them: Listen, if you don’t have time, just open the actual text and read these segments -- it’s easier than trying to dig around on SparkNotes, and it’ll get you farther anyway. Not everyone loved this strategy, and there are definitely critiques to level here, but it worked very well for my intentions.

The wins ended up being on both frontiers: first, the low interest students would actually look at the text itself and engage. Some at a cursory level, of course -- but many went on to engage beyond that because the door was wide open, and there was no inertia to overcome to genuinely re-engage with the actual material as there was with SparkNotes. Second, the high interest students found it to be a useful outline and starting point for their notes and thinking (especially high interest students who struggled with harder texts and could use it as scaffolding).

The point of this example is to encourage current instructors to ask the same question of our assignments today, in the era of AI. How can we structure our assignments in a way that the path of least resistance via AI still offers some genuine value for students? How can we manipulate the assignment context and parameters in such a way that we can control what that looks like and encourage at least a cursory level of engagement and learning even in the worst case scenario?

A toy example: Students are going to submit a written assignment for your class. Applying the thought process above, we’d say: Okay, the path of least resistance is that students dump a bunch of course materials into ChatGPT or NotebookLM and get a finished document out, and hand that in. But what if we didn’t allow that to be the end of the story? What if we assume some students will do this, and what would it look like to secure at least a small victory for their learning while outright enhancing the learning of those that don’t? One suggestion: require students to use their AI assistant of choice to review their draft before submission, and then have students submit:

1. Their original draft

2. A few paragraphs’ synthesis of what their AI assistant got right and what their AI assistant got wrong in the review

3. Their updated draft

Low interest students who weren’t intending to take the assignment seriously now have to grapple with the fact that any initial slop from an AI is not necessarily worth taking naively and need to engage at least with one or two more steps of cursory engagement. High interest students who weren’t intending to use AI at all now critically engage with outside evaluation of their own work and get reps in with the very necessary and increasingly important skill of taste-making and discernment with respect to AI output. To the extent the AI gives awful advice or amazing advice, the high interest student benefits. You really only need to look at #2 and #3, which adds relatively little in the way of grading volume/burden at the end of the day.

This mindset will have varying applicability across different assignments, fields, topics, and courses, but I think it’s increasingly important for us as educators, and valuable for students themselves, to be thinking creatively about how we can encourage a more sustained engagement with course material AND critical engagement with AI tools even among students taking the path of least resistance.

Second: Raise the ceiling

Raising the floor in that way is just half of the equation, so let’s return to one of the earlier visual diagrams for a moment:

If you’re in a situation that looks like this, one could state it bluntly as: a low interest student with minimal effort and intending to invest the least amount of time possible can now naively corral a basic AI assistant to trivialize the vast majority of your assignment. Now that sucks, and I hate that for all of us. It is not good, and it is a disservice to the effort and intention you’ve put into your coursework and your students’ learning.

But I want us to shift our attention away from the low interest student for a moment (the “raising the floor” section was really meant for them), and let’s focus on the high interest student for a moment. Why would it not be the case that a high interest student, with genuine effort and intending to invest meaningful time with thoughtful and creative use of complex AI support should blow the sky off your existing expectations of the assignment? And I would venture to argue that these high interest students stand to gain even more from judicious and creative use of AI as described than the low interest students. That would start to look something like this:

Before we talk much more about that concept, let’s talk about the second parable of AI benchmarking.

Put briefly: as AI models get better and better, it has become increasingly critical that we understand how they’re getting better, and by how much. If you’ve been following the contents of any major AI model release from the frontier labs recently, you’ll recognize images like this:

We call this process benchmarking. Teams across the data science and machine learning frontier have been doing this for ages, but the basic strategy is for labs and researchers to come up with complex batteries of tests that any AI model can be pitted up against, and new models’ performance on said tests can help us understand the relative power/capability of that model against other models in a fair comparison. Each row in the image above (a summary of benchmarking for Anthropic’s Fable 5 model) represents a different kind of benchmark test used across the industry: SWE-Bench Pro, FrontierCode (Diamond), OSWorld-Verified, etc. etc. Each column represents a different AI model’s score on said benchmark, allowing us to (selectively!!) compare results and describe (imperfectly!!) where the state-of-the-art is. It’s not important that you understand what each one is or does exactly, but just know they each represent a different angle of evaluation and attack to see how different models do under different testing batteries.

Contrast the benchmark results above for Fable 5, released in June 2026, with the press release and benchmarking results for Claude Opus/Sonnet 4, released just one year earlier in May 2025:

The benchmark tests used here are SWE-bench Verified, Terminal-bench, GPQA Diamond, and so on. If you’re looking closely, you will not see a single one of the same tests across these two images, only one year apart. Why is that?

One reason is that when a mainstream benchmark is released and frontier labs start using it, they start optimizing for them during model development. Internally as Anthropic or Google or OpenAI develop new models, they’re constantly checking to see, “Well how well does this new draft version of our model do against X benchmark?” You might hear this colloquially referred to as “benchmaxxing” now, or Goodhart’s law with a new coat of paint: “When a measure becomes a target, it ceases to be a good measure.”

The point is that, eventually, every major model release begins to blow the benchmark scores out of the water. If every model gets about the same high score on a benchmark, that benchmark is said to be “saturated,” and it stops being a useful differentiator of how good one model is versus another. Labs and researchers are constantly developing new and harder benchmarks for this very reason: As models get better and better, we need more difficult and complex testing batteries to distinguish them from older models and from one another.

What I want to suggest is that your assignments are very likely saturated in this same way. The way you’re structuring your assessments has stopped being a useful way to delineate student ability, effort, engagement, and learning in this paradigm, and we need to think about how we can develop that next wave of “benchmarks” that raise the ceiling, raise the bar on our expectations for what is possible in a given assignment. That looks something more like this:

We reposition the floor to what’s possible with AI at a naive, low-effort, low interest level, and we reposition the ceiling to what’s possible with AI at a complex, high-effort, high interest level.

A toy example: I co-taught a course in basic data management practices for social science grad students back during my doctoral program. We covered things like reproducible workflows, basic exploratory data analysis and data visualization with Stata, etc. Our end-of-the-semester assignment was basically: Take some public raw datasets, clean them and join them, generate a bunch of nice data visualizations, run some basic descriptive analyses and interpretations, and write up a nice memo synthesizing insights about some of the key findings related to a simple research question.

I don’t want to freak anyone out (and hopefully this isn’t a surprise to people anymore), but this whole-semester assignment is now doable in about 30 minutes of human effort with the right AI orchestration systems. You could probably still get it done in closer to an afternoon of focused work going simpler (i.e., vibe-coding in Claude Code or Cowork).

So what would it look like to adapt our standards and expectations for assignments accordingly? What kinds of intensive, involved, and interactive inquiry could we demand of students now that the marginal cost of another data visualization and data robustness check is nearly zero? What degree of intimacy of understanding can we expect from students about their datasets when developing flexible dashboards and rapidly iterating/improving/revising data processing pipelines is an afternoon’s work? How can we encourage students to cultivate and demonstrate critical intuition about the potential pitfalls, failure states, limitations, and errors of their data work that are actually no less robust and measured than our own -- and, ideally, ultimately superior?

If these questions I’m raising for my former data management students sound a whole lot like the sorts of questions I’ve been raising for my peer researchers regarding use of AI in their own work lately, you’re right.

But anyway, let’s get concrete for this example. First, let’s raise the floor: What if we ask students not just to produce this fully reproducible analytic pipeline and artifacts, but also document (in a nice and orderly fashion) the exact battery of intuition checks, interrogative questioning, robustness verifications, supplemental analyses, and exploratory questions across stages that they pressed their AI assistants for to ensure they knew what was happening and that it was as intended? We can demand not just that students produce the artifact, but that they’ve done their due diligence to understand whether and to what extent they can really stand by the quality of said artifact -- even if they ultimately sailed through with greater levels of AI assistance than we’d like. The bonus: You have a clear sense of what failure modes they’ve missed that you can more readily check with support of AI yourself, and that should be part of the feedback and critique you provide back to them to help them improve.

Next, let’s raise the ceiling. If data communication and storytelling is a core value in said class (I strongly argue it should be!!), let’s ask students not just to produce a data visualization for each key insight, but a battery of options for data visualizations for each key insight. Ask them explicitly to iterate on a variety of plot styles, framing, levels of interactivity, levels of animation, etc., and iterate towards their final version (documenting the trail of candidates); ask for a minimum of 10 meaningfully distinct prototypes per key plot. Some are going to be garbage, and some are going to be phenomenal beyond your or their initial imagination; students will have to (get to) demonstrate taste and discretion and creativity, and what they produce in the end is almost necessarily going to be better than what they could’ve before through this wider iteration cycle.

Your imagination with, and understanding of, AI is the binding constraint

You might immediately say, “Brian that sounds like a completely unreasonable amount of work to ask of a student” -- yes absolutely… in a pre-AI world. In a world where students have access to Claude or ChatGPT, this is three to four well-crafted prompts, an hour of sifting through candidate results, and then another 30 minutes of polishing the final version (important caveat: you should probably know what exactly are students’ allocations for AI use on their accounts before you require highly intensive AI work!). Let me put it another way: if your reaction to an initial suggestion for raising the ceiling is not a reflexive “Oh my god that sounds insane”, you’re probably not actually raising the ceiling. You genuinely need to think bigger.

This is a type of imagination and creativity that I think we all need to start broaching now, and it’s especially urgent to start thinking along these lines because:

1. What the new ceilings should look like will vary significantly between fields, topics, disciplines, tasks, etc., and no one can really know what that looks like outside of the current experts (i.e., you!). As I mentioned explicitly above: Not every assignment is as amenable to this as the examples I’ve picked out, but I suspect anyone who’s struggling with the issues of student AI use in a given assignment as described above has, by corollary, meaningful opportunity to explore and apply the raised floor/ceiling concepts raised here.

2. Your ability to think creatively about what the raised ceiling can/should look like is going to be directly constrained by your understanding of what is currently possible with thoughtful and careful use of AI in your field. If you don’t have a sense for what opportunities exist about your work with respect to AI, I’m not sure how you can productively or accurately think this through on behalf of your students.

3. For that reason, your students’ ability to tackle the raised ceiling is also going to be a pure function of your ability to demonstrate and model these new skillsets and disciplinary opportunities for them in your actual classrooms and labs. The apprenticeship model remains extremely valuable here: people often can’t do what they haven’t seen, so you have to show them and practice with them. E.g., no graduate student ever learned to give a genuinely good conference talk by just reading online guides; they need to see their PI and peers do it well and poorly, and they need to do it themselves poorly and then well with feedback. So how can you make similar opportunities for them to see responsible and rigorous use of AI along the lines of what I suggested above within your classroom, from you, so that you can ask them to do the same in your expectations of their assignments and coursework?

4. The ceiling will need to keep moving as AI models themselves improve. I don’t think it’d be radical to suggest that as AI gets better and better, any assignment you could make today will be saturated a year from now. That’ll probably look like this, over and over again:

   

   And… well, with models like Fable coming out and no slowdown to that rate of improvement in sight, it’s going to be a very interesting question as to what it means for all of us when the human component of the bars shown above start to become the minority of the overall bar. But let’s not broach that conversation for now.

If you’re feeling a bit overwhelmed by those four bullets together, you’re definitely not alone. This is a crazy, crazy time, and I’m honestly not entirely sure where this leaves the modal college instructor: someone who probably is not compensated enough for their teaching efforts, who has a huge variety of other responsibilities (some of which they care about a lot more), and who is not given particular agency/autonomy over how to revamp materials especially in light of broader policies/restrictions on AI use more generally at their institution.

So to the extent that you, the person actually reading this right now, are not immediately struggling with said constraints, I implore you to take very seriously the opportunity to engage with this reimagining of the floor and the ceiling in this new environment on behalf of others in your field. To the extent you can create room and space and time for these discussions among junior faculty or colleagues, I think it serves everyone to do so. And to the extent you can engage with your students themselves in discussing this, I think they and you will be all the more clear-eyed for it.

I also want to be clear: despite how it may sound, my position is not that everyone should be using AI in their research and in their work (though there are many who seem to be arguing that vociferously; I frankly disagree with them). My position is that everyone needs to have a critical and nuanced understanding of AI to be able to make that decision in the first place for yourself and for your students in a genuinely informed way -- and that necessitates getting your hands dirty with AI and earnestly testing things out for yourself [see more on my thoughts for how to do that more concretely in this post; and please do see my note re: conflict of interest in the appendix below!]. Given the rate of AI capability progress at this time, it’s also not tenable to think you just need to do your due diligence once and make a final decision; this is a constantly moving target you need to engage with. The number of people who I know haven’t yet updated their perceptions of AI since mid-2025 is striking and alarming given how radically different the landscape has become. I really honestly don’t like it any more than you do (don’t ask me about my work-life balance and mental health lately since going full-time on AI work!), but I think the reality is that this is the nature of the beast and the era of AI we now live in, whether we like it or not.

Again, my goal with this post is not to suggest that these are silver-bullet solutions for everyone to overcome to these radically complex problems, but I do hope these examples and ways of framing can spur useful discussion and intuition for directions to start moving in. They won’t apply to everyone, and they certainly aren’t the end of the conversation, so I’d love to hear: Does any of this land with you in your courses, your field, your assignments? Why, why not, and how so? Have you already been playing with these concepts in some way, and if so, what has that looked like? I’d genuinely love to talk through your current problems of practice/assignments and see what we can come up with together in comments.

That’s all for now; sorry this post was yet again way longer than it should have been (“shorter post today” womp womp). In any case, now that I’m doing less work travel over the summer, my goal is to try and release more digestible posts on a weekly-ish cadence. Til next time!

Appendix/Notes:

I want to acknowledge what might appear like a complete conflict of interest: here I am, telling you its urgent and important for the good of society to explore uses of AI in your/your students’ work… because I run a business teaching people how to explore uses of AI in their work. That’s a very reasonable interpretation, but I’m going to actually flip the causality on its head: I run a business teaching people how to explore uses of AI in their work because I think it’s urgent and important for the good of society to explore uses of AI in your/your students’ work.

I’ve said directly to clients and colleagues that I am very explicitly trying to work myself out of a job here by sharing and open-sourcing basically all the materials I’m producing along the way (much more to come via Substack and YouTube now that I’ve done all that workshop work over the past month!), and I am very much excited for a day where teaching others how to do this stuff stops being a part of my day-to-day: not because I hate the teaching (I love teaching!), but because it means so so so many amazing things for the state of scientific progress in our society if that’s the case. And it means we’ve moved well past this (frankly) dangerous and confusing and disruptive time with a lot of inherent collateral damage.

To put a fine point on it: the vast majority of material for this post was initially created as part of a paid workshop I crafted for a client. I’ve got 2-3 more posts distilled from that same workshop on the way, all of which will be free and readily accessible to everyone via this free blog. The business model is very explicitly people paying me to come up with the materials, and then me sharing them for free after.

You obviously don’t have to believe me, but I hope my pro bono and open source portfolio can ultimately speak for itself!

As LLM models continue to advance in capability and sophistication across the board, there are three questions that I think anyone who cares about the potential of AI for enhancing and accelerating and democratizing scientific progress should be closely following:

1. How good are frontier models actually getting at facilitating increasingly complex research workflows?

2. Where are open-weight models in performance relative to the proprietary models?

3. How exactly should we be thinking about the cost-for-performance trade-off along the frontier? How cheaply can we produce worthwhile research with AI assistance?

There’s a lot going on right now, but I am extremely excited to finally share some results from my neck of the woods on all three of these questions. Over the last few weeks, I ran 17 different frontier models through 51 different tests (2500 total runs!) to examine: How well do different AI models handle the complexities of rigorous quantitative data analysis workflows? Introducing DAAFBench: Orchestration!

The goal here was to very explicitly understand: which models are actually capable of managing the processes, procedures, and protocols necessary for ensuring responsible, reproducible, and rigorous research work with AI agents? This is distinct from raw coding/analytic power (which I’m bookmarking as my next endeavor, predictably: DAAFBench: Analytics).

DAAF layers together a suite of architectural defenses and strategies from the current frontier of AI best practices to maximize AI output quality and force Claude Code to operate more like a careful and thoughtful researcher at every opportunity. One of the core features of this system is something called “agentic orchestration,” or the complex process of weaving together multiple AI assistants in concert to tackle increasingly complex research workflows: anything from iterating continuously on a data visualization script to developing entire data analytic pipelines from a given research question.

DAAF (and other orchestration frameworks) does this by giving a series of intensive instructions, guidelines, and workflows to the AI assistants as they work, breaking tasks down into more concrete and tractable sub-tasks, facilitating better coordination, managing work tracking and documentation, and enforcing adherence to a set of unifying work principles (e.g., auditability, human-in-the-loop, rigorous self-verification processes, etc.). But a framework like this only works if the AI assistants are sophisticated enough to actually understand, apply, and remember these instructions thoughtfully and consistently.

DAAFBench: Orchestration is a series of bespoke benchmark tests explicitly designed to test adherence to the research protocols and process guidelines of DAAF and assess a given model’s suitability for operating along guidelines that make rigorous, reproducible, and responsible social science at scale possible with AI assistants. In sum, I present multi-dimensional performance assessments for 17 models (both official Anthropic models, as well as open-weights models accessible via OpenRouter, which is immediately compatible with Claude Code and DAAF) across 51 different tests designed to simulate key moments of orchestration decision-making and protocol adherence.

DAAFBench: Orchestration results are a crucial step in understanding how models actually perform under these conditions and allow for far greater nuance in thinking about the exact use of specific models (especially more expensive versus cheaper, and proprietary versus open-source) across these types of workflows (and within!). Most importantly, it allows us to more directly track whether and when locally-hosted models (e.g., on a home computer with consumer hardware) can start to tackle these sorts of tasks, opening the door for unprecedented access to analytic capacity going forward.

Different models also have empirically different strengths (e.g., consistency v. average score), and these results let users of DAAF (and other research orchestration frameworks) make informed decisions about which models to use for *specific aspects* of their workflows. Lots of potential optimization to explore here.

Some interesting takeaways re: Anthropic's models as of late:

1. Fable is *actually* in a class of its own.

   1. Fable 5 leads the scoreboard at a Perfect average score of 93.9% — 11.1 points clear of the next model. It is also by far the most consistent model tested: 88.2% of its repeated cases scored perfectly across every criterion on every single repetition. Given that errors and issues quickly percolate and multiply in these chained agentic systems, this is an enormously valuable characteristic relative to competitors, and likely holds similar additional value in terms of code quality and other moment-to-moment junctures not tested by this benchmark.

2. Opus 4.7 was genuinely worse than Opus 4.5/4.6, but Opus 4.8 was a strong return to form

   1. Claude Opus 4.7 was widely criticized for being far too stingy in its effort, perceived as a strategy by Anthropic to manage intense compute constraints at launch time. The testing bears this out empirically: Opus 4.5 and 4.6 scored at 73.3% and 71.1%, respectively, Opus 4.7 came in at 68.4%, and Opus 4.8 returned to form at 79.6%. The 4.7 dip is even more pronounced when focusing only on critical criteria completion (“Critical-only”: scores counting just the must-pass criteria): 83.1%/83.9%/75.3%/89.1% across the four Opus releases.

3. Sonnet 4.6 is an enormously high performer relative to its cost

   1. Sonnet 4.6 posts a Perfect average score of 82.9% — #2 overall, shockingly ahead of every Opus model in this corpus (best Opus: 79.6%) except in the Critical-only category. And it does so cheaply: running this benchmark’s full battery on Sonnet costs roughly 41% of what it costs on Opus 4.8. Recognizing that there is uncertainty/imprecision in these results, and many model capabilities not explicitly tested by these current tests, these numbers nonetheless suggest Sonnet 4.6 is more than capable of orchestration work on a budget with potentially minimal trade-off.

Very validating of my intuitions to date, and I cannot stress enough how wild it was using Fable last week. Definitely entering a new era, conditional on all the policy stuff getting figured out.

DAAF and Claude Code also allow out-of-the-box use with a whole host of broader models via OpenRouter, which allows us to examine frontier open-weight models as well. Some key takeaways there:

1. Open-weight models are highly capable by comparison -- a real win for self-hosting -- but with a crucial decline in consistency

   1. DeepSeek V4 Pro (72.6%) and GLM 5.1 (70.9%) lead the open-weight pack, with the considerably cheaper DeepSeek V4 Flash a close follower (70.3%) — all landing in the same neighborhood as most of the Opus line. That being said, while much cheaper in terms of price-per-token, differences in token consumption patterns still place them in roughly the same cost bracket as Sonnet (with the exception of DeepSeek V4 Flash). The empirical trade-off then is less about price versus raw performance, and more about the value of control and performance consistency. Open-weight models allow for self-hosting, modification/fine-tuning, and long-term support guarantees (i.e., not worrying about Opus 4.5 getting sun-setted in a few months when multiple analyses and products rely explicitly on it). But the testing battery also reveals a marked drop in performance consistency: the Anthropic models perform predictably for the same tasks (for better and for worse), while the open-weight models perform more erratically on repeated runs. Moreover, DeepSeek Pro carries the corpus’s second-highest timeout rate (26.8% of runs) across all models tested, so its scores come with a broader reliability asterisk as well.

2. Local-class open-weight models just aren’t there yet, unfortunately

   1. Qwen 3.6 27B, the best of the small open models here, manages 61.9%, and it’s at its weakest in perhaps the most important part of the overall orchestration pipeline: careful skill routing (basically: How often does it load the necessary grounding references before it responds to questions? 31.1%) and mode classification (basically: How often does it correctly assign a natural language query to the right workflow process? 46.7%). Gemma 4 31B timed out on 70 of 153 (45.8%) of its runs, often (per transcripts) not actually doing any work at all when queried: by far the worst reliability in the corpus. Some of these results may be simple harness incompatibility (Claude Code is not optimized for communicating with Gemma or Qwen, to be clear), but nonetheless point to more necessary improvements/development to be viable for rigorous research orchestration workflows. That being said, they will get there someday, and with DAAFBench, we’ll now be able to know when they do!
      

IMO, these are *the* frontiers to watch for a truly hopeful democratization of research capacity via AI! Self-hosting and governance, especially for sensitive data analysis work and research, are enormously valuable and unlock a ton of important use-cases. Local-hosting (i.e., being able to run this sort of work on a single consumer-grade PC) will take some more time yet, but we're getting there slowly and surely.

There’s a LOT more to explore and learn in the full interactive website explorer; I strongly recommend bookmarking a deep dive at some point if you do any quantitative data analysis with AI assistance! Would love to hear what people see and think, especially if you have any ideas for improving the value and relevance of this specific benchmark going forward. Many more models are coming out, and I intend to keep this page updated as more become available (e.g., GLM5.2, Kimi 2.7, etc.).

That’s all for now. Til next time!

TLDR: If you’ve been wondering how to use Claude Code as a quantitative researcher of any kind, what Agentic Orchestration is all about, or more generally are curious about how AI might benefit the world of science and research: I’ve finally made a very nice, very accessible, and hopefully very informative homepage for the Data Analyst Augmentation Framework (DAAF) -- daaf.openaugments.org -- and I think you’ll really want to take a look! I’ve spent an enormous amount of time on interactive explainers, intuitive diagrams, and a hopefully very compelling explanation of what AI for research can and should look like when we center responsibility, rigor, and reproducibility.

Even if you’re already deep in the weeds using Claude Code as a researcher, I hope it’s also a nice inventory of extremely valuable features/functionality you’ll probably want to adapt for your own setups!

It’s been just about three months since I first launched DAAF, and we’ve come quite a long way from the scrappy v1.0.0 version.

One of the greatest frictions I’ve navigated since launch is trying to explain to people what DAAF is, at least in part because it relies on the person understanding three big things:

1. What’s currently possible for quantitative data analysis/social science research with Claude Code in the ideal state

2. Why Claude Code out of the box is actually extremely bad at doing the things we’d want it to do for good data analysis and research, despite good data analysis looking a lot like good code (which, obviously, Claude Code was explicitly designed for)

3. How hard and complex and weird it really is to get Claude Code to operate more like what we imagine is possible in #1 from where we actually start in #2

In the months since DAAF’s launch, I’ve presented on its capabilities for grant foundations, a variety of R1 institutions, an enormous webinar of nearly 200 education researchers, research thinktanks/institutes, and more. After many conversations and a lot of product development on my end, I think I’ve finally honed a clear articulation of what makes DAAF -- in my mind -- a necessary step in getting the field caught up on where AI for research currently is and should be.

I won’t go on in detail here, since I think the website is by design going to be far more compelling than anything I can type out on a simple page. But I hope people find this useful on a lot of levels whether they want to use DAAF or not, and I hope it brings valuable attention to the exciting opportunities that we’re facing in a world where the scientific community collaborates to bend this wild lightning rod of AI towards ethical, responsible use together.

What are you waiting for? Go to daaf.openaugments.org to see nifty interactive terminal demos like:

Or fun visuals laying out the many features/capabilities of DAAF like:

Up next on the work docket, for those interested:

1. An in-depth video tutorial walking people through how to install and use DAAF! I’ve already drafted a really nice onboarding guide for the website here, but the video tutorial will be a nice companion for people who need a little more guidance to get moving.

2. A broadly accessible explainer video trying to communicate a sort of “Here’s what everyone needs to understand about AI in 2026” -- part video version of this post on a more useful mental model for AI progress, part open-access version of my introductory workshop on using AI for researchers/non-profits/higher ed.

3. R support in DAAF! Most of the work here is already done, I just need to test the hell out of it. Please let me know if you’re interested!

4. Open-source model benchmarking for DAAF! Figuring out, basically, what are the cheapest models that can still reliably follow DAAF instructions/context engineering practices/workflows?

That’s all for now. Til next time!

When I launched the Data Analyst Augmentation Framework v2.0.0 six weeks ago, I wrote that the major update was about going “from usable to useful” -- rebuilding the orchestrator system for maximum flexibility and efficiency, adding a variety of more responsive engagement modes, and deepening the roster of methodological knowledge that DAAF could pull upon as needed for causal inference, geospatial analysis, science communication and data visualization, supervised and unsupervised machine learning, and much, much more.

But while DAAF continued to get more capable and more useful for those actually using it… Well, it was still extremely annoying to use, generally obtuse, and hard to get started with, which means a lot of people who were interested were simply bouncing off of it.

That all changes with the v2.1.0 update, which I’m cheekily calling the Frictionless Update for three key reasons:

1. Installation happens in one line now

From a fresh computer to talking with a DAAF-empowered Claude Code in no more than ten minutes on a decent internet connection. This is really it:

Which means it’s easier than ever to get started with Claude Code and DAAF in a highly curated, secure environment. To that point, you still need Docker Desktop installed (I’ll talk about that more in a sec), but no more faffing about with a bunch of ZIP file downloads and commands in the terminal.

The simplicity of this is even crazier, given that…

2. DAAF now comes bundled with everything you need to make it your main AI-empowered research environment

No more messing around with external programs, installations, extensions, etc., it just works from the get-go with everything you need to thrive in your new AI-empowered research workflows with Claude from the moment you run the install line.

Thanks to code-server, DAAF automatically installs a fully-featured version of VSCode in the container, accessible in your favorite browser: file editing, version control management, file uploads and downloads, markdown document previews, smart code editing and formatting, the works. Reviewing and editing whatever you work on with DAAF has never been easier.

DAAF also now comes with an in-depth and interactive session log browser that tracks everything Claude Code does every step of the way. See its thinking, what files it loads and references, which subagents it runs, and look through any code its written, read, or edited across any project/session/etc. Full auditability and transparency is absolutely mission-critical when using AI for any research work so you can truly verify everything its doing on your behalf and form a much more refined and critical intuition for how it works (and how/when/why it fails!). Some of the most important failure modes I’ve discovered with AI assistants (DAAF included) is it simply doesn’t load the proper reference materials or follow workflow instructions; this is the single most important diagnostic tool to identify and fight said issues, which I frankly think everyone should be doing in any context with LLM assistants. This took a lot of elbow-grease, but I think it’s the single most important thing I could do to help people actually understand what the heck Claude Code gets up to and review its work more thoroughly.

These two big new bundled features are in addition to installing Claude Code, the entire DAAF orchestration system, bespoke references to facilitate Claude’s rigorous application of pretty much every major statistical methodology you’ll need, deep-dive data documentation for 40+ datasets from the Urban Institute Education Data Portal, curated Claude permissioning systems and security defenses, automatic context and memory management protocols designed for reproducible research workflows, and a high-performance and fully reproducible Python data science/analysis environment that just works -- no need to worry about dependencies, system version conflicts, or package management hell.

With the magic of Docker, everything above happens instantly and with zero effort in one line of code from your terminal. And perhaps most importantly (and why I will keep dying on the hill of trying to get people to use Docker): setting up DAAF and Claude Code in this Docker environment offers critical guardrails (like firewalling off its file access to only those things you explicitly allow) and security (like creating a convenient system for securely managing your API credentials in a way Claude can use but never see) that prevents all of the crazy “Claude Code bricked my hard drive and destroyed three years of work in 5 seconds” horror stories. I strongly and firmly believe that no one should be using these AI empowered tools just willy-nilly on their home or work computers; there are just too many ways things could go very very wrong.

It’s just too bad Docker is a huge pain in the butt to manage and relatively few researchers are familiar with it. Oh wait…

3. Everything you’d want to do with DAAF is now just one convenient utility script away

Users no longer need to think or worry about Git/Docker or pretty much any of the previous command-line frictions involved in managing your research files:

• Want to launch Claude Code in the secure DAAF Docker environment? bash run_daaf.sh

• Want to back up your research folder for safekeeping or sharing? bash backup_daaf.sh

• Want to reset your DAAF from a saved backup? bash restore_from_backup.sh

• Want to restart your Docker container to install new libraries? bash rebuild_daaf.sh

• Want to run VSCode for file management/editing? bash run_vscode.sh

• Want to run the session log explorer for auditing and review? bash view_logs.sh

• Want to view your analytic Marimo notebooks? bash view_notebooks.sh

• Want to update DAAF to the latest version? bash update_daaf.sh. You might even call that a… frictionless way to… update 👀

I built DAAF for researchers, many of whom are brilliant at methodology and domain expertise and statistical reasoning, but who didn’t sign up to become Docker administrators and mess around with weird file management issues. So the most important thing I could do for v2.1.0 wasn’t to make DAAF smarter -- it was to make the entire experience of using DAAF dramatically less painful and more intuitive for everybody.

Put #1, #2, and #3 above together, on top of the existing powerhouse of analytical updates and AI research workflow management tooling I put together for DAAF v2.0.0 a few weeks ago, and the interactive User Support mode I put together in v2.0.1 to help people not just use DAAF but actively learn from it (basically: ask Claude for help learning how to use DAAF’s workflows or understand how LLM assistants and context engineering works!), and now I think I can fairly confidently say:

DAAF is hands-down the best way to get started with Claude Code for data analysis and research

For the past several months, when people asked me “should I try DAAF?”, my honest answer included a lot of caveats. Yes, but the installation might seem a bit intimidating. Yes, but you’ll need to get comfortable with Docker. Yes, but I’m still really working on it week-to-week and updates can be a pain. Yes, but you’ll be reading files in a terminal and it’s kind of annoying to manage unless you figure out how to link VS Code into the system.

The caveats stop today. I have put hundreds and hundreds of hours over the last six months into making what I wanted all of my colleagues to have the second I realized what Opus 4.5 could do for statistical analysis back in November: a free and open-source toolset that makes it easy for any researcher of any technical capability to responsibly and rigorously use Claude Code to accelerate and enhance their research. The work is far from done, but DAAF v2.1.0 is finally something that I can hand to any of my colleagues and mentors from any point in my career, and know that they’re going to be in good hands.

DAAF is no longer just a simple instructions framework: it’s an all-inclusive, curated suite of tools that work together to implement a ton of best practices for using AI in the modern era. The analytical pipeline, the rigorous self-validation processes, the safety guardrails, the file management, the methodological Skills/references, the session logging transparency, the backup and update system, and the documentation. All designed for researchers who want to use AI to accelerate their work without sacrificing the rigor, reproducibility, and transparency that their work demands. I’ve been using this version myself on a variety of side-projects over the past few weeks, and I can confidently say this feels extremely good and powerful to use for real data work.

How to get started with DAAF v2.1.0

If you want to get started with DAAF from scratch, this page will walk you through the exact installation instructions. In the coming weeks, I’ll be launching the stand-alone DAAF website with a more visual walk-through, and I’ll also post a full installation and getting started walk-through tutorial video. More to come soon, I promise! Very long overdue on both fronts, and I don’t blame people for getting impatient with me there.

Want to learn a little bit more about how it all works before you dive in? Take a look at this super in-depth and interactive explainer I put together to show you how a DAAF analysis works from start-to-finish!

If you’re one of the over *1,000* folks who’ve already used DAAF to date, fear not: I also spent an enormous amount of time putting together a “migration” script that makes it painless and effortless for you to fully update DAAF to this latest version, no matter when you started and no matter how many framework customizations/edits you’ve made to it in the meantime. After that, you can use the aforementioned update_daaf scripts to stay up-to-date from here on.

This was a hellish design challenge, but I’m glad to have figured out some pretty clever ways to manage all the possible update conflicts by leveraging Claude Code directly to help users resolve things via Git. You can find all of the instructions for the migration in detail here, but rest assured -- it’s just a single command! It’ll back up your entire DAAF folder first just to be safe, detect what version you have installed, and then walk you through resolving any conflicts if they arise.

Please do tell me if anything weird happens when you try to run these scripts!! I will do everything I can to get that worked out with you. The folder backup is the most important and most well-tested part: as long as that goes off without a hitch, I can help you along with anything else!

And if you try it and it works -- tell a colleague. The best thing that can happen for this project right now is more researchers using it, stress-testing it, expanding it for others, and telling me what they need. If GitHub metrics are to be believed, we now have over 1,000 unique installs of DAAF. Help me keep making this a useful tool for more people, more researchers, more data scientists. DAAF is currently the worst it will ever be as long as the research community comes together to identify how we can make it better!

Less flashy but still very exciting updates and improvements

A few things that don’t make the cut for a headline for most people but meaningfully improve the experience:

• OpenRouter support (experimental). You can now run DAAF through OpenRouter if you want provider flexibility beyond a direct Anthropic API key. It works, but it’s early -- direct Anthropic access remains the recommended option and I’d flag this as a use-at-your-own-risk situation for now. But this is the beginning of being able to use DAAF with the whole world of open-source models like GLM5.1, Kimi K2.6, Gemma 4, etc. etc., which RADICALLY changes the game in terms of pricing and costs. For example, GLM5.1 seems extremely capable and similar to Opus 4.5, and it’s about 1/5 of the cost! I’m in the process of building an intensive “process adherence benchmark” to figure out which models actually are capable of following DAAF’s complex research workflow instructions well, so stay tuned for more.

• Environment variable support. Secure API key configuration now lives in a single environment_settings.txt file on your host machine, outside the container. DAAF’s safety system prevents Claude from ever reading it directly, and this adds a lot of convenience especially for people downloading data from access-restricted servers.

• Preliminary phase notes persistence. DAAF’s specialist agents -- the ones that do source research, data profiling, and synthesis -- now save their complete findings to disk as markdown files in output/preliminary_notes/. Previously, the coordinator held compressed summaries in its own working memory, which meant later stages of analysis were working from shortened versions of earlier findings. Now nothing is lost to summarization. This is a quiet change, but it genuinely improves analytical continuity across long sessions.

• Specialist agent word limits raised. General agents can now return up to 2,000 words (doubled from 1,000); data profiling agents up to 3,500 (from 2,500). Less truncation means more complete findings, same idea as the point above.

• Automated testing pipelines. Every proposed code change now runs through script quality scanning, unit test suites, full lifecycle tests, and pre-commit checks. This is the kind of infrastructure that’s invisible when it works -- and painful when it’s missing. DAAF is starting to look like a real software project rather than a research prototype, and I mean that in the best way.

I cannot overstate how much work went into making this feel simple for the end-user. Cross-platform shell scripting (for the above convenience and install scripts to work for MacOS and Linux and Windows) is one of those tasks that sounds straightforward until you’re three days deep into debugging why a specific version of PowerShell bundled on Windows 10 handles path separators differently than Windows 11, and you’re questioning every life decision that led you to this point. I had to learn how modular testing and CI pipelines worked, which I am glad exist and are as robust as they are, and I hope to not think about again for at least a little while. I suspect there are still many edge cases I couldn’t catch on my own; if you hit any issues, please tell me and I’ll do everything I can to get it sorted out.

What’s coming next

• Full-fledged R support. First-class R language support, plus dual-language handling for Python and R in tandem. This has been a long time coming -- I know a ton of people have been asking for it, and hopefully the wait will be worth it.

• Model Adherence Benchmarking. I’m building an automated benchmarking process to systematically test how well different Claude models follow DAAF’s conventions. This is the beginning of understanding which settings actually matter, and whether other models or providers are viable yet.

• More video tutorials. Expanding the library of guided walkthroughs and demos is long overdue, but will hopefully be extremely useful!

• Full standalone DAAF website with all features, documentation, help files, etc. in a much more navigable and user-friendly format than the existing GitHub.

That’s all for now. Just note I’ll need to take a bit of a mini-hiatus from public content creation as I power through several intensive university workshops introducing peer researchers to agentic AI and DAAF over the next few weeks. Til next time!

TLDR:

• Opus 4.7 is lazier than Opus 4.6, and it seems to react to prompting instructions meaningfully differently per Anthropic directly.

• That means any carefully curated workflows you’ve crafted to teach Opus 4.5/4.6 what context/documents/references to load at the right time may not work nearly as well for Opus 4.7. Incomplete context inevitably leads to rapid performance degradation and hallucination city, which is probably why many think Opus 4.7 is such a downgrade.

• This type of drift is probably going to be inevitable with model advancements and harness updates, because behavioral consistency is just too multi-faceted for model providers to optimize around -- especially as model and harness development velocity continue to accelerate. In other words: we really cannot take backwards compatibility as a given.

• Key takeaway: If you aren’t actively logging and benchmarking model adherence in the context of your specific workflows, you absolutely need to start doing so regularly ASAP, because this will definitely not be the last time this sort of issue happens with a new model launch. I’d argue this is also going to become increasingly likely for seemingly insignificant harness updates, as well.

• Bigger picture: Context engineering is still a very weird, very volatile frontier. Every model reacts slightly differently to different prompts, harnesses, context engineering techniques, and more. For my money, this is probably the biggest barrier to greater and more rapid societal adoption of AI.

The launch of Opus 4.7 this week has thrown a lot of the Claude userbase into disarray: is this a downgrade from Opus 4.6 despite the purported benchmark improvements? I mean, just take a look at some of the random Reddit threads I pulled into the image above from just the past 24 hours. This isn’t even an inflammatory cherry-picked subset: there is not a single positive headline about Opus 4.7’s performance in my entire feed across platforms right now. And this is coming hot off the heels of many, many complaints about Opus 4.6’s performance apparently degrading without any transparency over the past couple of weeks. Oof.

I can’t speak to every issue, and I can’t pretend to have a crystal ball to help explain why you, specifically, might be having problems if you are. But as I read more and more examples of frustrations people are having, I’m seeing at least one common thread and likely explanation for why this rollout has been such a mess -- and why this definitely will not be the last time it happens.

Let’s get back to basics for a second. I’ve spent most of the past month teaching fundamental AI principles to a wide variety of researcher audiences, and there’s one key lesson that is essential here:

Modern AI’s usefulness lives and dies on context

Pretty much every modern AI workflow relies heavily on getting the model to read and reference useful guiding information before it acts. In other words, is it loading the right contextual references (local files, websites, code documentation, etc.) at the right time to do your requested work as expected/desired? This is how we fight slop: Give AI an index of known-useful information it needs to begin with, and then let it search over when to surface key references based on the task at hand. This is the core design challenge for all context engineering, all agent orchestration systems, and basically any well-written Skill with progressive disclosure. To put a fine point on it: AI systems getting better at dynamically and intelligently curating their reference files while working is, in my view, the main reason why AI capabilities have been exploding since Opus 4.5 in November. It is the paradigm shift enabling everything crazy this year.

But without the right context loaded at the right time, most LLM assistants very ungracefully degrade into overconfident hallucination machines -- worse than functionally useless, they can become actively dangerous. Because they often don’t know when they’re doing it (though this is an active area of development), you probably won’t know either, and that can lead to many, many, many problems down the road well after it’s particularly easy to address.

As a simple illustrative example: you might ask Claude Code to help you change something in a given code file or document that ideally requires some good contextual understanding of how that change will need to percolate across multiple files. Claude has generally been good about doing some careful spelunking for relevant keywords across files, reading relevant design/style documents, tracing through some prior Git commits to identify recent related changes, and then coming up with a pretty comprehensive plan about what a proper edit will need to look like to address the task at hand and integrate carefully into the rest of the codebase.

Think about how many files it needs to read to do a basic task like this as well as you’d want. Think about how many lines of information it needs to examine in each of those files. What if it does those searches and decides: “You know what, let me just read the single relevant line in this file and then move on -- that’s probably all I need.” Or what if it decides, “Well, I don’t want to read too many of these, so let me just pick a few and move on.” Or what if it just… doesn’t actually read any of the files at all? What will the edits it proposes look like, then?

Ask yourself right now: How would you notice the difference in these scenarios as this is happening? And would you have any way of recognizing it before you move onto the next task?

It’s not dumber, it’s lazier

This style of degraded reference file searching/reading is probably what’s happening in at least ~80% of the issues I’m seeing people report with Opus 4.7 (and, likely in a similar way, reported 4.6 degradation over the past couple of weeks). It’s not that the model is outright dumber than Opus 4.6/4.5 -- it’s that it appears to be dumber because it’s doing a very bad job of choosing to load the right context/info/documents/references in the same way past models have. And because it’s not loading the proper info, it instead resorts to confidently responding with pure hallucination, or using only incomplete contextual information. In those circumstances, it’s not about intelligence but about needed context.

The root of the problem is then the model being some version of lazier, not dumber per se -- which aligns with what we know about Anthropic’s current heavy bottlenecks on their compute resources. They have every incentive to try and throttle users where ever possible (”token efficiency strategies”), and this is leading to major systems degradation because Opus 4.7 just isn’t loading the things it should be as comprehensively as it should be. It’s also the case that it’s simply responding to progressive disclosure and context engineering directions differently.

You don’t actually have to take my word for any of this: this is all explained directly in Anthropic’s recent guidance on how to adapt to Opus 4.7. This sort of public blog post is notable because Anthropic is explicitly telling us that the Opus 4.7 update has introduced breaking changes in a way I don’t think they’ve done for any past release. Put a different way, backwards compatibility is no longer a feature of Opus 4.7: workflows you made for Opus 4.6 and Opus 4.5 may not work well anymore, and it’s your job to figure out how to fix things going forward.

If you take nothing else away from the launch fiasco, it should be the crystal clear reminder that these sorts of model effort and context engineering drifts are likely inevitable as frontier models continue to develop. It is impossible for model providers to guarantee a new model will adhere as faithfully to your context engineering instructions the way you happened to write them for a prior model, especially because the ways people are using these models are becoming so much more diverse. In fact, even basic harness version updates (e.g., updating Claude Code from version 2.1.87 to 2.1.90) have similar potential to affect model adherence to context engineering instructions (e.g., if they change how certain core Tools like Read() or WebSearch() or Agent() are written or are specified in the System Message). Combine that with increasing pressure to launch models and harness versions with greater velocity, plus the use of said new models and harnesses to help build the next ones… And yeah, this is a formula for similar breaking changes all over the place from now on.

You should invest in better logging and monitoring systems ASAP

Because these sorts of updates are inevitable -- and frankly the whole dang point of AI technological advancement right now -- you are setting yourself up for a WHOLE WORLD of frustration in the next few months and beyond if you’re not explicitly designing for potential changes in models’ adherence to your context engineering pipelines.

What does that mean, practically? First and foremost, it means that you should have extremely detailed ways of tracking:

• Exactly what tools are my agents and subagents using, and when?

  • Exactly what files are being read/written via these tools?

  • Exactly how much of each file is being read/written via these tools?

  • What is the actual sequence of each file being read/written via these tools?

• Exactly how reliable and consistent are these tool patterns from run to run?

If you can’t answer these questions off-the-cuff, you’ve got work to do but don’t need to panic too much. Thankfully, many harnesses like Claude Code already track these via session transcripts automatically, which is an extremely helpful starting point. Just ask Claude Code to tell you where the .jsonl files are stored and tell it you should set up a system to start archiving and auditing them formally. I would also highly recommend changing your “showThinkingSummaries” to True in settings.json, alongside setting your /config verbose output on, to get better visibility into why Claude is making some of its context loading and tool calling decisions as it works.

But it’s one thing to track them and another thing entirely to actively monitor them: You need to have some ability to assess the extent to which your model is actually doing what you want, the way you want, according to some kind of opinionated stance on what optimal behavior actually looks like. One way to do this is by having a baseline set of prompts you can run on a regular basis with ground-truth correct sequences of tool calls, file reads, and file writes that you personally approve of. With that in hand, you can then run the benchmark prompts on any given day -- say, when a new model drops, or ideally something like weekly for your own peace of mind -- and have a clear sense of the extent to which the models are behaving according to your context engineering protocols or not. This is how we track degradation for your specific use-cases in a more coherent and robust way than just the vibes you often see across Reddit and X (e.g., “Oh Opus 4.7 really doesn’t get the carwash puzzle” over and over again). Don’t forget to run each such prompt multiple times, because the probabilistic/non-deterministic nature of LLMs means one test success doesn’t actually imply another later on.

To be clear, formulating these adherence checks will be extremely idiosyncratic and case-by-case based on exactly what your system looks like and what it’s supposed to do, and it becomes unfortunately non-trivial the more complex your orchestration system is. Relying on the standard tool-calling benchmarks that model providers tout is simply not a sufficient substitute for knowing how well these models will actually perform in your specific contexts.

For DAAF, as an example, I have a couple of benchmarking prompts that I run through every new version to check into how various files are read, how tasks are handed off, and what reference files are read by various subagents, across different styles of workflows I’ve built into the system. I recently updated the logging system in v2.0.0 to capture every agent and subagent transcript for any given project run, and spent a significant amount of time over the past couple of weeks using these logs to chart out every agent call, file read, and tool call from a sample project for my own edification and as an educational tool.

But while I’ve been doing these sorts of spot-checks manually since DAAF launch, you can certainly bet I’m going to be spending a lot more time over the next week or two formalizing these systems into a repeatable, quantifiable benchmark, and I think you should, too. Software developers already know that having a test battery suite is basically non-negotiable for system reliability, but making an analog for agentic AI systems is quite a different animal because it’s rarely going to be as simple as “Did this throw an error or not.” Working with your own AI to come up with scalable monitoring systems is a very good idea to make this task more tractable, but beware: Your checks will probably be better off NOT relying on LLM-as-judge or any qualitative support from the LLMs themselves, given the possible contamination/contagion effects of system degradation due to model/harness changes also causing LLM-as-judge monitoring system degradation.

While very unsexy work, this will save you enormous headaches in the long run. I would also argue such work will be amazingly valuable for three additional reasons:

1. You are going to keep improving and changing and adjusting the way your context engineering systems work. You should absolutely have a clear sense of how these changes affect downstream adherence to your system beyond just a handful of informal checks.

2. Given compute availability and cost pressures, every model provider is going to be surfacing more options and levers that users can pull to conditionally change model effort levels and other parameters. We already saw this with Opus 4.6 having multiple effort levels, and now with Opus 4.7’s Adaptive Thinking option. An adherence testing suite will let you explore the actual ramifications of these potential cost-saving options for your system, specifically.

3. This is also a way to give yourself a clearer sense of model/provider optionality and prevent vendor lock-in. Systems like this will allow you to better understand whether and when, say, open-source and local models start to hit sufficient capability that you can start offloading certain elements of your pipelines out to these instead of paying for the private frontier models (a major accessibility milestone I’m excited about for DAAF!). It also will help you better understand when you should start seriously considering competitors (e.g., Codex v Gemini CLI v Claude Code) and keeping your options open.

All said, it simply isn’t safe to assume that one model version will follow your instructions the same way as any other model version, so we need to be diagnosing and tracking this potential source of system performance drift diligently. Reliability and trust are going to be some of the most important currencies in this next phase of AI building, so time here is likely going to pay dividends in ways we can’t yet fully understand.

Using AI well is still just way weirder and more esoteric than most people realize right now

Last month, I wrote about how weird the current frontier of AI capability is right now. I’m finding the three-part mental model I proposed there more and more useful as time goes on:

I recommend reading the full article if you haven’t already, but the TLDR is that the quality of output from any AI right now is a combined function of (1) base model capability, (2) the specific frameworks and tooling it has available to it (e.g., why Claude Code is so much more capable than basic web interfaces), and (3) actual user prompting skill. I presented it as a sort of multiplicative function -- being sufficiently bad on any one of those fronts is enough to tank the whole operation, while putting all three together at a high level unlocks some really high heights on the developing frontier for AI capability. The other big idea I try to get across is that we’re really still just figuring out what it means to prompt models well, and how to make them use advanced Tools reliably while operating within complex orchestration frameworks. In other words, those latter two values of the model are shifting particularly fast in a way that’s generally not visible to the average person (likely only tracking the overall frontier model benchmarks alone).

This whole Opus 4.7 fiasco makes it clear that there are two more complications we need to incorporate into this model:

1. Base Model AI Capability can’t really be thought of as a singular value now that model providers are going to allow for varying token/thinking budgets and “effort” levels. It’s not as simple as “Opus 4.7 is better than Opus 4.6” when Opus 4.6’s extra-high thinking/effort model might far outplay Opus 4.7’s default middle thinking/effort mode. And exactly how those performance battles work out may be highly context-specific.

2. There are enormously weird interaction effects happening across these parameters, as well. That is: different orchestration frameworks and tools and harnesses will actually work better and worse with different models -- there’s not actually some linear measure from bad-to-good for framework/tool/harness quality. Similarly, what makes a good user prompt for one model may not be so for another model.

These two complications really reinforce my broad takeaway from the full post: it is still just way too weird and way too complicated and way too idiosyncratic to try and get the full squeeze of capabilities out of AI right now. AI isn’t actually getting simpler -- if anything, the Opus 4.7 launch makes clear that things continue to get only weirder as the months march on by. As long as this remains the case, I think it’s just not feasible for the vast majority of the working population to use AI tools well and reliably at scale, and so I have serious doubts about the speed with which AI can genuinely upend any particular profession, field, or society writ large as a result. That doesn’t diminish my other core recommendation from the post, though: It is still an enormously valuable time to get your hands dirty and start learning these intuitions and complexities yourself. Hands-on experimentation is the only way to begin surmounting this skill curve; the best time to start was six months ago, the second best time is now. See my full post linked above on the mental model for fuller recommendations on what that can look like to get started.

That’s it from me for now. I’ll be excited to learn more about what other issues rise to the surface from this launch, but until then: good luck out there!

Learnings from the past month since DAAF v1.0.0

It’s been just a little over a month since I first shared DAAF, the Data Analyst Augmentation Framework, with the world.

In that time, I’ve been deeply grateful to have so many fantastic conversations with colleagues, friends, family, fellow researchers, and more than a few skeptics, about what AI could and should mean for research and academia.

The consensus among people who’ve thought a lot about this is basically:

1. Everyone sees there is immense power and potential

2. Everyone sees there are a LOT of problems to grapple with

3. Everyone is looking at the general state of the world and the power-holders in our society and expecting the issues of #2 to greatly overpower the benefits of #1

I’d definitely agree with that assessment, and it’s in that assessment that I’m as motivated and energized as ever to keep building DAAF into something that can help contribute to getting us #1 while mitigating #2. As I say in my “Peer Review is Dead; Long Live Peer Review” article: we aren’t just passive observers here, and we have the opportunity to make the change we want to see. It’s a wild, upsetting time, but it feels good to at least try and channel that into something useful for others.

And thanks to all that nervous energy, DAAF v2.0.0 is a straight-up rebuild of DAAF; where v1.0.0 was functional, there were a LOT of meaningful limitations to it. It was highly focused on the Full Pipeline mode -- which, while very neat and a strong stress-test for its overall capabilities, was FAR heavier (and also WAY more token intensive) than most users would be comfortable with for using AI in their current workflows. Moreover, this focus on getting the Full Pipeline figured out was at the cost of really fleshing out how DAAF thinks about data and work. The basics were there, but ask it to do anything like a difference-in-differences, and you’re kind of out of luck (i.e., you’d end up relying on the generalist knowledge of vanilla Claude Code).

Key changes and updates for v2.0.0

DAAF v2.0.0 tackles these two problems head-on in a couple of really neat ways, in my opinion (and you can see the quick showcase above for some nice explanations of the new modes):

1. The Orchestrator system is rebuilt from the ground up by building out a comprehensive daaf-orchestrator Skill. This allows us to be way more token efficient, loading in precise workflow instructions at the right time, rather than trying to load comprehensive information about basically every workflow all at once. It took a little finagling to get it to load properly when interacting with the human (and, e.g., not being loaded by subagents doing work), but this was an enormous win allowing us to rapidly build out new engagement modes with relatively much less effort and risk (since all of them are now modularized and independent of one another, for the most part).

2. That work to modularize the orchestrator facilitated the rapid creation of several new engagement modes. This included:

   1. Upgrading the data-ingest process to a full-blown Data Onboarding mode with the level of care and rigor we really want when profiling new data sources (as high quality data profiling/Skills is the fundamental crux on which basically every other feature of DAAF hinges)

   2. Building out a much ligher-weight and more collaborative Ad Hoc Collaboration mode for those who want the flexibility of vibe-coding with the data smarts and care of DAAF

   3. Setting up a Reproducibility Verification mode that allows users to use DAAF to rapidly re-verify past projects from DAAF (or from other colleagues using DAAF!)

   4. And -- my personal favorite and most used -- the Framework Development mode, which allows users to make extensions and core adjustments to DAAF in a guided and highly rigorous way. This last one is huge: it makes it so that anyone should be able to extend and customize DAAF to their own liking, without necessarily needing the deep level of expertise and battle-knowledge of being deep in the guts of it all the time. Not surprisingly, I used it to build almost everything else in DAAF v2.0.0.

3. Framework Development mode allowed me to also rapidly develop new toolsets and Skills for DAAF to use across workflow modes. For example, the data-scientist skill providing data analysis expertise and thoughtfulness to basically every agent in the system now has a deep, deep roster of extended expertise it can pull upon as-needed for various methodological tasks: geospatial analysis, difference-in-differences designs, regression discontinuity, fixed/random/mixed effects models, complex survey analysis, and WAY WAY more. This is in addition to some really nice utility/support Skills like science-communication best practices, helping users translate between R/tidyverse and Python and also Stata and Python, and really nice incorporation of new best practices on AI use disclosure (GUIDE-LLM). The below is my absolute favorite scene in the new showcase demo, because it is REALLY overwhelming to think about how much more DAAF can do well now.

There’s much more to say across the board (improved logging, improved safety, improved environment reproducibility, better and safer Hook management), but I think those will be some of the core things people should be really excited about and will immediately notice on using DAAF since version v1.0.0. If you’re really interested in the nitty-gritty, you can see a full changelog on the GitHub here.

I can’t emphasize enough: It’s a different animal now, and I would LOVE to hear how people who are both new and not new to DAAF take to the new changes and capabilities to do actually useful work from here.

Other work updates

As part of this launch for v2.0.0, it’s probably prudent to share three bits of news:

• I recently left my job as Director of Data Science at Common App to push on DAAF and related projects to better support public-interest organizations, non-profits, and researchers working in pursuit of the common good in using AI technology responsibly via my new venture, Open Augments. It’s a big risk in some ways, but I’m betting on myself and utterly convinced of the urgent value of being able to provide more educational resources and support in this space full-time!

• As part of that identified need, I’m spinning up a new Community of Practice via Discord (it’s basically just Slack) for like-minded researchers wanting to explore this frontier of AI for Responsible and Rigorous Research. My dream is to bring together people of all disciplines, with all sorts of experience and perspectives, to help us hammer out these really critical and pressing questions of how we wrangle AI for something truly useful for science and society more broadly.

• Lastly, this Thursday (April 2), I’m giving a webinar introducing peer researchers to DAAF through the Association of Education Finance and Policy. You can register here -- it is free and fully open to the public! It’ll be recorded and shared here afterward for anyone who can’t make it live.

Phew, I think that’s everything for now. Truly and deeply excited to hear what people think of DAAF v2.0.0, and equally excited to hopefully see some of you in the new community Discord to talk about DAAF and all other things percolating through our various corners of the research world.

Thanks to everyone who’s been along for the ride since the early days; it’s meant a lot!! You know where to find me for more, until next time!

What can we be hopeful for in the age of AI?

Much is being discussed about the tectonic changes coming to academia, journal publishing, peer review, and scientific institutions writ large as a result of AI technological progress. While I don’t always agree with the fearmongering-adjacent framing of some of these articles, I think their urgency and the conversations they’re starting are absolutely necessary given the rapid rate of progress; it really does feel like you’ll find yourself falling behind if you so much as blink lately.

It’s maybe no surprise then that many articles in this conversation to date have been focused on what you stand to lose if you don’t get on board, especially if things continue as unchecked and uncontrolled as they currently feel. I’d like to add to this conversation by instead presenting what I think we collectively stand to gain, and the brighter future we can all collaboratively push towards, if we keep a crystal clear focus on our core values and our actual purpose as a scientific community.

To that end, I’m going to describe a roadmap of six concrete steps we can push towards that together could unlock a more hopeful AI-empowered research paradigm: one where we can rigorously deepen, refine, share, revise, and make actionable our collective understanding of the world at a pace that we can barely even dream of today. One that sheds many of the deeply-entrenched issues with our existing scientific and academic processes while allowing us to bring rigorous new knowledge to the people who could use it most at near-lightspeed. One where genuine scientific inquiry can finally match the pace of and combat the misinformation crisis spurred deeper by this very same AI revolution.

At the risk of sounding overly optimistic, I believe you’ll see a place for yourself in this scientific community of the future -- because I believe this new paradigm will make the expertise that you’ve cultivated in yourself and others more valuable than it has ever been for society and our progress as a civilization.

Importantly, I am not attempting to predict some likely future, but instead trying to envision one possible future that should excite you rather than frighten you. And if you are reading this, I want to emphasize that you are not a passive observer in these branching pathways yet to come -- you can help us get there. My hope is to call colleagues to action to be a part of this conversation, and moreover spur critical alignment in the scientific community to set high standards and expectations for anyone seeking to disrupt the broader research ecosystem with AI in this age of grifters and opportunists.

To be clear, you really need to be skeptical of anyone just vibing out with big crazy ideas about AI right now. I’ve been thinking about machine learning and data science methodologies for social science since about 2018 as a grad student, and went on to finish my Ph.D. dissertation largely focused on developing careful ways to apply LLM-related methodologies to education data back in 2022. I’ve continued to work on that frontier inside and outside of academia ever since, and more recently, I have been working basically non-stop since December developing what I see as one stab at the pivotal first step on this roadmap (more on that in a second). You should definitely still take everything I’m saying with an enormously large grain of salt. These are big speculations that need be interrogated by people with more varied perspectives and areas of expertise (especially those outside the social sciences!). Please bring your ideas, and absolutely push back and be critical of myself and any others entering this arena.

This is a big one, so with that context and preamble, let’s get started.

Table of Contents

• Building the foundation

  • Step 1. AI-empowered research tooling: AI allows us to rapidly accelerate the production of many scientific claims, and we must center transparency and reproducibility as core requirements

  • Step 2. AI-empowered reproducibility verification: AI allows us to rapidly verify the mechanical reproducibility of many claims, and we must modernize journals accordingly

  • Step 3. AI-empowered validity assessment: AI allows us to rapidly critique the appropriateness and rigor of a given claim, and we must make these tools broadly and freely accessible

• Creating the new paradigm

  • Step 4. AI-empowered research dissemination: Leverage AI to rapidly disseminate a wider variety of rigorous, reproducible, and validity-assessed claims to the public when and where it’s most relevant

  • Step 5. Human expertise where it is needed most: Refocus trusted academic institutions on becoming hubs of verification for complex research insights and research using private data sources

  • Step 6. Human effort where it is most valuable: Revalue rigorous data collection, robust data infrastructure, and methodological refinements as the highest-impact activities for advancing scientific knowledge

• The end of the beginning: Instantaneous knowledge generation/verification and exponential scientific advancement for the public good with all hands on deck

Building the foundation

Step 1. AI-empowered research tooling: AI allows us to rapidly accelerate the production of many scientific claims, and we must center transparency and reproducibility as core requirements

With the advent of Claude Opus 4.5 in November, we’ve seen an absolute explosion in the rate of capability progression for AI across domains. What does this look like in the broader field of scientific research, and how do we manage the inherent risks?

Where we are today

Skepticism about this is absolutely warranted, but I do think that on balance, we will continue to see rapidly expanding value for using AI to accelerate research pipelines on multiple frontiers when applied rigorously and carefully (if you find yourself highly doubtful of the possibilities here, I recommend my companion piece on why no one can agree on AI progress right now).

For now, we should expect that this sort of progress is jagged but inexorable: some domains of work can have large chunks of their research pipeline drastically accelerated today (e.g., quantitative data analysis using publicly available data), while others are going to see less of their work conducive to this degree of acceleration for longer (e.g., field work, archival studies, qualitative interviewing, etc. Also highly dependent on context-specific data protections/privacy, legal and bureaucratic restrictions, etc.). Think of this as realistically the piecemeal acceleration of specific tasks with varying degrees of improved efficiency, rather than wholesale automation (which I think we should be extremely skeptical of!). The human effort/time saved from tasks that can be accelerated gets refocused on tasks that cannot.

People are today broadly experimenting publicly and privately, producing tools and frameworks of all kinds across disciplines for people to test, react to, share, and build on as awareness of AI-assisted opportunities grow. In the meantime, there’s a lot of promise, a lot of worry, a lot of variability in quality, and a lot of disagreement.

Where we need to go

The rate of production for research claims on those frontiers initially more amenable to AI acceleration will increase at first steadily (initial tooling uptake) and then exponentially (broad tooling advancement interacting multiplicatively with frontier model advancement). At the same time, quality of research becomes substantially more variable: from outright hollow slop to novel new research that expands far past what was previously ever feasible. The former is inevitably orders of magnitude more common relative to the latter. As these advancements continue, more tasks will become amenable to AI acceleration, and more fields will begin joining that rollercoaster of trying to navigate the growing quagmire.

Left unchecked, this would begin part of the academic death spiral that people are concerned about: Research production (with increasingly wide variation in quality) is likely to scale significantly faster than existing systems can filter and review it, causing major bottleneck issues for the publishing pipeline. We simply will not have enough reviewers or expert time to go around, quality and timeliness across the board will suffer, and trust in our institutions will wane.

With that in mind, we cannot leave this unchecked.

Transparency and reproducibility need to become core requirements for any and all research produced with AI assistance. As a scientific community, we need to align on these standards and practices immediately by setting strict norms and expectations for both tool creators and users. This buys us four key victories in the short term:

1. LLM-based AI will always be susceptible to hallucinations; enforcing true transparency and reproducibility will both stem the tide of low-quality slop being produced (since users can better self-verify) and make it far easier to assess and hold people accountable for their work in the current systems (since this immediately supports efficiency for current review systems).

2. By enforcing auditability in all AI systems, we improve our collective ability to learn more about their current failure states, appropriate and inappropriate uses, and exactly how to harness them more effectively going forward as a scientific community. Our intuition for using these tools well will improve commensurately.

3. With greater auditability, we also improve AI’s own ability to self-monitor, self-correct, and self-improve iteratively. This allows us to automate the enforcement of higher levels of rigor for anything coming out of these systems (but, importantly, still cannot eliminate hallucinations).

4. Stricter alignment around the standards we hold for AI tools and frameworks will allow us to demand and develop higher-quality tools, sooner, further reducing the issue of slop per se while pushing the capability frontier past this informal/awkward experimentation phase.

This era of LLM-based AI really feels as if someone’s just dropped a series of power tools into the middle of a playground; every minute that goes by without someone stepping forward with actual recommendations for responsible use invites more and more potential collateral damage in the form of misinformation and pseudoscience. Even worse, it deepens misunderstandings and disagreements about how these AI tools can and should bring value to our disciplines, because there are dangerous AI tools being peddled to the public in the same venues as legitimately useful and careful AI tools.

This urgency is exactly why I have spent so much effort developing DAAF fully open-source (forever free, I promise I am not trying to sell you anything!) with enormous amounts of explanatory materials (perhaps to a fault) -- it is equal parts AI-empowered research tool useful for accelerating many quantitative data analyses right now, and genuine philosophical missive trying to firmly establish these standards of transparency, rigor, and reproducibility in practice for others to react to and iterate on. I don’t know if what I’ve put together is the full set of everything we need, but it’s at the very least a strong and useful starting point for the discussion.

Whether or not DAAF becomes a tool the research community uses more broadly, we collectively need to have this conversation and establish a minimum bar for what we can and should be demanding of people attempting to develop tools and disrupt in this space. These standards need to then also be applied consistently and vocally to users as well. We cannot accept anything less than full reproducibility and auditability from researchers leveraging AI tooling in their work; we must hold each other accountable to high standards of rigor and responsibility as peer researchers as we always do. To facilitate that accountability, we also need to redefine academic honesty to include proactive AI tool use disclosure.

These norms, together, can drastically reduce the chaos and harms of this experimental/exploratory phase of AI tooling uptake. Institutions, journals, professional organizations, and grant-funders must collectively lead the way here across communities as quickly as possible; this work can and should begin now.

Looking ahead: many AI-empowered research tools already exist today, and they are collectively the worst they will ever be. Even if we align around the stringent requirements of transparency, rigor, and reproducibility, the massive spike in research production facilitated by this tooling will still ultimately continue to accelerate and broaden across tasks/fields until our traditional systems for research review and publication choke.

Step 2. AI-empowered reproducibility verification: AI allows us to rapidly verify the mechanical reproducibility of many claims, and we must modernize journals accordingly

If better tooling will make it nigh costless for a researcher to produce enormous variations of analyses using the vast libraries of available public datasets (alone and in combination with one another), how could our current systems possibly keep pace?

Where we are today

Step 2 hopefully makes obvious the necessity of centering reproducibility in Step 1. If novel analyses facilitated by AI are, by design and by default, always required to be fully reproducible (which becomes a powerful initial filter for journals to use on its own), then it becomes trivial to also facilitate the mechanical verification of any submitted research results accelerated with AI. This will ultimately serve as the next line of defense against full-on waves of AI slop research. And to be clear, this is not hypothetical: the early stages of this systematizing are happening right now with researchers building tools to formally and fully verify the reproducibility of research results at scale.

This core requirement of reproducibility allows us to immediately and automatically answer the question of, “are we able to replicate this set of findings?” But we can actually go one enormous step further: we can automatically verify the full, reasonable possibility space of statistical specifications for a given analysis and thoroughly investigate robustness and alternative analyses as a matter of course. In this future where the marginal cost of exploring an alternative specification is nearly zero for both the authors and the reviewers, p-hacking may very well become a thing of the past. This is also not hypothetical; we have an open-source approach for this style of work today, released literally as I was writing this post:

Where we need to go

These types of tools together, when combined with a standard of strict reproducibility for all AI-empowered research, are what will give our existing systems a fighting chance in the near-term. The community alignment around tooling produced in Step 1 should meaningfully reduce, but cannot eliminate, the amount of slop making its way to the doors of journals (i.e., by improving the standards of production); the tools surfaced here in Step 2 then facilitate enhanced filtering through the massively expanded inflows of research submissions while maintaining a strong level of quality -- perhaps even higher than it would be otherwise.

While these proofs of concept give us a hopeful start, it’s a completely different ballgame to operationalize these expectations and verification processes at scale where the work actually happens. Most journals, academic institutions, conferences, and research institutions seem woefully underprepared for navigating the initial conversation about AI-empowered research tooling (Step 1), let alone the downstream need to receive said research products in equal AI-empowered measure.

This is an arms race with the quality and sustainability of our research publications at stake. Those who are able to influence these entities to modernize and build appropriate infrastructure should be doing everything they can right now to start moving that machinery even if still imperfect; this is admittedly far outside of my experience or wheelhouse, so I’ll have to let others provide more insight into what that might look like (see, e.g., Scott Cunningham’s “A Modest Proposal for Editors” post). But I suspect there are a lot of very difficult conversations to be had here around incentives, institutional inertia, prestige signaling, academic politics, and funding.

Step 3. AI-empowered validity assessment: AI allows us to rapidly critique the appropriateness and rigor of a given claim, and we must make these tools broadly and freely accessible

Now even if we can get things moving on that automation front, verifying reproducibility and building in safeguards against p-hacking are not the same as assessing empirical rigor and validity. How can we address that more directly?

Where we are today

Alongside Steps 1 and 2, this Step is the third foundational pillar on which the rest of this roadmap relies. We know that research is not just some simple data-in-statistic-out pipeline of truth; it is critical to understand the nuances of the data, build on what we’ve learned in the past, recognize the necessity of different methodological approaches, and be thorough and thoughtful about what interpretations or conclusions are actually supported by the given results/analyses. Something can be technically reproducible while flying in the face of all of these fundamental tenets of good science; while reproducibility is necessary, it is alone insufficient. We need to be able to also assess rigor and validity at scale.

You’ve probably caught on by now: we’ve already got multiple private (1 and 2) and open-source projects on the market today designed to help scale and automate peer review with AI assistance. Yet another open-source project was also posted as I was drafting this (I swear I’m writing as fast as I can!!):

If you’ve ever been burned by an unfair/unqualified/vindictive/generally-grumpy human reviewer (and who hasn’t?), you probably wouldn’t fight the claim that these AI reviewers will quickly be able to do a better job along certain vectors than some human reviewers (interesting new benchmark from one of the private options, Reviewer 3, here). And with far more favorable turn-around times! Based on some of the recent high-profile public praise for Refine.ink, there’s no reason to think this quality frontier won’t continue to advance quickly and present meaningful value-add for more and more fields across the board.

There is ample discussion today about how best to leverage these tools, with a lot of possibilities on the table. Do journals operationalize a low-bar AI review as a pre-filter before human peer review? Do journals employ an AI reviewer as a third/fourth reviewer in tandem with the typical human reviewers? Do we use AI reviewers to contextualize/refine/summarize the reviews of human reviewers?

Where we need to go

Yes, journals should be exploring these steps, and I’d also suggest that there’s an especially enormous value-add for pre-print archives like arXiv to begin incorporating automated reviewing (not necessarily as a filter, but as a contextualizing companion for posted articles). Making these tools better, more comprehensive, more aligned with scientific judgment and best practices, and free will be absolutely critical to any future for research.

But I ultimately do not think AI-assisted peer review is sufficient to handle the coming tide of AI-accelerated research production beyond the very short-term, even with the reproducibility requirements (Step 1) and verification filters (Step 2) in place. There needs to be a much more meaningful shift in how we conceptualize the role of the journal and of research publication writ large.

In the current academic system, keep in mind that peer review is only one way to reach the actual goal of maintaining high research quality and ensuring that a finding is situated in its proper context/hedging/cautions. I would also argue that peer review is often actually insufficient at accomplishing this goal. Great journals have a roster of great reviewers that assure higher consistency, but a paper’s ultimate standing after peer review is at least partially a function of who the editor is, who the reviewers are, and (given how rarely peer review is truly blinded in a world with increasingly common pre-print sharing and small conference circuits) the authors themselves. Far, far from a purely objective quality function.

Moreover, the current system for peer review often entirely obscures the revision process alongside any concerns that were left unaddressed regardless of their validity -- especially if a paper was shopped around to multiple journals with multiple rounds of disconnected peer review. Forget the fact that the average lay person has no idea about the hierarchy of journal quality for a given field, so “peer reviewed” is a noisy and often ill-applied indicator of research reliability even in the best of times.

All to say: If the true goal is to help end-consumers of research insights understand what research we can/should trust and how to reasonably apply it, attempting to simply band-aid over an already breaking peer review system seems like a fool’s errand.

So what comes next? Systems-level augmentation.

Creating the new paradigm

Step 4. AI-empowered research dissemination: Leverage AI to rapidly disseminate a wider variety of rigorous, reproducible, and validity-assessed claims to the public when and where it’s most relevant

We need something far more drastic for a long-term response to fix the publishing game. Steps 4, 5, and 6 together represent my hope for what comes next; we’re getting much more speculative from here, so buckle up and ready your healthy skepticism.

Zooming out for a second, Steps 1, 2, and 3 taken to their logical ends will give rise to a research production environment where:

• Researchers can produce a basically limitless array of quantitative claims using publicly available data, almost instantaneously and costlessly

• Said claims can likewise be almost instantaneously and costlessly checked for reproducibility, examined within the full possibility space of alternative specifications, and then contextualized with automated assessments for appropriateness and methodological rigor

You might say, “Well, this is going to be true for simple descriptive statistics, but it certainly won’t be true for highly complex causal or theoretical or qualitative or [x] work.” A bigger question to untangle, but I’ll grant it: we need only scope to simple descriptive statistics from public data for now anyway. I’d argue that the vast majority of scientific insight is reflected in these simple, descriptive, atomic claims, because underpinning any sufficiently complex paper with causal merit or new theoretical frameworks or [x] are hundreds (even thousands?) of these atomic descriptives.

And for said atomic descriptive statistics, how much value is actually added by human reviewers and the explicit approval of a given high-profile journal? I’d argue, if anything, we are actively destroying an immense amount of value to the public through current review systems.

Think: how many of these atomic statistics are rendered fundamentally inaccessible and unactionable to the public by burying them, as we currently do, in tables and appendices and figures and footnotes and motivation -- all in service to a single fancy, well-identified causal statistic? E.g., how many valuable year-by-year state level metrics are buried in any given difference-in-differences analysis? How many useful distributional facts about a running variable are lost to the appendices of a regression discontinuity paper? What golden insights about implementation fidelity and sample attrition are hidden in the robustness checks of a good field experiment?

And how many authors have been told their causal merit was found lacking, resulting in these hundreds of perfectly good and useful atomic statistics never once seeing the light of day? How many low-hanging descriptive projects have never even gotten off the ground because authors know well enough not to waste their time or resources on something that’s not going to be perceived by journals as “rigorous” or “impactful” or “relevant”? Think of how many such atomic descriptive stats have helped us understand our world and society but would never get published in an academic journal because they’re too simple (e.g., the vast majority of U.S. counties had higher republican vote shares in the 2024 presidential election versus 2020, one of my all-time favorite data visualizations).

How much useful knowledge are we leaving on the table? And for what?

Split the publication pipeline

If tasks that can be accelerated with AI will allow humans to shift their focus to tasks that cannot, our institutions should optimize similarly; we will all be far better for it.

That in mind, I think we’ll soon arrive at a new equilibrium with two separate pipelines for knowledge production and curation: one for the set of atomic, verifiable claims that we can produce at scale, and another for basically anything else that’s not conducive to the scaling/automation of the first. For now, let’s focus on that first pipeline; Step 5 will focus on the second.

With the tools available today, and certainly with the tools available a year or two from now, I envision an “open knowledge repository” for research publication where:

1. Researchers submit their atomic claims and code to a public repository of knowledge, wherein every single claim is subjected to the standard protocols of reproducibility verification and validity assessment.

   1. Said claims, once processed and ingested, can be situated in a vast knowledge graph linking it to related claims by topic, data source, methodology, findings, implications, and more.

2. Users can query this repository like a modern-day oracle for any research question, any topic of discussion, and any issue of the moment.

   1. On a query, the repository will trace the knowledge graph to automatically surface relevant atomic claims situated within the context and hedging and caveats tagged by the automated validity assessments, and interpreted accordingly.

   2. Said interpretation/synthesis will be frictionlessly and costlessly subjected to the same rigorous validity assessments as the underlying statistics themselves.

3. Every atomic statistic “node” generates a public tracelog for every time it is referenced by the querying system to better track usage and public utility.

   1. Repository systems actively scan and track mentions of specific statistics across news outlets, policy briefs and documents, and social media discussions to track the downstream use of any given statistic -- as well as the utility of upstream data sources, foundational literature, underlying methodologies, and more.

   2. Repository stewards facilitate highly accessible querying via multiple means. The same way X users can frictionlessly tag Grok to invoke general AI assistance (e.g., “@Grok tell me what this means” for better and for worse), repository stewards make the system queryable broadly where the conversations are actually happening and related decisions are actually being made on a policy and personal level.

Combined, this system allows for lightning fast knowledge generation and iteration on an honestly enormous number of frontiers, trackable metrics of genuine societal value and impact to help incentivize researchers producing inputs to said statistics, and the potential to intervene on and combat misinformation across public discourse more broadly. Fundamentally, this system democratizes access to and use of rigorous, verifiable information and scientific insights in a way that no iteration of our current system would ever come close to accomplishing on its own.

Who will build this? It’s the right question, and while I think I see the ingredients of this sprinkled across the community already, this is absolutely no trivial undertaking. A project like this necessitates working through exceptionally difficult and complex questions about governance, attribution, incentives, adversarial gaming, defining rigor, iterative revision, and dispute adjudication. For now, call it a wish, but I suspect many will agree on the necessity of it given the future we’re heading towards. If the community can align here, I think the leaders and contributors will surface accordingly.

Step 5. Human expertise where it is needed most: Refocus trusted academic institutions on becoming hubs of verification for complex research insights and research using private data sources

So that said, what of the journals?

To be clear, this open knowledge repository is scoped to be intentionally narrow in two ways. First, it only works for sufficiently atomic claims conducive to automated reproducibility verification and validity assessment. Second, it only works for claims that can be verified using publicly available data. Neither of these are trivial limitations.

The current journal publication system thus stands to gain an immense amount by quickly refocusing their efforts towards specializing in these two arenas.

By offloading the brunt of AI-accelerated claims and studies to the automation of the open knowledge repository, the existing institution of peer review is given a new lease on life. Qualitative work, theoretical contributions, more complex methodological developments, historical perspectives and syntheses -- anything that cannot be mechanically verified still has a home in the journal publication space where some version of peer review and more intensive/manual vetting for contribution and quality still exist.

Moreover, the existence of this open knowledge repository actually makes the task of reviewing certain complex research insights more tractable as well. For example, rather than ask reviewers to assess and pore over a multi-staged argument with layers of atomic descriptive statistics and prior literature (which each may have their own issues with data provenance, sourcing, rigor, etc.) building towards the argument of validity for a single unique argument/claim, those atomic descriptive statistics and accompanying verifications can now just be referenced as a simple matter of public record. The reviewing focuses entirely on validating a paper’s unique argument and contribution in context to speed up and simplify the peer review task considerably. In this and many other ways, I expect the format and scope of the academic paper will be amenable to quite a bit more streamlining in this future.

As for the other key problem for journals and institutions to tackle, consider that the mechanical verification and assessment of research as discussed works only insofar as the datasets used to produce them are themselves valid and trustworthy. The potential for data manipulation throws all the confidence produced by reproducibility and anti-p-hacking and validity assessments out the window, which means that these AI-empowered tools become completely ineffective in the face of privacy-protected or proprietary data that otherwise can’t be shared.

This is where our existing journal publications and institutions can leverage their legal resources and reputational trust to great effect. Journals can engage in highly restrictive and protective data use agreements with researchers leveraging private data sources for the strict purposes of data validation and then downstream research insight reproduction/validation, in a way that is simply not feasible for a public repository on multiple levels.

This enables them to serve as the sole verifiers and validators for any research with these datasets, cordoning off valuable insights under their banner and brand. If we think of the open knowledge repository as the core layer and foundation of research insights, an individual journal can then present a distinct extension layer on top of it. Said journal would produce their own knowledge graph populated with both carefully vetted and peer reviewed knowledge that isn’t conducive to automatic verification/assessment, and any knowledge derived from private datasets.

There are a lot of potential ecosystem ramifications to this two-tier system, especially when it comes to journal sustainability, public access to knowledge, public use of knowledge, and more. But I suspect journals will want to compete for the most valuable complex research insights and most uniquely valuable private data ecosystem additions -- which honestly is probably not all that far off from their current role today. The more of these net-new nodes of insight they generate on top of the existing open knowledge graph, the more viable it becomes for them to serve up their own information querying services to the public for a fee or for prestige. Journals are also highly incentivized to maintain high standards of quality and rigor to cultivate trust, such that the public respects and understands the validity of evidence from their knowledge graph despite a necessary lack of transparency.

I won’t go too far down this road, but the competition here is likely to be fierce and, ultimately, quite good for everyone. I don’t love the necessity of restricting access to enormous tranches of knowledge, but I’d argue this is still a meaningful net improvement on the nightmarish accessibility and utility issues of modern-day journals given the automated querying/synthesizing of research insights made possible here.

Step 6. Human effort where it is most valuable: Revalue rigorous data collection, robust data infrastructure, and methodological refinements as the highest-impact activities for advancing scientific knowledge

Given Steps 1-5, we will have a paradigm where one of the primary limiters to our generation of new knowledge and insight is the availability of data to analyze. And maybe most optimistically: every new data source brought into the public arena is now multiplicatively more useful. Researchers have every incentive to mine any new data for new insights as quickly as possible -- slam it up against existing public datasets, slam it up against private datasets, update earlier analyses that are now outdated, go crazy. Traversing and reporting back on the complete surface of all analyses and possible specifications is a matter of course in this new world, and that process will only happen faster and faster with the underlying improvements to core AI technology. If all of this can happen with relatively little human intervention, where does this leave the frontier of work for most human labor in the research sphere?

In this new paradigm, collecting that data and sharing it out becomes one of the highest-value opportunities for any researcher, especially if the open knowledge graph allows for proper and immediate attribution to the data providers. The sexiest arenas of work in this future? Lab experiments, field experiments, new sensor/measure creation, qualitative data collection, surveying, archival data collection/aggregation, and so on. And as anyone in those fields can tell you right away, there will always be more than enough work to go around even if AI tooling on these frontiers expand. Data collection is a fractal: the more we gather, the more we know we don’t know.

This is, I hope, how we transcend the “looking for your keys under the lamppost” problem inherent in today’s work. When collecting new data is so strongly incentivized because of its readily attributable value and impact, what does that unlock for us in terms of valuing data collection for new populations, geographies, topics, and approaches that our current system actively disincentivizes?

Going one step further: ensuring high quality data collection and rigorous documentation of data provenance/methodology will be seen as entirely non-negotiable. With this paradigm, the frontier of progress is not just new nodes, but also improving existing nodes (and thus all nodes that exist downstream from them). In a world where every public dataset can and will be scrutinized from every possible angle, issues and concerns will be laid bare in multiple ways and percolate downstream into substantial hedging and validation issues. Collectively, we know that garbage in = garbage out, and no efforts or expenses should be spared to ensure that the inputs of our new scientific community are as manicured and documented as possible.

To the extent that these are unavoidable, errors and issues found in data provenance will be able to percolate almost immediately through the graph, and that leaves an arena of opportunity for researchers to iterate on and update affected/stale nodes. Methodologists across the board will thus also have plenty to do, as the open knowledge graph should similarly make it easy to understand how such skills and expertise are absolutely fundamental to the downstream knowledge generated. New and better methodologies, approaches, and critiques create a similar arena of opportunity for those early appliers of said techniques to update existing nodes.

And the sexiest institutions in this future? The open knowledge graph tracing and telemetry system will finally be able to offer the immense credit and flowers long overdue to public data curators and infrastructure providers. Basically everything in this system fundamentally hinges on having open, reliable, and free access to a wide and growing array of public datasets. Someone needs to keep those lights on and keep those trains running. The value to humanity that these infrastructural entities currently provide is enormous (e.g., IES and Urban Institute in my corner of the education world), yet often go under-acknowledged despite how many hundreds or thousands of studies leverage them for critical table-setting or assumption checking along the way to their fancier statistics. Investing/reinvesting in these trusted infrastructure providers will be absolutely pivotal for this future to be at all possible.

The end of the beginning: Instantaneous knowledge generation/verification and exponential scientific advancement for the public good with all hands on deck

Researchers come at the work from all different angles, but my suspicion is that we are all aligned on wanting to marshal the skills and expertise we’ve each cultivated over many years towards helping our broader society move forward and become better for more people than it is today.

I’m hoping that the prospects presented here are genuinely exciting to you from that perspective: a world where our work can be more useful, more accessible, more expansive, more interconnected, more inclusive, and -- I’d argue -- more rigorous. An academic system that can finally grow past the many fault lines and inefficiencies and injustices we all see and recognize but can’t figure out how to resolve as of yet.

I won’t lie to you: I truly have no idea how likely this future is. I have no idea if I’ve completely misread where some of these strands and roads lead. I can’t possibly know if this new scientific community even as described will actually have a place for everyone. Writing this post has taken a lot out of me (no AI, see disclosure in the footnote!), and the endless running down of possibilities and alternative scenarios has me feeling a bit like Doctor Strange in this moment:

But I am willing to be hopeful. I am willing to strive for it. I think there is reason to look forward to what we can build, if we do so together with eyes wide open and in honest collaboration with as many people at the table as possible. The doom and gloom -- while understandable given how truly wild all of this *gestures vaguely* is -- serves as a sinister, disempowering force. But because this will almost certainly be a painful, complex, and harrowing transition even if we play all of our cards right every step of the way, we need all hands on deck. I urge you not to give in and not to check out when there is so much intensive and delicate work to go around in trying to fully realize each Step as described above.

There is infinitely more to say on all these points, and infinitely more hedging and cautions and caveats I’d like to add. But at this point, I think I just need to open the doors for others to weigh in, react, refine what I’ve pieced together here, and start thinking of more ways we can arrive at a better future -- any better future than the one it feels like most people are currently anticipating. What here feels most realistic? What feels like fantasy? What steps am I missing? Where can we iterate, and where should we start over from scratch? Is this a future you see yourself and your colleagues in? Is this a future you even want to be a part of? Where do you think you could see yourself contributing to moving us forward? And then more concretely, what does the funding model for any of this look like? What of the next generation of researchers? What about skill formation and atrophy among the current generation? What are the ethics of boiling the seas for this vision of science? How do we avoid actually deepening the existing inequities of our system in this transition? It’s all endlessly complex; I simply can’t wait to hear what ideas people bring to the table to keep this conversation moving and growing.

These questions will be hard, but our future can still be bright if we cultivate critical awareness and alignment across the scientific community, push back against the AI grifters/hype-sellers, and actively choose to make it so on our terms.

Who knows what amazing things can come after that?

Further reading: If you want to hear more elaboration from me on a few related points, you might find some of my DAAF FAQs focused on philosophical questions valuable. If you’re not sure how to start diving in with everything that’s happening in AI progress right now, I recommend my more accessible companion post on Why no one can agree about AI progress right now. I have also linked about a million articles and posts throughout this article; I think I’ve presented only a handful of unique ideas here re: academia’s future, with most of my job really just being putting others’ thoughts together into a more unified roadmap. Please do take the time to dive into what I’ve collected here from folks like Alex Imas, Emily Bender and Alex Hanna, Solomon Messing and Joshua Tucker, Andy Hall, Margaret Mitchell, Jessica Hullman, Susan Bowles Therriault, Rachel Garrett, and Kyle Fagan, Sean Westwood, Scott Cunningham, Joshua Gans, Noah Smith, Alexander Kustov, and David Yanagizawa-Drott.

Disclosure/Acknowledgments: Not a single word of this was written by AI despite my many attempts to coax something useful out of it. It was helpful in suggesting a few tonal revisions, but not remotely as helpful as many amazing colleagues who were willing to step up and help on early drafts. Sorry Claude, you’re still just not there yet! Huge thank you to Erika Tyagi, Jae Yeon Kim, Preston Magouirk, Daniel Stone, Eric Giannella, Katharine Sadowski, Amanda Lu, and Cody Sigmon for their thoughtful engagement and feedback; any remaining errors and issues are my own. Also, big shoutout to Zara Zhang for the phenomenal frontend-slides skill that facilitated all the very nice mental model graphics throughout this and my other articles!

At time of writing, it feels like there are exactly two camps battling openly in the AI discourse right now:

1. People who believe that AI/LLMs are fundamentally flawed and can never truly be a threat to many/most types of human work and labor

2. People who believe we are only a handful of months away from full labor market collapse due to how rapidly AI/LLMs can now replace entire industries

You’re probably seeing it play out in your own professional circles and friends, and you might yourself strongly identify one way or the other. Why is there such a stark divide on what AI can and cannot do when we’re all roughly looking at and using the same things? Even within the very same fields that are allegedly being the most disrupted (e.g., software development), there are people at either pole claiming vehemently that the other side are ill-intentioned, bad-faith actors. Is it that one side is simply full of ignorant luddites too blinded by their own desperate self-preservation? Or is it that the other side is simply full of AI grifters too blinded by their own self-interest and dire need to keep the investor gravy-train rolling?

As I think more and more about what makes DAAF work, as well as why I think it has only recently become possible for a tool like DAAF to even exist (imo, circa the release of Opus 4.5 in November, 2025), I think I have an explanation that reveals why both sides are surprisingly more justified than we might initially expect: Too many people are thinking about AI progress as one singular measure focused on purported frontier model quality, when it is really three intertwined and interdependent measures.

Now, the model-specific focus is an understandable way to think about things: This is the visible frontier that every AI company touts, and that the average end-user tends to see and remember most readily. Opus 4 is now Opus 4.5 is now Opus 4.6! GPT4 to GPT5 to GPT5.2 and then GPT5.3! Gemini is now Gemini Pro and Gemini Flash and is better and more capable than ever! Wow! But this simplistic impression from the marketing is what’s leading us to the current state of the deeply disillusioned camp versus the overly-hyped camp.

While the raw model capability is absolutely a core component of how we should be thinking about AI progress, this is missing two additional dimensions that kind of change everything, at least in part because they’re currently pretty obscure to the regular person who’s only occasionally dropping into ChatGPT or NotebookLM as a casual user.

A better mental model: The Mind, the Body, the Instructions

A complete picture for understanding the frontier of AI progress, and what AI can actually do, requires then three total core pieces:

1. “The Mind”: Progress in base AI model capability. I.e., the big model advancements we see in the news and usually result in a model having more training data, thinking in more complex ways, and generally able to take in more contextual info before acting.

2. “The Body”: Progress in accompanying AI orchestration frameworks and tooling. I.e., infrastructural advancements allowing models to run code scripts at will, or search through provided files/the internet dynamically, or delegate a task to another fresh AI/LLM, or load up specific contextual expertise on demand. Claude Code and Cowork are enormous advancements over basic chat interfaces on this frontier.

3. “The Instructions”: Progress in user input and skill. I.e., how a person actually tries to explain their request to an LLM -- in terms of descriptiveness and process described in their original request, how they intervene for setbacks/revisions, and what baseline material references they point the LLM to.

Let me try and explain why these two other dimensions matter so much and what this all means together using a physical robotics metaphor. To wildly oversimplify it, imagine that you have a super-intelligent humanoid robot assistant in front of you, willing to do whatever it is you could possibly ask it to do. For the sake of discussion, and because I’m really fun and imaginative, let’s say that I want it to fold my laundry.

• If I give this super-intelligent robot a comprehensive tutorial of how to fold each and every piece of my laundry step-by-step (great instructions), but it only has simple Lego-style straight arms and claw hands (incapable body), it is still going to utterly fail at folding my laundry.

• Conversely, if this super-intelligent robot now has perfect human-like hands (capable body), but I just say, “Get to work!” while in another room without any further clarification at all (vague instructions) -- it’s probably not going to infer that I specifically want it to fold my laundry and fail at the intended task.

In this example, even if we have super-intelligence fixed, having even one other failing core component (incapable body or vague instructions) utterly crushes the chances of getting anything useful out of our robot friend. Each piece is necessary, but would alone be insufficient to attain any kind of success here.

This, I think, is the state of things with the AI/LLM assistants of today, and why people are so at odds about what AI/LLM assistants can and cannot do. To draw it out:

• If I have Opus 4.6 (great mind; one of the most capable models ever) try to summarize a new academic paper for me, but I don’t actually let it search the internet to find the paper I’m talking about (incapable body, no tool calls)… it’s more likely to desperately make something up from its fuzzy general “memory”, and I’m going to think it’s actually quite dumb and unreliable.

• If I have Opus 4.6 (great mind) working within Claude Code with access to the full library of all great poetry ever written (capable body), but I don’t actually tell it what kind of poets or poems I want to draw on for inspiration and why (bad instructions)… well, the odds that I eventually get a love poem I actually like out of that process are quite low.

In both cases, the media hype for Opus 4.6 land plainly false if not outright manipulative -- which would be fair given my experiences with it.

Wait, does this mean you’re saying I’m the problem here? The AI hype train is real?! Well, not exactly, because…

Everything is just way weirder than you think

So the far extremes are painfully obvious about ramifications, but to be clear: there’s a whole world of range within the “Body” and the “Instructions” dimensions that are far from a simple bad-versus-good binary. And what makes this even more complex is that what we even think is bad, good, better, and phenomenal on these two dimensions (body/instructions), are constantly shifting and changing day to day and even model release to model release. Getting better output out of an LLM is not nearly as straightforward as we might think; these things are way weirder and more idiosyncratic than that. For example:

• Best practices on frameworks/tooling (Body) are generally coalescing towards the idea that we want to give LLMs sufficient context to do a task… but also, it seems like there are a lot of ways to provide fundamental context to the LLM that end up making the LLM perform worse than providing no context at all. Why???

• It seems intuitive that providing thorough, comprehensive and well-structured prompts (Instructions) will produce better output from the LLM… but also, Anthropic now specifically recommend against being too thorough in your instructions because our instructions can actually be worse than Opus 4.6’s improvised plans for completing certain tasks. Oh…?

• It stands to reason that providing repetitive, redundant instructions in your prompt would cause issues for the LLM (Instructions)… but also, just copying-and-pasting your prompt twice in a row seems to uniformly increase many models’ performance. ?!?

• One of the superpowers of more refined AI agent platforms like Claude Code, Claude Cowork, Codex, and Antigravity is the ability to let the agent search over and reference a ton of different files (Body)… but also, if your documents are in PDF format instead of Markdown format, it might fundamentally misunderstand your documents and/or fully hallucinate content because it got confused during the file-read. Oye…

And we don’t even really know which of the above apply equally to Opus 4.6 versus Opus 4.5, let alone Opus 4.6 versus Codex 5.3. It’s absolute madness.

But for now, let’s just hold fixed the models themselves. Even in that state, what exactly it means to take best advantage of them via the frameworks and tooling and user prompting is extremely confusing and unclear and constantly developing. It’s hard enough to know how to talk to Claude -- how do you tell Claude how to talk to the subagent Claudes it launches? What does it mean when the subagents can now talk to each other? Getting the most out of the current frontier of AI models now means something totally and completely different than it did in October of 2025 because of these quickly fractalizing new complexities in orchestration and context management. There are just so many dimensions to optimizing the frameworks/tooling and user prompting side of things that people are simply not aware of nor privy to, and I cannot emphasize enough how quickly that field is developing as people are experimenting and learning more and more.

The information gradient drives the hype gradient

And it still gets weirder. This experimentation and skill-sharing is richest in random circles online as people report back field notes from their own varied experiences -- reddit forums, X threads, and GitHub repositories -- and the true frontier of best practice is no longer being generated by the traditional experts in this space. It’s coming from people who are tinkering on the weekends, pushing Claude Code into weird use cases and extremes, engaging in wildly extended 12-agent coding sessions with Codex, setting up OpenClaw to vibe-code away every problem in their life via WhatsApp, or letting Claude Cowork blast through producing slide deck after slide deck for a month’s worth of sales meetings.

This is why I think it’s been anyone’s game since Opus 4.5 launched in November; information is power in this paradigm because it produces extreme capability gradients from one person to the next. Foot_sniffer6467 on Reddit got to be king for a week back in January because they spontaneously uncovered an early way to get subagents to communicate on tasks synchronously while vibe-coding their first SaaS app (joke example, but versions of this are literally happening every day now).

So when it comes to the question of: What can AI do right now? It depends so completely on how much you’ve been paying attention to these weird developments in the space that fundamentally change your intuition for what you think good prompting means, and what the best practices are for helping Claude Code cook on a codebase, and etc. etc. Thus, what kind of output you’re really getting when you interact with LLMs today, and what you think is currently possible with AI, hinges accordingly. The range of inputs is enormous, so the range of outputs is now similarly enormous -- on one end, we’re seeing “Oh LLMs are too dumb to understand you need to bring your car to use a carwash”, and on the other end, we’re seeing LLMs make my PhD and years of data science experience feel less useful by the day. Almost everything in the space of Body/Instructions has changed in the last four months; blink and you have absolutely missed it.

This was one of the strangest and most disorienting experiences of working on DAAF in basically complete isolation until launching two weeks ago: as I got better and better at understanding how to work with Claude Code and all its idiosyncrasies over the course of a month (what the hell is a “Contract?” What do you mean I need to write an “Invocation pattern” for each subagent? Why do we keep calling things “Gates”???), I was suddenly seeing such a WILDLY different frontier for AI capability than anyone else I was talking to. There’s just no way that the average person casually asking Gemini to do deep research for their upcoming trip to Japan would think something like DAAF is remotely possible, and I absolutely do not blame them.

Okay, so, then… where are we really right now with AI progress?

Going back to the original two camps (pro-AI hype grifters and anti-AI luddites), I find myself uncomfortably more aligned with the AI hype grifters. As we unlock more understandings in the frameworks and tooling layer (Body), and as we individually advance on best practices for prompting (Instructions), we are unlocking multiplicatively more and more value out of the models in front of us. Meanwhile, advancements in the Body/Instructions are now directly contributing to faster improvements on the model frontier (Mind) with OpenAI's Codex famously accelerating its own development. These new models, in turn, will unlock more capabilities and frontiers for the Body/Instructions to advance; this flywheel is spinning, and it will only spin more aggressively from here. Based on how truly wild the last four months since Opus 4.5 alone have been for Body/Instructions, we are, I think, firmly in the “exponential” progress phase that people have been predicting.

That being said, I still think the AI-detractors are a particular flavor of correct, but perhaps for a reason they themselves won’t love: improvements in the frameworks and tooling (Body) and prompting (Instructions) are currently still too weird and confusing for the general public to get sufficiently good AI output quality on any sufficiently useful/complex task. I have been doing almost nothing but try to keep on top of this frontier over the past three months, and I still barely feel like I can synthesize what’s going on at any one time.

And as mentioned earlier: we can have the best AI models (Mind) out there in the world, but if everyone in the world remains, on average, terrible at prompting, we’re not really going to see the mass transformation of society that AI hypers are dreaming of and doomsaying about on the timelines they imagine. Of course, this is going to be an active field of development and progress of its own (i.e., how do we make it more frictionless to be great with LLMs? It is likely at least partially solvable), but I think the AI-hypers drastically underestimate how much inertia will need to be overcome to really start moving the needle on that overall user skill market (at least partially informed by my past life as a high school English teacher; teaching is very hard). They might argue the luddites don’t need to get on board for their jobs to nonetheless be automated away. I’m not so sure about that -- and until then, the true current frontier capabilities of AI that I and others are playing with might as well not exist to the vast majority of society.

What should I do from here? Use the mental model as a guide

I think one thing is extremely clear to me: People who can take advantage of the information gradients of this strange in-between time for AI adoption stand to gain an immense amount personally and professionally, whether we like it or not. Put a different way: I don’t think that there’s any benefit whatsoever to remaining tied the narrative that AI is/will be useless and all of this buzz is driven by self-interested hype. If you stay in that camp, you risk willfully missing a boat that doesn’t need to be missed, and it will get harder and harder to catch up as the frontiers of AI Body/Instructions best practices develop into more and more obtuse areas of literature. This is why I’ve put so much energy into education around DAAF (and this post, as one example!) -- it’s not enough, in my mind, to give people nifty AI tools they can use. They need to understand how to catch up and how to push ahead on their own. Teach a man to fish, sort of thing.

So what does that look like? Ultimately, my big brain plan here is that if you understand this Mind-Body-Instructions mental model, you already have a clear schematic for how to push your own AI output quality upward from here.

Start by building your prompting skill and intuition (Instructions): push yourself to use these assistants however you can. When it’s convenient, and when it’s not convenient. Understand their failure modes, and see if you can talk through them to find mitigation strategies. Pay attention to how your responses and the specific types of phrases you use shift its behavior. Be deliberate in testing out different communication styles and tones and vocabularies. Understand what seems to correlate with success, and test your understanding to really see if that correlation holds under different circumstances with different tasks or different models.

Then, start pushing on the frontiers of more advanced frameworks/tooling (Body): take the plunge into using something more comprehensive like Claude Cowork or Claude Code (personal bias: I really did design DAAF to be one of the fastest and safest ways to get started with Claude Code for newcomers!!). Once you’re in, start asking Claude how to get better at using Claude Code. Ask it exactly how and why Claude Code is different and more capable than the basic web chat interfaces by having it research its own online documentation materials. Get it to search online in these weird GitHub/Reddit/X communities for emerging best practices as of March 2026. Tell it to launch a subagent to do X or Y web search that informs a follow-up subagent’s task to do Z, and watch what gets communicated back and forth to you and between the agents. Point it to some big fancy repository you find from folks like Chris Blattman or new Claude Code product feature announcements from Boris Cherny and ask Claude what these things do, why they might be valuable, and how you might incorporate them into your own workflows. Use Claude to make your own skill-building Skill, and make new Skills inspired by others or tasks you actually do day-to-day. Your goal is to get better at harnessing these frameworks and tools to your own ends -- use whatever scaffolding and information you can find to that end, especially Claude itself.

The models (Mind) will progress forward inexorably from here, and people who cultivate their skills on Body/Instructions are going to reap more and more benefits from those advancements relative to people not pushing at all. I don’t want to sound like that other really dramatic guy about something big happening, but I don’t think he’s wrong. It feels like a really pivotal time, and I’m hoping we can get as many good, thoughtful people armed with the critical awareness and skills they need to flourish in this weird future as possible.

There’s much more to say, but that’s all I think makes sense to try and fit into one already very long post. I hope this mental model explains the core weirdness of the current discourse to help people stop talking past each other, and I hope it moreover provides an actionable way for people to get themselves off the sidelines of this increasingly critical frontier. If you’ve found it useful from either perspective, I hope you’ll share this post with people you care about to help bring them up to speed, too.

Experiment, push into weird and uncomfortable places, ask good questions, share things freely with one another, and don’t lose sight of the ways this newfound capability can and should be used to help others.

You know where to find me if you want more ideas. Much more to come! Good luck out there, until next time.

I launched the Data Analyst Augmentation Framework last week with 40+ education datasets from the Urban Institute Education Data Portal as its main demo out-of-the-box, but I purposefully designed its architecture to allow anyone to bring in and analyze their own data with almost zero friction.

In my newest video, I run through the complete process of teaching DAAF how to use election data from the MIT Election Data and Science Lab (via Harvard Dataverse) to almost perfectly recreate one of my favorite data visualizations of all time: the NYTimes "red shift" map tracking county-level vote swings from 2020 to 2024. In less than 10 minutes of active engagement and only a few quick revision suggestions, I'm left with:

• A shockingly faithful recreation of the NYTimes visualization, both static *and* interactive versions

• An in-depth research memo describing the analytic process, its limitations, key learnings, and important interpretation caveats

• A fully auditable and reproducible code pipeline for every step of the data processing and visualization work

• And, most exciting to me: A modular, self-improving data documentation reference "package" that allows anyone else using DAAF to analyze this dataset as if they've been working with it for years

This is what DAAF's extensible architecture was built to do -- facilitate the rapid but rigorous ingestion, analysis, and interpretation of *any* data from *any* field when guided by a skilled researcher. This is the community flywheel I’m hoping to cultivate: the more people using DAAF to ingest and analyze public datasets, the more multi-faceted and expansive DAAF's analytic capabilities become. We've got over 130 unique installs of DAAF as of this morning -- join the ecosystem and help build this inclusive community for rigorous, AI-empowered research!

If you haven’t heard of DAAF, learn more about my vision for DAAF, what makes DAAF different from other attempts to create LLM research assistants, what DAAF currently can and cannot do as of today, how you can get involved, and how you can get started with DAAF yourself at the GitHub repository.

Bonus: The Election data Skill is now part of the core DAAF repository. Go use it and play around with it yourself!!!

DAAF explicitly embraces the fact that LLM-based research assistants will never be perfect and can never be trusted as a matter of course. But by providing strict guardrails, enforcing best practices, and ensuring the highest levels of auditability possible, DAAF ensures that LLM research assistants can still be immensely valuable for critically-minded researchers capable of verifying and reviewing their work. In energetic and vocal opposition to deeply misguided attempts to replace human researchers, DAAF is intended to be a force-multiplying “exo-skeleton” for human researchers (i.e., firmly keeping humans-in-the-loop).

The base framework comes ready out-of-the-box to analyze any or all of the 40+ foundational public education datasets available via the Urban Institute Education Data Portal, and is readily extensible to new data domains and methodologies with a suite of built-in tools to ingest new data sources and craft new Skill files at will!

With DAAF, you can go from a research question to a shockingly nuanced research report with sections for key findings, data/methodology, and limitations, as well as bespoke data visualizations, with only five minutes of active engagement time, plus the necessary time to fully review and audit the results. To that crucial end of facilitating expert human validation, all projects come complete with a fully reproducible, documented analytic code pipeline and consolidated analytic notebooks for exploration. Then: request revisions, rethink measures, conduct new subanalyses, run robustness checks, and even add additional deliverables like interactive dashboards, policymaker-focused briefs, and more -- all with just a quick ask to Claude. And all of this can be done *in parallel* with multiple projects simultaneously.

By open-sourcing DAAF under the GNU LGPLv3 license as a forever-free and open and extensible framework, I hope to provide a foundational resource that the entire community of researchers and data scientists can use, learn from, and extend via critical conversations and collaboration together. By pairing DAAF with an intensive array of educational materials, tutorials, blog deep-dives, and videos via project documentation and the Open Augments Field Guide Substack (MUCH more to come!), I also hope to rapidly accelerate the readiness of the scientific community to genuinely and critically engage with AI disruption and transformation writ large.

I don’t want to oversell it: DAAF is far from perfect (much more on that in the full README!). But it is already extremely useful, and my intention is that this is the worst that DAAF will ever be from now on given the rapid pace of AI progress and (hopefully) community contributions from here. What will tools like this look like by the end of next month? End of the year? In two years? Opus 4.6 and Codex 5.3 came out literally as I was writing this! The implications of this frontier, in my view, are equal parts existentially terrifying and potentially utopic. With that in mind – more than anything – I just hope all of this work can somehow be useful for my many peers and colleagues trying to “catch up” to this rapidly developing (and extremely scary) frontier.

Learn more about my vision for DAAF, what makes DAAF different from other attempts to create LLM research assistants, what DAAF currently can and cannot do as of today, how you can get involved, and how you can get started with DAAF yourself!

Never used Claude Code? No idea where you’d even start? My full installation guide walks you through every step -- but hopefully this video shows how quick a full DAAF installation can be from start-to-finish. Just 3mins!

So there it is. I am absolutely as surprised and concerned as you are, believe me. With all that in mind, I would *love* to hear what you think, what your questions are, what you’re seeing if you try testing it out, and absolutely every single critical thought you’re willing to share, so we can learn on this frontier together. Thanks for reading and engaging earnestly!