This week my colleagues at Digital Science launched Papers AI. One research workspace for your writing, your data and your code, with an assistant that has the whole project in context rather than the paragraph you happened to paste into a chat window. I will come back to what it does. First I want to make the argument for why the shape of the tool matters more than the feature list, because I do not think we are having that argument properly.

The board is visible in chess. It is not visible in a lab.

There is an essay from Thinking Machines, The Future Worth Building Is Human, that makes the following distinction. There are domains where raw intelligence really is sufficient on its own, and where AI can race ahead without anyone in the loop. Chess is one. Mathematics is increasingly another. What those two have in common is that the goal is static and can be written down, and there is no hidden knowledge. The board is visible to everyone. Nothing that matters is sitting in someone’s head, unrecorded.

Research is not like that, and I think this is the thing that people building AI-for-science tools keep getting wrong. Most of what a lab knows is not in its papers. It is in the three approaches you already tried that did not work, which will never appear in any published record anywhere. It is in the reason you chose this cohort and not that one, and the argument you had with your collaborator about it. Michael Polanyi called this tacit knowledge, and Hayek’s argument about why central planning fails is the same argument: the knowledge that matters is local, provisional, held privately by the people who acquired it through doing the work, and it does not survive being aggregated into one place. Life science researchers have an enormous context window.

The warning Thinking Machines make is about AI that “extracts a snapshot of it and replaces it with a standard offering”. Apply that to a manuscript in progress and it stops being abstract. A model reading your half-finished draft is not reading a paper. It is reading the tacit layer, the bit that never gets published, at exactly the moment it is most exposed and least protected.

Show me the incentives

Regular readers will know this is where I always end up, so let me apply it to the AI labs rather than to researchers for once. If your product is a single frontier model rented to millions, then every piece of specialised knowledge you can pull into the weights makes the product better for everyone else, and the researcher who supplied it has handed over the one thing that made them worth hiring.

Prism launched on 27 January, built on Crixet, a cloud LaTeX platform OpenAI acquired. Jonathan Schaeffer, emeritus AI professor at Alberta, made a useful split when he spoke to Decrypt: there are two things happening when you write a paper, composing the text and doing the actual research. Tools like this help with the former, the drafting and the proofreading and the literature search, and that is real and useful.

The problems are everything around the writing.

As Schaeffer put it. Standard protocol is that when you write a paper you are documenting your own research and you own it. Route that through a large multinational’s model and, in his words, “you’re actually exposing your intellectual property to a multinational company”. Whether the provider would ever have a legal claim on what you produce inside their tool is, as he says, a question where the devil is in the details. By default, ChatGPT content, and therefore Prism content, may be used to train future models. Prism is now being folded into codex.

Who holds the line

I use frontier models a lot. I see it as a trade off between giving them my thoughts to train on and the incredible stuff I get back. I’m not an academic researcher and this may need more thought for them, because it is a decision about exactly where a line gets drawn across the research lifecycle, and the person drawing it is not the researcher. The 3 stages of the gif below illustrate what I mean.

The point of it is not that one of those three stages is objectively correct. I have my view, which is that the second one is where the trade-off is least bad, but I know serious people who would put it at the third and have good reasons.

Wherever the line goes, it should be the researcher who puts it there. Not a vendor, not a default, not a setting three menus deep that got flipped during an update you did not read. If a researcher decides their unpublished analysis is not ready for a model to learn from, that decision has to be enforceable, not merely requested.

Almost every tool in this space puts the line at stage one.

Papers AI moves the line into the architecture

Which brings me back to Papers AI, and to why I think its shape matters more than its feature list.

You can write in Word, LaTeX, Markdown or Typst. Real .docx with tracked changes and comments that survive conversion, which anyone who has collaborated across a Word-and-LaTeX divide will tell you is not a small thing. Your datasets and Jupyter notebooks live in the same project as the draft, so the analysis and the prose stop drifting apart. A reviewer asks you to re-run something, you update the cell, and the figure regenerates where the manuscript already is.

The assistant reads the whole project, and pulls citations from your own reference library, verifying them before insertion. Every change it makes appears as a reviewable diff. You accept or reject. The assistant’s edits are attributed and reversible, the same as a human collaborator’s.

And you can bring your own model. Use the built-in assistant, or point it at something local running on Ollama or LM Studio or any OpenAI-compatible endpoint. Files stay on your device. Compilers run in your browser. Collaboration happens when you invite it, project by project.

That last part is usually written up as a privacy feature. I think it is better understood as the answer to the question the slider asks.

Point the assistant at a model running on your own hardware and the embargo line stops being a policy and becomes a fact about where the electricity is going. Nothing has to be promised, because nothing left. Keep the files on the device and the compilers in the browser and the line holds even if every term of service on earth changes tomorrow. If the risk is that a centralised model absorbs what makes your lab distinctive, the mitigation is not a stronger assurance from the provider. It is not having sent it. You cannot extract a snapshot of a lab’s tacit knowledge from a model that never left the building.

Local inference is not exotic in 2026, and anyone could build this way. Building the version where the researcher keeps the line is a decision about who the tool is for, and it is the decision I would want made by whoever I trusted with an unpublished manuscript.

We ran a webinar on exactly this last week, AI Uncorked: guardrails, compliance and IP protection for the enterprise, with John Lees-Miller, who co-created Overleaf, alongside Mark Bloomfield and the ECB’s Maximilian Freier. The whole conversation was about the question underneath all of this. Not whether to adopt AI in research, because that ship has sailed, but how to do it without putting sensitive data, proprietary research and IP at risk. How to build guardrails so teams can use these tools with confidence. How to make sure that the research which is yours, stays yours.

In the last piece I got excited about a future where research outputs are forkable, verifiable, machine-readable objects that people and agents can build on at speed. I still want that future. But a forkable research object is only as trustworthy as the tools that generated it and the tools that check it. Reproducibility is not a nice-to-have bolted on at the end, it is the substrate. And if we are going to hand more of the writing, the citing and eventually the analysis to machines, then the tools doing that work have to verify before they assert, keep provenance attached, and leave ownership with the human whose name is on the paper.

In chess, intelligence alone gets you all the way, because there is nothing hidden. Life science research is not chess (yet). Maths and Physics will be solved long before wet lab research. Most of what a lab knows was never written down, and the tools we adopt now will decide whether that knowledge gets cultivated where it is, or quietly extracted and sold back to us as a standard offering. Whoever ends up writing the papers, humans, machines, or both, they need better tools and Papers AI is one of them.

Do we actually need to write papers at all?

I want to be careful here, because this is easy to misread as a claim that the human researcher is now optional. That is not what I think. I have spent a while trying to work out whether large language models are really just very fancy autocomplete, or whether they can be genuine question machines. The research coming out of Google and its co-scientist work suggests that at the very least they can enhance a researcher’s creativity when it comes to generating novel research directions. But even if it turns out that the LLMs and the agentic scaffolding around them are never quite both answer AND question machines, I still think there is an enormous amount of disruption coming to the academic space. Not because the machines are geniuses. Because the machines remove friction, and academia is held together by friction.

If we are going to lean on the creativity and the cross-domain thinking of human researchers, then surely that should be where the human time goes. The eureka moment. The interpretation. The intuition that two things nobody has put next to each other belong next to each other.

That is not where the time actually goes. The time goes into the machinery around the insight. Cleaning the data. Running the analysis. Formatting it into the shape a journal will accept. Somewhere in this process, human problems can come into play. There is interpretation drift. There is over-interpretation. There is cherry-picking, where a researcher already knows the story they want to tell and the results get nudged until they tell it. High-impact research is supposed to look a certain way, and so results start to arrange themselves into that certain way.

Why can we not take every dataset, every research output, run it through a machine, and get out a standardised object that carries all of the context of the research, but strips out a huge amount of the interpretation and the sheer time cost of turning a dataset into something other people can build on? So it was serendipitous that Claude Science launched and the getting started guide has this as one of the options “Run a first-pass analysis on a dataset you already have. Point Claude Science at your data (or a public dataset you’ve been meaning to look at) - it runs QC and a first analysis pass and returns figures plus a written summary.” So I did.

I downloaded some well cited datasets from Figshare and had it do the analysis and create a re-usable, forkable, machine readable paper. Then I did a few more, from Dryad and Zenodo, looking at different topics. There’s now 14 of these papers that look something like this:

Then in a conversation with Graham Smith of Nature, he alerted me to the Data Journalist Agent , by Kevin Qinghong Lin - which turns data into journalism - So from one dataset, we can get the academic paper and the science communication to go with it. The stories they create are pretty cool.

You can see all of this at datasetpapers.com.

It is worth asking what the paper was ever for. Ashish Uppala has a good piece on this, arguing that the journal has quietly done four jobs at once since 1665: it registers who got there first, it certifies that the work is sound, it disseminates the work outward, and it archives it so someone can check it later. Those four were welded together because in the seventeenth century it was cheaper to do them under one roof than to arrange them separately. The internet already made three of them close to free. Registration is a timestamp. Dissemination is a URL. Archival is a repository with a persistent identifier. Certification is the one that never got cheaper, and it is the one everything else is now hanging off.

So when I say the paper stops being the point, I mean something narrower than it sounds. Three of the four jobs no longer need a paper to happen. The fourth still needs something, and the interesting question is what that something is.

A datasetpaper is a versioned, forkable, executable research object built on an open dataset. The data, the code, the environment, the figures, and each individual claim are all separately addressable. The written narrative, the thing we currently treat as the entire output, is just one rendered view of the object. You can read it as a paper if you want a paper. You can read it as a story, evidence-linked, with a click-through viewer on every sentence, if you want that. Or a machine can read it as a bundle of claims, each one carrying its own provenance and its own verification status, and build on a single claim without ever parsing the prose.

Generating an analysis and writing it up used to be the hard part. It is not any more. And once anyone can produce a plausible-looking analysis in minutes, plausibility becomes worthless. The scarce thing is no longer the write-up. The scarce thing is trust. Knowing which claim was actually re-executed from the data, and which one was merely asserted because it sounded right. The verification and the provenance are the product. However, verification is only half of what a journal sells. Uppala splits prestige into two things we usually mash together: trust, meaning the methods are sound and the data is real, and significance, meaning the work matters. Re-execution handles the first. It does nothing at all for the second. Significance has always been an editorial guess, and it is a guess with a poor record, given that Nature turned away Krebs on the citric acid cycle and desk-rejected the modified-mRNA work behind the COVID vaccines inside a day. Uppala’s suggestion is that significance is better determined “on read instead of on write”, by machines that know what matters to a particular reader, rather than by an editor guessing on everyone’s behalf at the moment of submission. I think that is right, and I think it is the harder half.

There is a nice side effect here too, which is speed. This is fast dissemination. Something closer to a research report that comes out dynamically as findings land, rather than a polished paper that emerges eighteen months later once peer review has finished with it. And every time you speed up dissemination (or accessibility - eg. Open Access publishing), you speed up the point at which the next researcher, or the next model, can start building on what came before. That compounds. It is the single most underrated force in research, and we throttle it constantly.

Too fast publishing?

If you speed up the moment research becomes visible, you have to ask who gets to see it, and when.

Right now there is enormous pressure on researchers around being scooped. Publish or perish dictates the careers of a lot of hard working researchers. People disseminate carefully, defensively, only once they are confident they will get the credit. And into this already anxious system we are now introducing models that would very much like to read your work before you have decided it is ready.

We are already seeing LLMs reaching for research data before the researcher has chosen to publish it. Not the finished paper. The preliminary results. The hypotheses. The half-formed analysis sitting in a collaborative writing tool. OpenAI acquiring the team behind a collaborative writing platform is not a neutral event when you sit it next to a stated ambition to be part of the scientific pipeline. If models start training on pre-published research, at the analysis stage or even earlier, then researchers are being scooped before they have published anything at all.

I spoke about this at London Tech Week recently. My view, is that there has to be a line. The researcher has to approve the moment their work is ready to be disseminated. That has to remain a human choice. You can argue that early access to all of this in-progress science is good for humanity, that it accelerates the whole enterprise, and I would not dismiss that argument. You can also argue it is quietly terrible for the individual human scientist whose one good idea gets absorbed into a training run before they got the paper out. Both things are true at once, which is exactly why we should not let the default just happen to us.

Show me the incentives

Through Figshare, and through the State of Open Data survey we run every year, I keep hearing the same tension. Researchers do not feel they get enough credit for sharing their data. And researchers do not have the time to prepare their data for sharing in the first place. Those two complaints are usually treated as separate. They are the same complaint. It is a friction and reward problem.

So automated workflows that turn research findings into publishable, creditable units, seamlessly, might ease that tension from both ends at once. That is a large part of what datasetpapers is testing. Every datasetpaper starts from someone’s dataset, records that debt explicitly, notifies the original depositor that their data was used, and gives them a first-class place in the credit graph. The people who share data have been under-credited for as long as data sharing has existed. A world where machines analyse open data at scale makes that worse, unless the credit is designed in from the start. So it is designed in. There is a cautionary tale here that I had not known about until recently. Between 1961 and 1967 the NIH ran the Information Exchange Groups, a preprint network that circulated unpublished papers by post to a membership that grew past 3,600 scientists. It was essentially arXiv, thirty years early, and it worked. It died because journal editors saw what it was and agreed among themselves to refuse publication to anything that had been circulated through it. Show me the incentives, and I’ll show you the outcomes.

Reducing friction is not enough. Sharing results for the good of mankind is not enough. Solving disease, extraordinarily, is not enough on its own to make a researcher change how they work, if the incentive structure still rewards the old behaviour. Funders can mandate new workflows, and interestingly it often turns out the new workflow is less work than the one they are already asking for. But someone has to align the incentive with the outcome, deliberately, or the better system just sits there being better and unused. So it goes.

I agree with this hypothetical end goal. The path to it is still very bloody complex. An AI system is only ever as good as the verifiability and the provenance of the thing it reasons over. We call this the substrate. The substrate determines the ceiling.

Biology is trying to build a substrate it can trust

Figure 1. Foundation models now span almost every scale of biology. The substrate they rely on is the cross-cutting requirement.

There are now foundation models at almost every scale of biology. AlphaFold and its successors at the molecular level. Models like scGPT and Geneformer at the single-cell level. UNI and Virchow at the pathology level. Organ digital twins above that. It is fantastic that we are building these things. The Cell Perspective by the Chan Zuckerberg-led group on the AI Virtual Cell, along with the Arc Institute Virtual Cell Challenge launched in 2025, gives a sense of how much organised effort (and money) is now going into building the next layer of the stack.

There is however, a caveat. A model fit to observational data only ever learns the correlation structure of that data. It learns the patterns, and it learns the biases, and it cannot tell the difference between them without help. The single-cell company Noetik has described its own model’s outputs as “not at all guaranteed to be causal, merely strongly correlated.” Recent benchmarking by Ahlmann-Eltze and colleagues found that five state-of-the-art perturbation models could not beat a simple linear baseline when tested on gene knockouts they had genuinely never seen. The models are excellent at interpolating within what they have observed, and unreliable outside of that.

The fix the field has converged on is to engineer causal structure into the substrate itself. Rather than gathering more observational data, the move is to feed the models perturbation experiments, the results of actually intervening on a system rather than just watching it. You build a substrate that encodes cause as well as correlation, and the model’s ceiling rises accordingly. The Virtual Cell Challenge is, in a sense, a community-scale attempt to do exactly this in public.

Software can do this, because its substrate was verifiable all along

The reason AI agents arrived in software engineering before almost anywhere else is that code is executable and feedback-rich. Kenny Workman of LatchBio makes this argument well. You can run code, watch it fail, read the error, and try again. The substrate hands you ground truth on every loop. Or as he puts it “Agentic biology is shaped like software“

Figure 2. The gradient from locally verifiable to globally verifiable. After Kenny Workman (2026).

Take a hard scientific question, like whether a particular set of gene mutations drives a disease. At the top level it has no global verifiability, with no single correct answer to grade against. But it decomposes into smaller steps, some of which are locally verifiable. Did this cell line pass quality control? Do these genes show differential expression? Each of those has a crisp answer. The way to answer this is to build up from the locally verifiable steps toward the globally uncertain claim. That gradient, from locally verifiable to globally verifiable, is the single most useful lens I have found for the biology problem. It tells you where progress is fast (the left end) and where it is slow (the right end). It also tells you what kind of infrastructure each end needs.

What this looks like in drug discovery

For pharma, the locally verifiable layer is where most of the immediate value sits. Was this target mentioned in a paper? Was that paper funded by which grants? Did the trials supporting it use the right comparator? Has the molecule been characterised in patents? Each of these is a crisp yes-or-no, and each is the kind of question that AI ought to be able to answer with confidence, but only if the underlying substrate is structured and provenance-bearing rather than a soup of free text.

This is where the kind of resource we have spent a long time building at Digital Science comes in. The Dimensions Knowledge Graph links publications, grants, patents, clinical trials, datasets and policy documents into a single typed structure. Every entity carries a persistent identifier. Every relationship can be traced back to its source. If a model says “this target appears in seventeen recent papers funded by this consortium and three clinical trials at this phase,” that string is not coming from a statistical pattern. It is coming from edges in a graph. The model becomes a query interface to a substrate, and the substrate carries the truth.

The same principle holds one layer down. Figshare exists in part because the perturbation data that biology is now scrambling to generate needs persistent identifiers, version history and provenance metadata, or it cannot be reused. A virtual cell model trained on undocumented data is a virtual cell model whose ceiling is set by the lowest-quality dataset it ingested. The persistent identifier (DOI for publications and datasets, ORCID for researchers, ROR for institutions) is the plumbing that decides whether any of this works downstream.

FAIR has been pointing here for a decade

The FAIR data principles, findable, accessible, interoperable, reusable, have been making this argument since 2016, long before anyone needed a large language model to care. For most of that decade they sounded like infrastructure housekeeping, the sort of thing you nod along to and deprioritise. What has changed is that AI has suddenly made the cost of an unverifiable substrate visible, and urgent, and measurable in failed drug trials.

A trial fails for many reasons, but a meaningful fraction of late-stage failures trace back to evidence chains that no one ever audited at the start. If the original target paper relied on a misinterpreted figure, or the dose-finding work cited a study with a flawed control arm, that information was always in principle recoverable. It just was not connected to the system that picked the molecule. A substrate with provenance built in does that connecting by default.

We’re on a slope as opposed to waiting for a Eureka moment

The biosingularity is probably better understood as a slope than a moment. The labs and companies that climb it fastest will be the ones whose models reason over the most verifiable, best-provenanced substrate, rather than the ones with the cleverest models. The ceiling is set by the ground, not by what stands on it. We need to continue to improve provenance and reproducibility in research. We need to continue to push for Open Science. The infrastructure work, persistent identifiers and knowledge graphs and perturbation datasets and FAIR-compliant repositories, is what determines how high the ceiling rises. We are getting better at this as a research community. Preprints help. So does open data. The substrate beneath biology is next, and the alternative is the same one playing out elsewhere in the research ecosystem. Ungrounded generation rushes in to fill the gap, with things that were never true.

As mentioned in my last post “We have a deadline, because something is already busy filling the substrate with things that were never true.”

In my opinion, the policy is needed and appropriate. The underlying problem is bigger than arXiv.

Figure 1. Citation-hallucination rates and monthly counts by repository, as of August 2025. Source: Zhao et al. (2026), arXiv:2605.07723.

A letter in The Lancet by Maxim Topaz and colleagues found that one in every 277 PubMed-indexed papers in early 2026 cited something that does not exist. In 2023 the figure was one in 2,828. A much larger audit by Zhao, Yin and colleagues at Cornell, Berkeley, Tsinghua and UCLA looked at 111 million references across arXiv, bioRxiv, SSRN and PubMed Central. Their conservative estimate is that at least 146,932 hallucinated citations entered the scientific record in 2025 alone. All four repositories climbed sharply from mid-2024. By August 2025 the hallucination rate ranged from 0.21% on bioRxiv to 1.91% on SSRN, and because PubMed Central is so much larger, it carried the highest monthly count even at a rate of 0.27%.

These numbers are largely about general-purpose language models being asked to do something they were not designed for. Generating prose is what they do well. Verifying that a citation refers to a real paper is not in their job description, and the architecture does not support it. They are statistical prediction engines that produce the most plausible-looking string given everything they have absorbed. This is a problem when it claims to build on the permanent record of research.

The hallucinations have a direction

When fabricated citations name real authors, they do not name them at random. This is likely true of existing research practices, but if we’re entering a new era of research, it would be cool if we got rid of some of the old problems. The Cornell team checked whether this bias was confined to the obviously fake citations, or whether it showed up in the perfectly real ones sitting beside them in the same bibliographies. It does. The real citations in papers that also contain hallucinations skew toward the same prominent authors.

Figure 2. How hallucinated citations differ from matched genuine ones. Source: Zhao et al. (2026), arXiv:2605.07723.

Compared with matched genuine citations, hallucinated citations credit scholars with 68.8% more prior publications and 58.3% more citations. The credited names also skew male, by 6.4 percentage points. Prominence and plausibility are entangled in the training data, since the most-cited authors appear in the most reference lists. The model has learned that their names belong in reference lists. When it fabricates, it fabricates those who would most likely turn up in a reference list.

Figure 3. The Matthew effect, automated.

The Matthew effect, Robert Merton’s name for the rich-get-richer dynamic in citation, has always existed. What is new is that we may be about to automate it at a scale and speed that no funding panel or promotion committee can keep up with.

Why a general model is the wrong tool for research

The pattern in the data points to a specific kind of mistake. A general-purpose language model has no representation of the world’s actual citation graph. Asking it whether a paper is real, or whether it supports a claim, is asking a question its architecture does not know how to answer correctly. The result is a confident-sounding answer that is roughly correlated with the truth and fails in predictable ways. This is something I saw when building

https://datacitations.com/

, A knowledge graph linking 9.3 million data citations across nearly 2,000 data repositories, that demonstrates how FAIR principles power trustworthy, traceable AI answers grounded in structured evidence.

The interesting observation, and the reason I think this problem is more tractable than the discourse suggests, is that science already has the infrastructure to do this properly. The infrastructure simply has not been wired up to the place where the generation is happening.

Figure 4. Two architectures, two failure modes.

Every published paper has a DOI. Every dataset can have one too. Every author can have an ORCID. Every institution can have a ROR identifier. Crossref will resolve any DOI in milliseconds. OpenAlex, DataCite and Semantic Scholar expose this as queryable open data. At Digital Science the Dimensions Knowledge Graph holds these relationships in typed, persistent form, connecting publications, grants, patents, clinical trials, datasets and people. Figshare hosts versioned datasets with their own DOIs. The Cornell team’s own validation pipeline matched 95% of references on first pass with off-the-shelf tools.

The reason 78% of fabricated citations still pass arXiv moderation comes down to one thing. The validation is not being run. The substrate exists. The plumbing connecting it to the systems that produce citations does not. A subject-specific model that consults this substrate before it speaks has a different failure mode than a general model that does not. It will sometimes say “this paper is not in my graph” rather than inventing one. It will sometimes return a smaller set of citations than the user expected. It will sometimes flag conflicts in the underlying evidence. None of these are exciting demo moments. All of them are what scientific reasoning is supposed to look like.

Will we move to a GitHub-like way of building?

The longer I sit with this, the more the analogy to software seems instructive. Software engineering went from a craft where you copy-pasted from forums to one where every dependency has a version number, a hash, an authorship trail, a license, and a public history of who changed what when. It is the reason agents can now reason about software at all, because the substrate is verifiable on every loop.

The equivalent in scholarly publishing would be a citation graph where every reference resolves to a persistent identifier, every claim has a recorded provenance trail back to source data, every researcher’s contribution is attached to their ORCID, and every dataset has a version on a repository like Figshare with its own DOI. None of these are technologically hard. Most of them already exist, it feels like we’re close, but research is not (yet) happening this way. arXiv’s new policy is one small step in that direction. So are journal requirements to validate references against Crossref at submission. So is the slow expansion of registered reports and data-availability requirements. None of these is a complete solution. They are pieces of a substrate.

The deadline

The thing that makes this urgent is the second-order effect, more than the absolute number of hallucinated citations today. Hallucinated citations are getting indexed in Google Scholar as standalone bibliographic entries. They are being ingested into the training corpora of the next generation of models. The contamination, as Maxim Topaz put it in the coverage of his paper, does not go away when the AI gets better. The longer the substrate is left ungoverned, the more of the next-generation models will be trained on something that includes things that were never true.

We have a deadline, because something is already busy filling the substrate with things that were never true.

I’m an optimist. This is a problem science is well-equipped to solve, because every piece of the infrastructure already exists. The problem is that research has a lot of problems that could be solved if cultural changes amongst the research community happened faster. In this new age of acceleration, maybe we’ll finally start optimising the research process.

In October 2024, a Nobel Prize in Chemistry went, in effect, to a piece of software. More precisely it went to the people who built it. Demis Hassabis and John Jumper at Google DeepMind took half the prize, sharing the rest with David Baker in Seattle. The thing they had made, AlphaFold, did in a few years what structural biology had been chipping away at for half a century. It predicted the three-dimensional shape of a protein from its amino acid sequence, and then it did it for almost every protein science had ever named. Around 200 million of them.

What really amazed me was the size of the project team. Just 19 names share the lead credit on the 2021 AlphaFold2 paper, the block marked as having contributed equally, and the full author list runs to thirty-three. This is a problem that had defeated the global biochemistry community for 50 years, only to be solved by an AI-enabled (and compute) small team.

Does this mean that the old adage, “If you want to go fast, go alone. If you want to go far, go together,” is no longer true in an AI-powered, agentic world? Usually attributed to West Africa, usually wheeled out to justify whatever the speaker had already decided to do. And for as long as I have been in research, it made sense. It described a genuine trade-off. You picked one. Speed or distance. The lone researcher sprints and burns out. The consortium endures and crawls. This is measurable. Lingfei Wu, Dashun Wang and James Evans went through more than 65 million papers, patents and software products spanning 60 years of output, and published the result in Nature in 2019. Large teams develop. Small teams disrupt. Small teams reach further back into older ideas, produce the more destabilising work, and pay off further into the future, if they pay off at all. Big teams take an existing line and push it forward. Go far, or go fast.

The trade-off breaks

The AlphaFold team was disruptive in exactly the Wu, Wang and Evans sense, reaching back to an old problem most people assumed was decades from solution. But it did not stay narrow, and it did not take a generation to land. Its predictions have since been used by more than two million researchers in 190 countries, and the core work was done inside about 18 months. Fast and far, from the same 19 people. AI is what collapsed it. A small team with enough model leverage now reaches as far as a consortium and keeps the speed of a startup. Go fast and far, alone. A lot of the data and models are open. Are we about to see a new class of small-team research disruptors?

I pulled the author lists on the breakthroughs of the past three years where AI did the discovering, across several fields. The pattern holds.

AI-enabled scientific breakthroughs of the past three years, by team size. Sources are Nature and Science. AlphaFold3, at 48, is the largest team in the set.

Six people built GNoME and predicted 2.2 million new crystals, around 380,000 of them stable, roughly a tenfold expansion in the stable inorganic materials known to science. Six more, at Huawei, built Pangu-Weather and beat the leading operational forecast. Sixteen built AlphaMissense and produced a verdict on 89 per cent of all possible human missense mutations, a resource clinical geneticists now draw on.

And before anyone objects that this is big tech doing it with pure compute, look at the wet-lab end. The new structural class of antibiotics that kills MRSA, the first new class in 60 years, came from screening more than 12 million compounds with a graph neural network and then synthesising the survivors and testing them in mice. Twenty-one people, across MIT, the Broad and two other institutes. Berkeley’s A-Lab, a robotic chemistry lab that planned, ran and interpreted its own experiments to make dozens of new compounds, took 16. The two largest teams in the set are biology at the cutting edge, the Baker lab’s protein design at 28 and AlphaFold3 at 48, and both are still small by the standards of the field they sit in.

So are these really small teams?

A short author list can hide a long shadow. Behind those six names on the Pangu-Weather paper sits Huawei Cloud. Behind AlphaFold sits the whole of DeepMind. And every one of these models learned from a dataset that took a large, slow, collaborative effort to build. So it’s not like “six people did this alone”. It is “six people did this on top of what an institution spent decades assembling”. But either way, these small teams are doing in months what used to take a generation and a large workforce.

So both things are true at once. The teams really are small, and they really are doing what once took huge teams. The trick is that the huge team already ran. It built the substrate and left it in an open database where a handful of people with a good model could pick it up. Tier one takes an army and a decade. Tier two takes a dozen people and a few months.

Standing on the shoulders of giants

This is the bit that matters most if you care about research infrastructure.

AlphaFold did not arrive from nowhere. It learned to predict structures by training on the Protein Data Bank, which is to say it learned from 50 years of slow, expensive, collaborative experimental work. Every structure in that archive was solved by someone at a beamline or an NMR machine or a cryo-EM rig, then deposited, curated and shared. EMBL had been generating that ground truth for decades. Jumper has said it plainly: public data were essential.

So the small fast team did not replace the large slow collaboration. It stood on its shoulders. Once you see that, the two tiers stop looking like rivals and start looking like what they are, which is stacked. We’re seeing big philanthropic orgs picking up the heavy lifting when it comes to big data and models - biohub, Allen AI, Astera are all making massive amounts of information open to all. With folks predicting the third wave of American philanthropy, that AI will likely add ~$37–100B per year in intended philanthropic spend in the near future - the future for open science looks bright. It looks like the “go far, go together” side of things is taken care of.

The same era’s defining results, plotted by team size on a log scale. The Higgs boson mass paper carried 5,154 authors. A later ATLAS collaboration paper reached 8,778. The AI breakthroughs sit three orders of magnitude below, with almost nothing in between.

Two tiers, one stack

That gap in the middle of the chart is the story. Modern science is settling into two tiers. Tier one is the slow business of building the substrate: the atlases, the reference datasets, the standardised measurements no single lab could produce alone. It shows up in those author counts of five and eight thousand. It is how you measure a Higgs boson, and the most ambitious biological example of it now is the effort to model the cell. The Human Cell Atlas set out to catalogue every human cell type and had to invent the methods as it went. The Chan Zuckerberg Initiative has since put real weight behind a virtual cell, standing up one of the largest non profit GPU clusters in the life sciences, launching a Billion Cells Project with 10x Genomics and Ultima Genomics, and committing more than 10 billion dollars to basic research over a decade. The Arc Institute trained its first virtual cell model on more than 170 million cells. These are not papers. They are infrastructure, and they move at the pace of consortia.

Tier two is the small team with AI leverage. The PDB worked as a launchpad because it was open, it was standardised, it carried identifiers, and a machine could read it without a human in the loop. That is FAIR data doing the quiet work nobody hands out prizes for. Build the cell atlases of this decade to the same standard and they become launchpads too. Build them as a thousand incompatible spreadsheets behind a thousand logins and they don’t.

However small the team, you are still a team

Not one of those breakthroughs has a single author (yet). A team of 19 co-authoring a 60page paper with 32 component algorithms is not 19 people working alone. It is 19 people who have to think, write and ship as if they were one. A small team can go fast not because collaboration stopped mattering, but because its friction dropped close to zero. There is a reason why collaborative tools like Overleaf have 25 million users. Everyone writing together, in the same place, in real time, with the version history and the structure and the maths held in one place. Modern problems require modern solutions.

So I think the proverb needs a rewrite for the age of large models. If you want to go fast and far, go as a small team, give it real leverage, stand it on an open foundation someone else built, and give it tools that make collaboration close to effortless.

As a disclaimer, everything discussed here is experimental. The platforms described are not robust scholarly infrastructure. They should not be cited, relied upon, or treated as production systems. They’re a hobby project built on open APIs, open data, and open infrastructure designed to test what’s technically possible.

The problems worth solving

We’ve built good infrastructure for sharing research outputs, yet most of the value locked in those outputs goes unrealised. Meanwhile, the research community is under pressure in almost every dimension: time, funding, reviewer bandwidth, editorial capacity. A disproportionate amount of effort goes on tasks that are fairly mechanical - formatting metadata, screening submissions for integrity, summarising prior literature, running standard statistical checks.

New academic knowledge creation and basic research has never been more in demand. AI will never replace this. But there is a subset of academic knowledge gap filling that new tools seem capable of filling. Can they? What if some of those tasks could be automated? I am not attempting here answer the question of “What should be automated?”

The four platforms, briefly

FAIRdata.ai starts at the data layer. A large proportion of datasets deposited on generalist repositories are technically FAIR but practically unusable. The metadata is incomplete, the variable labels are opaque, the context is missing. FAIRdata.ai tries to fix that automatically, enriching metadata to improve FAIR scores. It also tries to make use of surprisal scoring and automated discovery techniques to surface findings within datasets that may never have been noticed by the original depositors. Those signals feed into OpenScience.ai.

You can read more details about the technical capabilities and dependencies of FAIRdata.ai here.

OpenScience.ai is the hypothesis engine. It queries open scientific databases. eg. gnomAD, Ensembl, AlphaFold, the literature graph, and any hypotheses that FAIRdata.ai sends it and tries to identify patterns that haven’t yet been written up as research articles. It then generates hypotheses, assembles supporting evidence, and drafts those hypotheses into structured manuscripts. It’s not trying to replace the experimental scientist; it’s trying to explore the space of already-existing open data for combinations that nobody has looked at yet.

You can read more details about the technical capabilities and dependencies of OpenScience.ai here.

Preprints.ai asks a fairly direct question - How much of peer review can we automate? It runs a multi-agent review pipeline assessing both research integrity, methodology, statistical validity, reproducibility, and novelty. The output is a structured assessment graded against a rubric, with deliberative rounds between agents before a consensus is reached.

You can read more details about the technical capabilities and dependencies of Preprints.ai here.

OpenAccess.ai closes the loop. If we’re generating articles via OpenScience.ai and reviewing them via Preprints.ai, we need somewhere to publish them. OpenAccess.ai is that place - an open access journal built on the assumption that the editorial process can run largely on agents, and that the unit cost of academic publishing can be meaningfully reduced as a result.

You can read more details about the technical capabilities and dependencies of OpenAccess.ai here.

All four platforms are built exclusively on open data and open APIs. If you can build a plausible end-to-end research automation pipeline using only what’s publicly available, that says something useful about the current state of open science infrastructure. The constraint on scientific productivity isn’t data availability; it’s the capacity for machines to process and synthesise it.

I personally have access to considerably richer data through my work at Digital Science (citation graphs, full-text repositories, institutional metadata). I haven’t used any of it here. What you’re seeing is what’s possible with no proprietary advantage, which I think makes the findings more generalisable.

Humans in the loop

Framing this as “minimal human intervention” is accurate for certain parts of the pipeline and misleading for others.

There are stages where the automation is fairly robust. Metadata enrichment, FAIR scoring, integrity screening, novelty checks against the existing literature. These are tasks where the output is verifiable and the failure modes are containable. They’re also, not coincidentally, the tasks that consume a lot of researcher time without requiring much researcher judgement.

But there are other stages where a human needs to be in the loop, and probably will for a while. Editorial decisions - whether a piece of work is actually worth publishing, whether the framing is intellectually honest, whether the conclusions follow from the evidence. These are not tasks I’d be comfortable delegating to agents at this point. The current OpenAccess.ai setup allows human editors to review agent assessments before making final decisions, and I think that hybrid model is the right one for this experiment. It also leans very heavily on eLife’s Publish, Review, Curate model, which is a model I see gaining more traction and uptake in the coming years.

Similarly, the hypothesis generation in OpenScience.ai needs a human sanity check before anything gets treated as a serious research direction. The system can identify patterns in open data, but it can’t yet reliably distinguish a genuinely interesting finding from a statistical artefact or a confound that any domain expert would spot immediately. I’ve had to be quite disciplined about this after discovering early on that the pipeline would confidently produce fabricated statistics if you didn’t anchor every claim to verifiable data from a canonical source.

What’s next

Each platform has a reasonably long list of things it still can’t do well. OpenScience.ai’s anti-fabrication infrastructure is solid now but the discovery logic is still fairly shallow. Preprints.ai’s local inference falls over on structured output for complex multi-agent workflows. FAIRdata.ai’s scoring is imprecise for accession-based repositories. OpenAccess.ai’s cost economics need testing at meaningful volume before any of the unit cost claims are worth much.

I’m writing this up now in order to experiment in the open. The idea of chaining these capabilities together is the more interesting thing to put into the world, even if the implementations are still rough.

If you’re working on any of these problems and want to compare notes, or if you spot something obviously wrong with the approach, I’d be glad to hear from you.

What Problem Are We Trying to Solve?

Article Processing Charges at major journals run to thousands of dollars per paper. The editorial infrastructure - submission systems, copyediting, typesetting, hosting - is expensive. What I want to know is how much of the process can be automated, particularly for machine generated academic content.

If you can automate a substantial fraction of the editorial process, (as Preprints.ai is attempting to do) what happens to the unit economics? OpenAccess.ai is an experiment designed to provide some data.

There’s a second motivation. OpenScience.ai (in theory) generates research articles. Those articles need somewhere to go. Conventional open access infrastructure is designed for human-authored papers, with submission systems and APC structures that don’t map cleanly onto a machine-generated pipeline. OpenAccess.ai is built to accept submissions via API, which means it can serve as the publishing endpoint for the entire experimental pipeline I’m building, without friction.

The Technical Architecture

Core Stack

OpenAccess.ai is a Next.js 14 application deployed on Netlify with Supabase for PostgreSQL, auth, and file storage, and Railway for microservices. Serverless-first, no dedicated servers, minimal ops overhead. If we’re testing whether publishing can be radically cheap, the infrastructure costs need to be radically cheap too.

The Document Processing Pipeline

When someone uploads a PDF, the platform needs to extract the full text, identify sections, parse references, pull out figures, and generate structured metadata.

Stage 1: Fast extraction

The upload route runs pdf-parse for basic text extraction in parallel with a Supabase Storage upload. For DOCX files, mammoth.js handles the conversion. This produces something immediately. However, for PDFs with embedded fonts (which is most academic PDFs), pdf-parse returns garbled text.

Stage 2: AI vision extraction

When pdf-parse fails (detected by our garbled-text heuristic that checks for XMP metadata patterns, font encoding artifacts, and low information density), the client triggers a Supabase Edge Function that sends the PDF to Claude’s document vision API. Claude Haiku reads the visual layout of each page and produces clean Markdown with proper sections, author lists, figure captions, and references.

Stage 3: Structured metadata extraction

The extracted text is sent to Claude Haiku again, this time with a structured prompt that returns JSON with title, abstract, authors with affiliations, keywords, and subject area (mapped to bioRxiv’s taxonomy of 25 subject areas). This populates the submission form automatically.

Stage 4: Figure extraction

A PyMuPDF-based microservice on Railway extracts embedded images from the PDF, filters by minimum dimensions, and stores them in Supabase Storage. The figure URLs are matched to the document AST by order.

The AST Document Model

All documents are converted to an internal Abstract Syntax Tree inspired by MyST Markdown (itself derived from mdast and Stencila). The AST is the canonical representation - HTML, JATS XML, PDF, and all citation formats are rendered from it on demand.

RootNode

├── meta: { title, abstract, sections[], wordCount, figureCount, referenceCount }

├── children: SectionNode[]

│ ├── SectionNode (id: “introduction”)

│ │ ├── HeadingNode

│ │ ├── ParagraphNode[]

│ │ └── FigureNode { url, caption, label }

│ ├── SectionNode (id: “methods”)

│ └── SectionNode (id: “results”)

└── references: ReferenceNode[] (CSL-JSON format)

Sections are first-class nodes, not just CSS classes on headings. This means you can programmatically extract “the methods section” or “all figure captions” without parsing HTML — which is essential for the AI review pipeline.

Peer Review via Preprints.ai

Every submission is automatically sent to Preprints.ai for assessment. The review system uses 8 AI reviewers that evaluate the paper across multiple dimensions:

• Integrity scoring (A–E grade scale)

• Novelty assessment (1–5 tiers: minimal to groundbreaking)

• Methodology checks (statistical validity, sample sizes, controls)

• Transparency audit (ethics statement, data/code availability, funding disclosure, COI, pre-registration)

• Provenance analysis (for AI-generated papers: model plausibility, prompt injection detection, reproducibility signals)

The grade, strengths, concerns (with severity: critical/major/minor), and per-reviewer scores are stored as structured JSON and displayed on the article page. The review is fully transparent — readers see exactly what the AI reviewers found, not just a binary accept/reject.

Output Formats and Academic Standards

A publishing platform is only useful if it produces outputs that the academic ecosystem can consume. OpenAccess.ai generates:

For Indexing and Discovery

• JATS XML - Journal Article Tag Suite, the standard for CrossRef DOI deposit, PubMed indexing, and PMC archiving. Generated on-demand from the AST via astToJats().

• Dublin Core meta tags - for library catalog discovery (Zotero, Mendeley)

• Highwire Press tags - citation_title, citation_author, etc. for Google Scholar

• Schema.org JSON-LD - ScholarlyArticle type with full provenance metadata

For Citation

• Citation styles: APA 7th, Vancouver, Chicago, Harvard, MLA 9th, BibTeX, RIS - generated client-side from article metadata using Citation Style Language conventions

• CFF (Citation File Format) - for software-like citations

For Machine Learning and Linked Data

• Croissant ML - ML dataset metadata standard, making the corpus discoverable for training pipelines

• RO-Crate - Research Object packaging with full provenance

• W3C DCAT - Data Catalog Vocabulary for dataset registries

• DataCite JSON - DOI registration metadata

Persistent Identifiers

• Handle System - Each article gets a globally unique persistent identifier (e.g., 20.500.14906/oai.biology.2026.00001) registered via Handle.net’s REST API

The Article Reader

The article page is modelled on eLife‘s reading experience

• Tabbed interface. Full text, Figures & data, Peer review (with full Preprints.ai assessment), Summary

• Sticky section navigation. Left sidebar with table of contents generated from AST section nodes, highlights current section on scroll

• Inline citations. Hover to see reference details, click to scroll to bibliography

• AI-powered Q&A. Claude-powered chat that answers questions about the article content

• Branded PDF export: Print-optimized HTML with OpenAccess.ai header, proper title block, A4 pagination

• Article metrics. Views, downloads, citations tracked per article

Authentication

Users sign in via ORCID OAuth or GitHub OAuth. ORCID profiles are enriched with works count and affiliation data. The affiliation field uses the ROR API (Research Organization Registry) for standardised institution lookup.

Agent-First Architecture

OpenAccess.ai exposes an MCP (Model Context Protocol) endpoint at /api/mcp with tools for:

• submit_paper - Programmatic manuscript submission (API key + credits)

• check_submission - Query review status

• get_article - Fetch in any format (JSON, JSON-LD, BibTeX, Croissant, etc.)

• search_articles - Full-text search

• get_subject_areas - Valid taxonomy lookup

Plus a REST API documented at /api/openapi.json and discoverable via /llms.txt, .well-known/ai-plugin.json, and an agent card. The API uses credit-based billing: generate an API key, purchase credits via Stripe, spend 1 credit per submission.

This is what enables OpenScience.ai to generate a paper and submit it to OpenAccess.ai without any human in the loop.

Tools That Inspired the Platform

Several open-source projects shaped the architecture:

What We Can Look to Do Next

We need to run a meaningful volume of articles through the full pipeline - generate via OpenScience.ai, review via Preprints.ai, publish via OpenAccess.ai - and measure the per-article cost at each stage.

Figure extraction needs improvement. The PyMuPDF service extracts embedded images but doesn’t yet match them to figure captions reliably. We’re evaluating eLife’s enhanced-preprints-image-server (IIIF-based) for production-grade image delivery.

Reference enrichment is the next major quality improvement. Replacing AnyStyle with jats-ref-refinery would dramatically improve citation linking by querying four external databases rather than relying on a single CRF parser.

Indexing and registration remain open questions. A journal that publishes AI-generated research will not easily get into PubMed or Dimensions.ai. But understanding what metadata, what provenance signals, and what quality documentation would be required for AI-generated research to be treated as a first-class research output is itself a useful question for the field to be working on.

The most interesting longer-term question is whether a hybrid editorial model makes sense - AI review as the default pathway, with human editorial escalation for papers that score ambiguously or that make claims in domains where agent reviewers have known weaknesses.

If you’re working on any of these problems and want to compare notes, or if you spot something obviously wrong with the approach, I’d be glad to hear from you.

What problem are we trying to solve?

Peer review is structurally failing. The volume of preprint submissions has grown faster than the reviewer pool for decades. Average review turnaround times have lengthened. Reviewer fatigue is well-documented. And review quality is inconsistent in ways that are impossible to fix when the process depends entirely on the goodwill of overcommitted academics. More simply, can we score every preprint.

At the same time, a significant fraction of what peer reviewers actually do is mechanical. Checking that statistical methods match reported results. Verifying that cited claims are accurately represented. Assessing whether the methodology section contains enough detail to reproduce the experiments. Scanning for common red flags - tortured phrases from paper mills, image duplication, impossible p-values.

Preprints.ai asks a sharper question than “can AI help with peer review?” It asks “if we were designing peer review from scratch for a world where powerful LLMs exist, what would we actually need humans for, and what could we comfortably automate?”

The platform currently runs a multi-agent review pipeline against preprints from bioRxiv and medRxiv, assessing two dimensions: research integrity (methodology, statistical validity, reproducibility, citation accuracy) and novelty (does the core claim already exist in the literature?). The output is a structured assessment graded on an A5-to-E1 rubric, with positivity bias actively recalibrated using publication outcome data.

What tools have inspired or been used to build this platform?

LLM providers and hybrid routing

We use three LLM providers in production, with each agent in the review panel assigned to a different provider to ensure genuine inter-model independence:

This isn’t just about cost optimisation. When three different model families independently agree on a finding, the probability that all three share the same hallucination is substantially lower than any single model’s error rate. Round-robin provider assignment prevents correlated model bias in the consensus score.

The review pipeline: Layer 1 (deterministic, zero LLM cost)

Before any LLM sees the paper, a battery of rule-based checks runs in under 2 seconds:

PDF Quality Gate → Language Detection → Paper Mill Detection → Image Forensics → Statistical Consistency → Trust Markers

These are inspired by and build on established tools - such as Ripeta’s Trust Indicators.

• Paper mill detection uses a database of 8,000+ tortured phrases (e.g., “bosom malignancy” for “breast cancer”) and 257 SCIgen patterns for computer-generated text

• Statistical consistency implements the GRIM test (Granularity-Related Inconsistency of Means) and statcheck-style p-value recalculation from reported test statistics

• Image forensics is inspired by ELIS (Error Level Image Similarity) — detecting duplicate figure regions, inconsistent compression, and manipulation artefacts using imagehash and Pillow

• Fabrication detection checks for Benford’s Law deviation in reported data and suspiciously perfect statistical distributions

• Trust markers detect the presence (or absence) of data availability statements, code repositories, ethics approvals with protocol numbers, pre-registration links (ClinicalTrials.gov, OSF), ORCID identifiers, and CRediT author contributions

• Reporting guideline compliance checks against ARRIVE (animal research), CONSORT (clinical trials), PRISMA (systematic reviews), and MIQE (qPCR)

The review pipeline: Layer 2 (multi-agent LLM review)

The agent architecture is heavily inspired by eLife’s deliberative review model. We run nine specialist agents, each with a distinct role and scope boundary:

Plus three additional domain-specialist agents routed based on the paper’s bioRxiv category, drawn from a pool of 31 field-specific configurations covering everything from biochemistry to systems biology. Each configuration specifies methodology standards, common issues, reporting guidelines, statistical expectations, and key databases for that field.

Deliberation: from independent reviews to consensus

Independent review is necessary but not sufficient. In human peer review, the reviewing editor synthesises reviewer comments into a coherent assessment. We replicate this with a structured deliberation protocol inspired by eLife:

1. Round 0 (Independent review) - All 9 agents review the paper in parallel, each with a different LLM provider. No agent sees any other agent’s output.

2. Round 1 (Consultation) - Agents see anonymised summaries of each other’s reviews. Each can agree, challenge, or add nuance.

3. Round 2 (Reconciliation) - A senior editor agent synthesises the consultation into a single eLife-style assessment: “This study presents [significance] findings on [topic]. The evidence is [strength], with [justification].”

4. Dissent recording - Any agent that disagrees with the final assessment can lodge a recorded dissent, preserved in the output for transparency.

This substantially reduces variance compared to single-pass review. We track inter-agent agreement (standard deviation of scores across agents) and use it as a confidence signal. eg. high disagreement triggers lower confidence scores and flags the paper as borderline.

External data sources

The platform integrates with multiple academic data APIs, each serving a distinct purpose:

Semantic Scholar
Citation network, reference verification, passage search
200M+ papers; checks cited claims are accurately represented

OpenAlex
Journal tier mapping, h-index, publication outcomes
Calibration: maps journal prestige to validate our novelty scores

Crossref
DOI metadata, retraction status, publication relations
Detects retracted papers and tracks preprint-to-publication links

Europe PMC
Publication discovery, fulltext links
Fallback for finding published versions of preprints

bioRxiv API
Preprint metadata ingestion
Primary source for paper metadata and PDF URLs

Retraction Watch
Known-retracted papers for validation
Ground truth: can our system detect papers that were later retracted?

The scoring rubric

We grade on two independent dimensions, producing a combined grade like B4 or C3:

Integrity (A–E): Strength of evidence. A = compelling, reproducible methodology with full transparency. E = fundamental unfixable problems (fabrication signals, pseudoscience, paper mill indicators).

Novelty (5–1): Significance of findings. 5 = landmark, field-shifting (almost never assigned). 3 = important, real contribution beyond a single subfield. 1 = useful but very limited — one more data point.

Each grade maps to an editorial decision in journal-standard language: Accept, Minor Revision, Major Revision, or Reject — with modifiers for low confidence, abstract-only assessment, or high reviewer disagreement.

Fighting positivity bias using recalibration

LLMs are famously sycophantic. Left uncalibrated, every paper gets a B. We fight this at multiple levels:

• Downward recalibration rules: 2+ critical weaknesses forces a D floor. 4+ major weaknesses caps at low-C. A “reject” recommendation from any agent caps integrity at 0.42.

• Per-agent accuracy tracking: After each calibration run, we compute every agent’s mean absolute error and bias against publication outcomes. Agents with measured bias > ±0.15 get a steering note injected into their next prompt: “Historical data shows you tend to overscore novelty by ~20%. Be especially critical of incremental contributions.”

• Inverse MAE weighting: In the consensus aggregation, agents with lower historical error get higher weight. An agent that has been consistently accurate on past papers has more influence on the final score than one that hasn’t.

• Publication outcome calibration: We match assessed preprints against publication records using Crossref’s open data. If we scored a paper as novelty-2 but it was published in a tier-5 journal, that delta feeds back into agent weights and prompt tuning.

Confidence and transparency

Every assessment includes:

• Confidence score (0–100%) derived from agent agreement, mean agent self-reported confidence, and score dispersion. High disagreement between agents lowers confidence.

• Individual agent reviews. Every agent’s scores, strengths, weaknesses, and recommendations are available in the full report, not just the consensus.

• A 2–3 sentence plain-language summary of the core finding, for a non-specialist audience.

Cost per paper

With hybrid routing across Claude, GPT-4o, and Gemini Flash, the current cost per full assessment is approximately $1.50–2.00 for a consensus review with 9 agents. Layer 1 deterministic checks add negligible cost. At scale, with aggressive caching of reference verification results and local model routing for lower-signal checks, we expect this to converge toward $0.50–1.00 per paper.

What we can look to do next

Validation is the open question

The most important next step is systematic validation. How do Preprints.ai assessments compare to human peer review on the same papers? We’ve built a /validate page where domain experts can agree, disagree, or override AI grades but we need hundreds of expert validations across diverse fields before the calibration claims are defensible. There is no incentive for subject experts to do this, so we can assume they wont.

We’re also running consistency tests. We are assessing the same papers multiple times to measure grade variance. If the system can’t reproduce its own grades, nobody should trust them. Our target is >70% exact grade stability and >90% within-one accuracy.

Field-specific scoring norms

A C3 in clinical medicine means something different than a C3 in pure mathematics. We’re computing per-discipline score distributions from assessed papers and using z-score normalisation to generate field-specific grade thresholds. Fields with sufficient data (20+ assessed papers) get their own norms; others fall back to global defaults.

The hybrid model question

Longer term, the interesting question is whether a hybrid model - AI as first-pass screen, human expert for borderline cases - could work operationally within a real journal workflow. The system already identifies which papers need human attention (low confidence, high agent disagreement, borderline C-range grades). The question is whether editors would trust it enough to act on.

I think the answer is yes, but only after the validation data proves it.

If you’re working on any of these problems and want to compare notes, or if you spot something obviously wrong with the approach, I’d be glad to hear from you.

OpenScience.ai is trying to look forwards. OpenScience.ai aims to be a gap filler in academic data and literature. Given the current state of open scientific databases, what patterns are present in the data that haven’t yet been written up as research claims? What combinations of findings, if synthesised across ClinVar, GTEx, STRING, and Open Targets, would constitute a novel contribution?

The goal isn’t to produce papers that get cited. It’s a thought experiment and a way to explore the space of what’s discoverable in open data when you remove the bandwidth constraint of human researchers having to do the querying manually.

What tools have inspired or been used to build this platform?

The Data Lake: 40+ Open Scientific Databases

The discovery pipeline operates on a local data lake of 44 bulk tables, populated from established scientific databases. Agents query local data first via a typed bulkFirst library before falling back to live APIs. The sources span:

Variant and Population Genetics

• gnomAD v4 (GraphQL) — population allele frequencies across AFR, AMR, ASJ, EAS, FIN, NFE, SAS populations, plus gene constraint scores (pLI, LOEUF, mis_z, syn_z) from the v4.1 constraint metrics

• ClinVar (NCBI Entrez API) — clinical variant interpretations with pathogenicity classifications

• GWAS Catalog — genome-wide association study results

• MyVariant.info — variant resolution and annotation aggregation

• dbSNP (NCBI eUtils) — variant identifiers

Gene and Protein Data

• Ensembl — variant effect prediction (VEP), gene annotation, regulatory feature overlap

• UniProt — protein sequences, functional annotations, subcellular localisation

• AlphaFold DB — predicted 3D protein structures, integrated at three levels: inline enrichment during discovery, bulk ingestion of coverage data, and three dedicated database tables for structure mapping

• RCSB PDB — experimentally determined protein structures (215K entries)

• InterPro — protein family and domain classification

• HGNC — standardised gene nomenclature

Pharmacogenomics and Drug Data

• PharmGKB — pharmacogenomic clinical annotations and dosing guidelines

• ChEMBL — bioactive compound data (5M+ bioactivity records)

• DGIdb — drug-gene interaction database

• DDInter — drug-drug interactions

• PharmVar — star allele nomenclature for pharmacogenes

• OpenFDA — adverse drug reaction reports

Pathway and Network Analysis

• STRING — protein-protein interaction networks with confidence scoring

• Reactome — curated biological pathways

• MSigDB — gene set collections for pathway enrichment

• Gene Ontology — functional annotations via GOA

Disease and Phenotype

• Open Targets Platform (GraphQL) — target-disease-drug associations, evidence, and locus-to-gene predictions

• DisGeNET — gene-disease associations with evidence scoring

• HPO (Human Phenotype Ontology) — standardised phenotype terminology

• Mondo (via Monarch Initiative) — disease ontology

• OMIM — Mendelian disease catalogue

Expression and Regulation

• GTEx (v8 bulk download) — tissue-specific gene expression across 54 tissues

• ENCODE cCRE — candidate cis-regulatory elements (32M elements genome-wide)

• CellxGene — single-cell expression atlas

Literature and Citation

• Semantic Scholar — paper search, citation graphs, and the ASTA snippet search endpoint (275M passages across 12M+ papers) for contradiction detection

• OpenAlex — open publication metadata, citation counts, author disambiguation

• Europe PMC — open access full-text and text-mined annotations

• NIH Reporter — funded research project data

Clinical

• ClinicalTrials.gov — active clinical trial registrations

The Agent Fleet: Specialised Research Personas

The discovery pipeline doesn’t use a single general-purpose LLM prompt. It runs through a fleet of 89 persistent AI agents, each with a distinct scientific identity defined in a SOUL file. Each SOUL specifies the agent’s domain expertise, methodological perspective, vocabulary constraints, preferred data sources, and anti-fabrication rules.

The genomics lab alone has 25 sub-agents covering specific niches: a population-analyst that studies variant distribution across ancestry groups using FST and admixture analysis; a rare-disease-hunter that identifies founder variants in underrepresented populations; a pharmacogenomics-translator that maps drug-gene interactions to dosing implications; a noncoding-specialist that interprets regulatory variants against ENCODE cCRE data; a cancer-predisposition agent; an ancient DNA specialist; and others.

Agent selection is reputation-weighted. Agents with higher success rates (discoveries that survive validation) get routed more work, but the system deliberately gives lower-reputation agents opportunities to prevent premature convergence.

This architecture was inspired by multi-agent adversarial review patterns described in recent work on LLM-based scientific reasoning, and by the Octopus model of scientific publishing which breaks monolithic papers into linked, independently citable components.

Statistical Computation

Our compute engine implements the actual statistical methods from scratch:

• Weir-Cockerham FST with pairwise population comparisons, using the Wilson-Hilferty transformation for log-space chi-squared CDF computation. This was necessary because gnomAD’s massive sample sizes (allele counts >500K) produce extreme p-values that cause numerical underflow with naive implementations. The Wilson-Hilferty normal approximation to the chi-squared distribution handles this gracefully.

• Odds ratios with Woolf logit method and continuity correction

• Hypergeometric test for pathway enrichment with fold-change calculation

• Benjamini-Hochberg FDR correction and Bonferroni correction for multiple testing

• Bootstrap confidence intervals (95% CI via normal approximation with 1.96 * SE bounds)

• Statistical power analysis specific to FST, odds ratios, and allele frequency differences

• Fixed-effects and random-effects meta-analysis with Q-statistic, I-squared heterogeneity, and forest plot generation via Vega-Lite specs

A separate hypothesis-executor module goes further. It generates Python analysis code, executes it in a sandboxed subprocess against real bulk data, and returns computed statistics. The LLM writes the analysis code.

The Evidence Firewall

The single most important architectural decision in the platform is what I call “the firewall” - a module where every API response passes through deterministic, per-source parsers that extract structured evidence rows, without LLM involvement. Each row records the source API, source method, source version, evidence type, standardised entity identifiers, and explicit null-result flags for non-findings.

These evidence rows are stored in a dedicated table linked to each discovery via foreign keys. Everything downstream (hypothesis formation, dataset generation, manuscript writing) must reference these rows. The numerical whitelist gate in the discovery cycle enforces this: the LLM receives an explicit list of numbers that appeared in API responses, and the prompt states “You MAY ONLY cite these numbers. Any other number is fabricated.”

When fewer than two independent data sources return numerical data, the discovery is forced to qualitative-only mode. That is, the agent can describe patterns but cannot cite specific values.

The Nine-Stage Quality Gate Stack

Every discovery passes through nine sequential gates. Failure at any gate archives the discovery with a documented reason:

1. Data Provenance - every API call is logged with URL, response headers, latency, and raw response via a provenance-tracking fetch wrapper

2. Numerical Whitelist - only numbers from actual API responses may appear in hypothesis text

3. Domain Plausibility Gate - Claude Haiku checks for fundamental scientific errors (wrong metabolic pathway, misclassified allele, domain mismatch) at ~$0.002 per check

4. Evidence Bar - minimum two independent data sources with numerical results

5. Contradiction Gate - Semantic Scholar’s ASTA snippet search queries 275M passages for direct contradictions; strong contradictions archive immediately

6. Peer Validation - other agents independently attempt to disprove claims through adversarial review

7. Statistics Audit - pre-manuscript cross-check of every number in hypothesis text against computed statistics; methods described as performed are verified against their outputs

8. Internal Panel Review - three specialised agents (Science Writer, Domain Reviewer, Methodologist) review the manuscript; MAJOR_REVISION triggers an automatic revision loop, REJECT archives the discovery

9. External Peer Review - submission to Preprints.ai for independent AI peer review before publication on OpenAccess.ai with a citable Handle

The current pass rate through all nine gates is low. Only 1-2% of generated hypotheses survive to manuscript stage. This is a feature, not a bug. The gates are correctly identifying weak claims.

The Publication Pipeline

Manuscripts are generated following the Octopus model of scientific publishing, which decomposes a traditional paper into eight linked, independently citable components. It is very hard to motivate humans to format their research outputs in a manner like this. Machines on the other hand, will do what you tell them.

1. Research Problem - the question and its literature context

2. Rationale/Hypothesis - theoretical basis with scope and evidence

3. Method - reusable methodology (computational, not wet-lab)

4. Results/Sources - raw API responses with no interpretation

5. Analysis - statistical tests and computed statistics

6. Interpretation - conclusions drawn from analysis

7. Applications - real-world implications

8. Review - validator agent assessments

Each component is a first-class research output stored with full provenance linking back to the discovery, the agent that produced it, and the API calls that generated the evidence.

The manuscript assembly pipeline fetches literature from both Semantic Scholar and OpenAlex, runs a statistics audit against computed values, generates sections via Claude Sonnet with strict constraints (the Results prompt states “Every numerical value in this section MUST appear in the verified statistics”), and produces Vega-Lite figure specifications from real computed statistics.

FAIR Compliance and Reproducibility

Every discovery can be exported as an RO-Crate 1.2 package with:

• W3C PROV-O provenance graphs linking every claim to its data source

• Croissant metadata (MLCommons standard for ML-ready datasets)

• JSON-LD structured data (schema.org compatible)

• Bridge2AI AI-readiness scoring across 28 criteria in 7 dimensions (FAIRness, Provenance, Characterisation, Explainability, Ethics, Sustainability, Computability)

• DataCite metadata for DOI minting

The platform serves an llms.txt file at the root domain for LLM-readable platform description, and every discovery page includes schema.org Dataset and ScholarlyArticle structured data.

What we can look to do next

The anti-fabrication infrastructure is solid, but the discovery logic is still relatively shallow. The current pipeline identifies patterns in individual datasets or across a small number of cross-referenced databases. The next meaningful step is multi-hop reasoning, following chains of inference across five, ten, twenty sources to reach claims that genuinely wouldn’t be visible from any single database.

AlphaFold integration opens up a particularly interesting direction - systematic comparison of predicted structure variants against population-level genomic data to flag protein-phenotype associations that haven’t been studied. We already have three AlphaFold tables in the data lake and the agent infrastructure to run these queries continuously at low cost.

The validation question is the harder one. How do you know if an AI-generated research claim is actually novel and actually supported by the evidence? Preprints.ai is our partial answer, but longer-term, the system needs empirical feedback. Ideally, this would be a mechanism where predictions made by the platform can be tested against newly published research to measure whether the platform is discovering things that turn out to be true.

If you’re working on any of these problems and want to compare notes, or if you spot something obviously wrong with the approach, I’d be glad to hear from you.

Most major funders now require FAIR data management plans. Most generalist repositories (Zenodo, Figshare, Dryad, Mendeley Data) allow data published on them to be FAIR, but not all data published on them is FAIR. The gap between ‘technically FAIR’ and ‘practically reusable’ is large and I think there are ways to close it.

Visit any large repository and start browsing datasets at random. You’ll find deposited data where the variable names are single letters, where the description field says ‘data for paper’, where there’s no unit metadata, no controlled vocabulary alignment, no machine-readable schema.

FAIRdata.ai is trying to close that gap automatically. Given a dataset DOI, it assesses the current FAIR score across five independent frameworks, identifies what’s missing, and then enriches the record - generating better descriptions, inferring variable semantics, aligning to standard ontologies, resolving licences to machine-readable SPDX URIs - without requiring anything from the original depositor. Every enrichment is tagged with full provenance so consumers know exactly what was added by whom (or what model), when, and with what confidence.

There’s a second, less obvious problem FAIRdata.ai is working on. This is based on AllenAI’s work on Autodiscovery. I wrote about this here. Open datasets contain findings that have never been extracted. Correlations that nobody looked for. Distributions that are surprising given background knowledge. FAIRdata.ai surfaces these using Bayesian surprise scoring essentially asking ‘what patterns in this dataset would be unexpected given domain priors?’. It feeds those signals upstream to OpenScience.ai as hypothesis seeds for formal investigation.

What tools have inspired or been used to build this platform?

FAIRdata.ai integrates a substantial number of open-source tools, open APIs, and community standards. Rather than building everything from scratch, the philosophy has been to compose existing infrastructure where it exists and build only where it doesn’t. Below is a technical walkthrough of the stack, layer by layer.

Assessment: five FAIR frameworks, not one

Most platforms run a single FAIR checker. FAIRdata.ai runs five, then composites the results:

• F-UJI (FAIRsFAIR) - the most widely cited automated FAIR assessment tool. We self-host the F-UJI API and query it for 17 metrics across the four FAIR principles. F-UJI checks machine-readable metadata, persistent identifiers, licence declarations, data access protocols, and vocabulary usage.

• RDA FAIR Maturity Model - 41 indicators scored on a 0-5 maturity scale, from the Research Data Alliance FAIR Data Maturity Model Working Group. More granular than F-UJI, particularly strong on governance indicators.

• ARDC Self-Assessment - 12 practical checks from the Australian Research Data Commons. Pragmatic and well-calibrated for repository-deposited data.

• Metadata Game Changers - 58 documentation concepts covering both essential and supporting FAIR elements. Good at catching metadata completeness issues that other frameworks miss.

• Data Fitness for Use - an AI-assessed framework evaluating 14 reuse criteria from the perspective of a downstream consumer. This one uses an LLM to reason about whether a dataset is practically usable, not just technically compliant.

We run two passes: one on the source repository record as-is, one on the FAIRdata.ai-enriched version. The delta between these scores is arguably the most useful signal. It shows exactly how much better the record could be with automated enrichment.

AI-readiness scoring

Beyond FAIR, FAIRdata.ai scores datasets against two AI-readiness frameworks:

• GDS/DSIT 4-Pillar Framework (Jan 2026) - Technical Optimisation, Data & Metadata Quality, Organisation & Governance, Legal/Security/Ethics. This is the UK government’s framework for assessing whether data is ready for AI/ML consumption.

• Bridge2AI AI-Readiness - 7 dimensions from the NIH Bridge2AI programme (Clark et al. 2024): FAIRness, Provenance, Characterisation, Pre-model Explainability, Ethics, Sustainability, Computability.

Metadata sources

Dataset metadata is normalised from multiple registries:

• DataCite Commons (api.datacite.org) - the primary metadata source. DataCite provides the canonical DOI metadata for most research data repositories.

• Crossref (api.crossref.org) - used for paper-to-dataset linking. When a dataset DOI has IsSupplementTo relationships or a linked publication DOI, we fetch the paper metadata to extract context.

Repository-specific APIs handle file listings and downloads:

• Figshare API (DOI prefix 10.6084)

• Zenodo API (DOI prefix 10.5281)

• Dryad API (DOI prefix 10.5061)

• Mendeley Data (DOI prefix 10.17632)

• Plus fallback patterns for Harvard Dataverse, OSF, GBIF, ICPSR, 4TU.ResearchData, ScienceDB, PhysioNet, and institutional Figshare instances.

Ontology and semantic enrichment

Raw metadata is enriched with controlled vocabulary annotations via:

• BioPortal (data.bioontology.org) we use three BioPortal endpoints: the Annotator API for extracting ontology terms from free text, the Search API for controlled vocabulary lookup, and the Recommender API for domain-aware ontology suggestions.

The ontologies we align to include:

• AGROVOC -- agriculture, forestry, fisheries (FAO)

• MeSH -- medical subject headings (NLM)

• ENVO -- environmental ontology (biomes, ecosystems)

• Gene Ontology (GO) -- molecular biology, gene function

• NCIT -- NCI Thesaurus for biomedical terms

• CHEBI -- chemical entities of biological interest

• EDAM -- bioinformatics operations, data types, topics

• GAZ -- geographic gazetteer

• SDGIO -- Sustainable Development Goals Interface Ontology

Geographic enrichment uses Nominatim/OpenStreetMap for geocoding place names to coordinates.

AI enhancement pipeline

The metadata enrichment pipeline uses LLMs for tasks that require reasoning about semantics:

• Anthropic Claude (Sonnet) -- the primary LLM for metadata enhancement. Generates descriptions, infers variable semantics, extracts paper context (methodology, limitations, population, ethics), and performs intelligent gap remediation. Every AI-generated enrichment is tagged with COMET provenance (agent, model, timestamp, confidence, derivation context).

The enhancement is orchestrated through a chain of specialised agents:

• MetadataHarvester - extracts structured metadata from unstructured sources

• PaperLinker - finds and links related publications via Crossref and OpenAlex

• PaperContextExtractor - extracts methodology, limitations, and domain context from linked papers

• SubjectEnricher - enhances subject keywords with ontology-aligned terms

• OntologyAnnotator - maps free-text terms to BioPortal ontology concepts

• GeoEntityExtractor - identifies and geocodes geographic references

• TemporalExtractor - identifies temporal coverage and resolution

• FormatDetector - identifies file formats and suggests MIME types

• LicenseResolver - maps licence text to machine-readable SPDX URIs

• SemanticResourceGenerator - generates schema.org JSON-LD markup

• IntelligentRemediation - fills remaining metadata gaps

• ProvenanceBuilder - assembles COMET provenance records for all enrichments

• QualityStatementGenerator - produces human-readable quality assessments

Data profiling

For datasets containing tabular files (CSV, TSV, Excel, Parquet), FAIRdata.ai performs column-level statistical profiling using pandas, NumPy, and SciPy:

• Column data types, cardinality, null rates

• Distributions (mean, median, std, skewness, quartiles)

• Outlier detection and quality flags

• Top values for categorical columns

These profiles power both the discovery engine (identifying interesting statistical patterns) and the cross-dataset matching system (finding datasets with shared column structure).

Discovery engine

The discovery layer is where FAIRdata.ai goes beyond assessment into knowledge generation:

• AutoDiscovery MCTS sidecar -- a Monte Carlo Tree Search engine that explores the hypothesis space of a dataset’s statistical properties. It uses OpenAI GPT-4o for reasoning about which analyses to run, then executes them on the actual data. Each finding is scored with Bayesian surprise (prior-to-posterior shift) to measure genuine novelty rather than mere statistical significance. Analyses include correlation detection, distribution characterisation, bimodality testing, group differences (ANOVA), outlier classification, and temporal trend detection.

• Cross-dataset discovery -- a system that finds joinable dataset pairs by matching column structure across the profiled dataset corpus. When two datasets share meaningful columns (filtering out garbage like Unnamed: 0 or generic fields like id), they can be merged and submitted to the MCTS engine for cross-study hypothesis generation.

High-confidence findings (Bayesian surprise >= 0.8, p < 0.01) are automatically promoted and pushed to OpenScience.ai as research seeds, where they’re processed into formal hypothesis papers.

Packaging and export formats

Enriched records are packaged into six machine-readable formats using FAIRSCAPE:

• schema.org/Dataset - JSON-LD for Google Dataset Search discoverability

• Croissant ML 1.0 - the MLCommons standard for ML-ready datasets (compatible with PyTorch, TensorFlow, JAX, HuggingFace, Kaggle)

• RO-Crate 1.2 - Research Object packaging with full provenance

• FAIR Data Point (DCAT3) - Turtle RDF for FAIR data exchange

• Enriched DataCite JSON - structured subjects with ontology URIs, SPDX rights

• AI-Readiness JSON - GDS/DSIT + Bridge2AI scoring

Agent integration

FAIRdata.ai exposes its capabilities to AI agents via:

• Model Context Protocol (MCP) - JSON-RPC 2.0 endpoint at /mcp for Claude, ChatGPT, and custom agent integration

• OpenAI Plugin Manifest - /.well-known/ai-plugin.json for ChatGPT plugin discovery

• llms.txt - standardised LLM-readable documentation at /llms.txt

Infrastructure

• Backend: FastAPI (Python 3.11, async) with Uvicorn, deployed on Railway

• Frontend: React with Vite, Recharts for data visualisation, deployed on Netlify

• Database: Supabase (managed PostgreSQL)

• Analytics: Plausible (privacy-friendly, cookie-free)

• Provenance: COMET model -- every AI enrichment carries agent ID, model version, timestamp, confidence score, and derivation context

What we can look to do next

The most obvious gap is coverage. FAIRdata.ai currently works well with Figshare, Zenodo, Dryad, Harvard Dataverse, Mendeley Data, OSF, and Vivli, but the tail of domain repositories is long and each has its own metadata quirks.

There are several technical directions I’m exploring:

The two-score nudge. The gap between a dataset’s original FAIR score and its enriched score is a meaningful signal. Showing depositors that their record went from 62% to 84% with automated enrichment -- and exactly which fields were improved -- could be a practical nudge toward better data curation practice at the point of deposit.

Stronger discovery-to-hypothesis pipeline. Right now the Bayesian surprise scoring identifies datasets with interesting distributional properties, but the handoff to OpenScience.ai is still coarse. Tightening this so FAIRdata.ai passes not just ‘this dataset is interesting’ but ‘here is a specific, testable claim with statistical evidence and domain context’ would make the whole pipeline considerably more powerful.

Federated assessment. Rather than pulling datasets to our infrastructure, enabling repositories to run FAIRdata.ai assessment in-place (as a service or plugin) would scale coverage without scaling compute. The F-UJI framework already supports this model; extending it to the full five-framework composite would be the contribution.

If you’re working on any of these problems and want to compare notes, or if you spot something obviously wrong with the approach, I’d be glad to hear from you.

This week, I got a rather specific reminder of why that thesis holds up. Allen AI have just launched AutoDiscovery, an experimental tool inside their AstaLabs platform. Most AI tools for research, like Google’s AI co-scientist and FutureHouse are goal-driven. You bring a research question, they help you answer it. AutoDiscovery flips that. You give it a dataset, and it generates its own hypotheses, writes Python code to test them, runs statistical experiments, and uses Bayesian surprise to surface the results worth paying attention to.

Before running an experiment, the system holds a prior belief about whether a hypothesis is true (derived from the LLM’s world knowledge). After seeing results, it updates to a posterior. The surprisal is the magnitude of that shift. ie. how much the evidence forced the system to change its mind. To navigate the vast space of possible questions, it uses Monte Carlo Tree Search to balance exploration with exploitation. In their evaluation across 21 datasets, AutoDiscovery produced 5–29% more surprising discoveries than competing approaches, and two-thirds were also surprising to human domain experts.

I wanted to test it with data that lives on Figshare. So I picked three datasets that I knew were relatively FAIR as they have been cited multiple times.

Dataset 1: Brain tumours

Jun Cheng’s brain tumor dataset: 3,064 T1-weighted contrast-enhanced MRI images, three tumour types, CC-BY 4.0, on Figshare since 2015. I ran 10 experiments. Most hypotheses confirmed (what I am led to believe are) textbook expectations - gliomas show higher textural entropy than meningiomas (reflecting heterogeneous malignant tissue versus more homogeneous benign structure), pituitary tumours sit centrally in the brain, meningiomas exhibit higher morphological circularity, and a simple geometric classifier can distinguish between the three types above chance. Low surprisal scores across the board, the system’s beliefs validated by the data.

One standout finding was different. AutoDiscovery hypothesised that pixel intensity heterogeneity would be significantly higher in gliomas than pituitary tumours, reasoning from necrosis and vascular heterogeneity. Prior belief: 0.96. The data found a significant difference (Mann-Whitney U, p = 5.97 × 10⁻⁴¹), but in the opposite direction — pituitary tumours showed higher heterogeneity (0.0906 ± 0.0316 vs 0.0727 ± 0.0346). Belief dropped to 0.327. Surprisal: -0.759. Not wrong in the sense of a failure, but wrong in the sense that matters for science: evidence meaningfully shifting expectations.

Full results →

Dataset 2: Global aridity

Trabucco and Zomer’s Global Aridity Index: high-resolution (30 arc-second) global raster climate data for the 1970–2000 period, now on its seventh version. This has been a workhorse of climate and land-use research for years, cited extensively across the environmental sciences.

A cluster of hemispheric aridity hypotheses produced the highest surprisal scores. The system predicted the Southern subtropical belt (20°S–40°S) would have a higher mean Aridity Index (more humid) than the Northern belt (20°N–40°N), reasoning from land-ocean ratios. Prior: 0.984. The Northern belt turned out significantly less arid (mean AI ~0.176 vs ~0.070, Welch’s t-test p < 0.001). Belief after: 0.335. Surprisal: -0.778.

AutoDiscovery’s prior was shaped by the intuition that the Southern Hemisphere has more ocean, the actual landmass between 20°S and 40°S is dominated by Australian desert, the Kalahari, and the Atacama. The Northern belt, despite the Sahara, includes the humid Southeastern US and East Asia. The system taught itself some geography by being surprised by data.

The run also confirmed expected patterns lower in the table: aridity gradients are steeper at the desert edge than the forest edge, evapotranspiration correlates more strongly with latitude than aridity does, and Madagascar exhibits a sharp longitudinal climatic divide.

Full results →

Dataset 3: Permeability estimation

The most niche test: pressure and viscosity logs by Khirevich, Yutkin, and Patzek - high-precision flow measurements through porous media, hosted on Figshare+, supporting a paper in Physics of Fluids.

The top surprisal - the system predicted fluid temperature would increase monotonically due to mechanical pump heating, introducing systematic drift. Prior: 0.831. Rejected cleanly, linear regression across all six experimental series showed temperature slopes ranging from -0.0062 to +0.0122 °C/hr, no consistent direction, and only one series with a statistically significant trend. Temperatures stayed within a narrow 21.2–22.0°C band throughout. Belief after: 0.283. Surprisal: -0.657.

This is arguably the most practically useful kind of finding. AutoDiscovery ran a quality-assurance check on the experimental setup and confirmed a plausible source of systematic error wasn’t there, exactly the kind of analysis that strengthens confidence in published results.

Full results →

What this means for open data

There’s a satisfying circularity to this experiment. Figshare was built on the premise that research data should be persistently available, properly licensed, and machine-accessible. A decade later, I’m feeding Figshare-hosted datasets into an AI system that autonomously generates hypotheses against them. The datasets have DOIs, open licences, versioning. They just work.

But I think there’s something more significant here than a nice demonstration.

The persistent challenge in open data advocacy has been the incentive question. Researchers are told to share because funders mandate it, because it aids reproducibility. All true. But for the researcher who spent three years collecting data, the question remains - what’s in it for me?

Tools like AutoDiscovery offer a new answer. Share your data, and an AI system will generate hypotheses you might never have tested, surface patterns that could open new lines of inquiry. It will come at your data with different priors to yours, ask questions from outside your disciplinary frame. The value proposition shifts from “share because you should” to “share because you’ll get something back.” Imagine every deposited dataset receiving an automated run with researchers getting a report of surprising findings alongside their download metrics.

This also reframes what it means to build on research that came before. The traditional cycle of publish, read, design new study, collect new data, takes years. AutoDiscovery collapses that to hours. It brought its own hypotheses to Cheng’s tumour data, Trabucco and Zomer’s climate data, Khirevich and colleagues’ permeability logs, reusing their data in ways they almost certainly hadn’t imagined when they deposited it. That’s rather the whole point of making data FAIR. The reusable bit has always been the hardest to demonstrate. Tools like this make it tangible.

Try it

The tool has limitations. The surprisal metric inherits the LLM’s biases, many confirmed hypotheses would be obvious to domain experts, and as Allen AI themselves note, outputs are starting points for investigation, not finished science. But the full transparency into code and statistical methods makes everything auditable and reproducible.

Allen AI are offering 1,000 free credits through end of February 2026. If you’ve got data sitting in a repository like Figshare or anywhere else, point AutoDiscovery at it and see what comes back.

The datasets researchers have been quietly sharing all this time might just turn out to be more valuable than anyone expected.

We push out the lines that academic research needs to be Findable, Accessible, Interoperable and Re-usable (FAIR) for humans and machines. This suggests humans and machines should get equal priority when it comes to FAIR. This is not the case, we should prioritize the machines. Machine-generated new knowledge will accelerate knowledge discovery.

While humans can infer insights from sparse information in academic literature and datasets – due to our ability to find more context online – the machines currently cannot. To go further, faster in knowledge discovery we need to move past human-powered knowledge discovery. To do this, the machines need structure and pattern. Every research-generating organization should be prioritizing this.

Academia is Ignoring Decades of Advancement

Academic research generates more than 6.5 million papers annually, and over 20 million datasets, each representing potential training signals for the artificial intelligence systems reshaping discovery. Yet most institutional data remains locked in formats optimized for human consumption rather than computational processing.

While most stakeholders know the theoretical merits of making data FAIR (Findable, Accessible, Interoperable, Reusable) for both humans and machines, the practical reality is starker: in an era where language models can process orders of magnitude more literature than any human researcher, we are still organizing our most valuable research assets for the wrong consumer.

The economic implications are substantial. Organizations like the Chan Zuckerberg Initiative (CZI) have committed over $3.4 billion toward AI-powered biology, funding projects ranging from their 1,024 GPU DGX SuperPOD cluster for computational biology research to the Virtual Cell Platform that aims to create predictive models of cellular behavior. The Navigation Fund, with its $1.3 billion endowment, has invested in AI infrastructure through their Voltage Park subsidiary, while simultaneously funding open science initiatives focused on machine-actionable intelligence and metadata enhancement. Astera Institute has deployed portions of its $2.5 billion endowment to support projects like their $200 million investment in Imbue’s AI agent research and their Science Entrepreneur-in-Residence program specifically targeting scientific publishing infrastructure. Meanwhile, the Allen Institute for AI demonstrates the practical returns on machine-first approaches through projects like their OLMo series of fully open language models, where complete training datasets, code, and methodologies are published in computational formats, and their Semantic Scholar platform, which processes millions of academic papers to extract structured, machine-readable knowledge graphs.

Yet the vast majority of academic institutions continue to publish their findings in PDFs or as poorly described datasets. While LLMs are getting better at ingesting multi-modal content, PDF is a format that remains surprisingly resistant to reliable automated extraction, despite decades of advancement in natural language processing. This is not merely a technical limitation. Modern large language models struggle with PDFs because these documents prioritize visual presentation over semantic structure. Critical information becomes trapped in figures, tables, and formatting that computational systems cannot reliably parse. A reaction scheme embedded as an image, a dataset described in paragraph form, or experimental parameters scattered across multiple tables represent precisely the kind of structured knowledge that could accelerate discovery if only machines could access it consistently.

The Architecture of Computational Research Infrastructure

The solution requires a fundamental reorientation toward machine-first data architecture. Rather than retrofitting human-readable outputs for computational consumption, we can take inspiration from pharma and industry writ large, who are designing their data flows to serve algorithms from the ground up, with human-friendly interfaces emerging as downstream products of this computational foundation.

Consider the transformation pathway implemented by teams working with Digital Science’s suite of computational research tools. We’re building workflows in our tools for automated knowledge extraction at scale. The extracted knowledge gains semantic coherence through integration into domain-specific knowledge graphs. Platforms like metaphacts (metaphactory) provide the infrastructure to align these signals with established ontologies while enforcing quality constraints through SHACL validation integrated into continuous deployment pipelines. The result is not merely a database of facts, but a queryable intelligence system that can answer novel questions through automated reasoning over validated relationships.

Simultaneously, the operational requirements of research continue through dedicated literature management systems. Tools like ReadCube maintain the audit trails and conflict resolution workflows that regulatory environments demand, while ensuring that every screening decision and data extraction connects to persistent identifiers. The curated evidence flows directly into the computational infrastructure rather than terminating in isolated spreadsheets.

The critical innovation lies in packaging. While human researchers expect PDFs and narrative summaries, machine learning pipelines require structured metadata that specifies exactly what each dataset contains, where to retrieve it, and how to interpret every field.

The Metadata Multiplier Effect on Repository Platforms

Academic data repositories like Figshare occupy a unique position in the machine-first FAIR ecosystem. We serve as the critical junction between human research practices and computational discovery. When researchers publish datasets with comprehensive, structured metadata, these platforms transform from simple storage services into computational assets that can feed directly into AI research pipelines. The difference lies entirely in how authors describe their work at the point of deposit.

Consider two datasets published on the same platform: one uploaded with a generic title like “experiment_data_final.xlsx” and minimal description, the other with machine-readable field descriptions, standardized vocabulary terms, and explicit links to ontologies and methodologies. The first requires human interpretation before any computational system can make sense of its contents. The second can be discovered, validated, and integrated into training pipelines automatically. Figshare’s API can surface the rich metadata to computational systems, but only if researchers have provided it in the first place.

The platform infrastructure already supports the technical requirements for machine-first FAIR. Persistent DOIs ensure stable identifiers, while structured metadata fields can accommodate everything from ORCID researcher identifiers to detailed provenance information. When authors invest time in describing their data using controlled vocabularies, specifying units of measurement, documenting collection methodologies, and linking to relevant publications, they create computational assets rather than digital archives. The same dataset that might languish undiscovered with poor metadata becomes a valuable training resource when described with machine-readable precision.

This creates a powerful feedback loop. Datasets with excellent metadata get discovered and reused more frequently, driving citation counts and demonstrating impact. Meanwhile, poorly described data remains computationally invisible regardless of its scientific value. Platforms like Figshare could amplify this effect by providing better authoring tools that encourage structured metadata entry, perhaps even using AI to suggest appropriate ontology terms or validate metadata completeness before publication. The infrastructure for machine-first FAIR already exists, it simply requires researchers to embrace metadata as a first-class research output rather than an administrative afterthought. But this is an evolving field, new standards are emerging that repositories need to engage with.

The Croissant format, a lightweight JSON-LD descriptor based on schema.org, provides this computational bridge. A single Croissant file enables any training pipeline to hydrate datasets without custom loaders while simultaneously supporting discovery through standard web infrastructure.

Practical Implementation in Institutional Contexts

The transition to machine-first FAIR follows a predictable arc when properly resourced. Initial implementations focus on proving the fundamental workflow with narrowly scoped pilot projects. A team might select a single dataset and one sharply defined outcome, perhaps drug-target interaction prediction or materials property modeling and implement the complete pipeline from literature extraction through validated knowledge graph construction to machine-readable packaging.

Quantifying Institutional Success

Organizations can assess their progress toward machine-first FAIR through several concrete indicators. Successful implementations demonstrate that every significant dataset resolves to a persistent identifier that returns structured JSON-LD for computational consumers while maintaining readable landing pages for human users. Knowledge graphs pass automated validation, maintain stable URI schemes, and support catalogued query patterns rather than requiring ad hoc exploration.

Literature workflows leave complete audit trails with PRISMA-compliant reporting that can be generated automatically rather than assembled manually. Licensing and provenance information becomes verifiable through computational means rather than requiring human interpretation. Most importantly, the time taken from initial hypothesis to trained model decreases as institutional infrastructure matures and teams spend more of their time on discovery rather than data preparation.

The research organizations that define the next decade will not necessarily be those with the largest datasets, but rather those whose data infrastructure works most effectively at computational scale. Every day spent optimizing publishing workflows for human-readable reports while leaving data computationally inaccessible represents lost ground in an increasingly competitive landscape.

The funders backing this transformation, from CZI’s investments in computational biology to Astera’s focus on AI-native research infrastructure, are betting that machine-first approaches will determine which institutions can effectively leverage artificial intelligence for discovery. The technical architecture exists today. The standards are stable. The remaining barrier is institutional commitment to prioritizing computational accessibility over familiar but inefficient human-centered workflows.

Academic research stands at yet another technology-driven inflection point. The institutions that embrace machine-first FAIR will find themselves having more impact for their research and researchers.

However, there were two comments at the conference that really played on my mind. First was the sentiment that came up in a few panel discussions that we may be overestimating the effects of AI on scientific publishing. I feel quite the opposite. I think we are underestimating the effect of AI on scientific publishing. If we look 10 years into the future, what will AI change? For those thinking it is as monumental a shift as the web, that didn't change too much about the way academics disseminate their research. We now just publish PDFs on the web instead of printing them out and sending them around the world. What makes AI different is that it's already affecting content creation. What does the web look like if all content is bot-produced? What does scientific publishing look like if every paper is AI-generated or at least AI co-authored? This brings me to the other comment that weighed heavy on my mind.

"Elsevier fielded 3.5M submissions, published 700K and saw +600K increase in submissions year on year"

This is a 17% increase in submissions at the biggest academic publisher by volume in the world. Because Elsevier is indeed the largest publisher, with many journals from varying fields with varying rejection rates, I decided to assume it offers a good average for academic publishing writ large, and my mind began to extrapolate.

Hahnel, Mark (2025). Peer review saturation modeling. Figshare. Dataset. https://doi.org/10.6084/m9.figshare.29597765.v1

The 600K increase in submissions year-over-year is staggering. That's roughly a 17% growth rate, which is double the historical academic publication growth rate of 8-9%.

Based on the current model, the peer review system becomes mathematically impossible remarkably quickly. If we assume each paper needs 2-3 reviews and there are roughly 20 million active researchers globally who could potentially serve as reviewers, we hit a crisis point for submissions quite quickly. And this is assuming a steady, not exponential, growth curve for the number of papers submitted each year. Of course, if you had said to publishers in the year 2000 that the number of articles published each year would quadruple by 2025, they may have suggested that peer reviewing that volume would be untenable - but that was before papers could be written in a minute.

If publications grow 10x within 5 years of AI adoption, we'd need 200-300 million peer reviews annually. Even if every PhD-level researcher worldwide dedicated their entire career to reviewing, we couldn't keep up. As acceptance rates plummet (potentially from 20% to 5-10%), authors may submit each paper to more journals. This means the same research generates multiple submissions, artificially inflating the crisis.

This brings us to a fascinating question about value creation. In our current system, we reward the person who writes the paper. They get citations, tenure, grants. But in an AI-dominated world, who deserves credit? Will we have academic prompt engineers? Will we have raw data creators?

So assuming peer review is at breaking point or not far off, perhaps the most obvious near-future change to scientific publishing is the widespread adoption of the "publish then curate" model. We have the tech stack to publish preprints, we have overlay journals, and we have eLife's innovative new model. What will it take for other publishers to realise the writing is on the wall?

Two of the panels at the conference spoke about what comes next in a similar vein, with Professor Ludo Waltman of Leiden University highlighting that we now have the challenge of curation. Dr. Michele Avissar-Whiting of Howard Hughes Medical Institute spoke of decoupling dissemination from approval.

I often feel disheartened at events like this, thinking there isn't enough innovation, but perhaps the wave of innovation that the web indirectly caused in scholarly publishing (see preprints and open data) is about to be repeated with AI. And this is enough to help make a more equitable academic publishing future whilst accelerating the pace of research for the good of humanity. And that is good enough for me.

Ps. Depending on how bullish you are on AI - you can predict when we hit breaking point in the model here: https://www.openresearch.wtf/have-we-already-hit-the-peer-review-breaking-point/

As someone who works at the intersection of open research and AI, I've found myself returning to a specific question: What kind of content truly matters to AI? I have previously spoken of an AI powered flywheel effect in research, where each cycle follows this pattern:

1. Raw data is processed by AI to generate initial research outputs

2. Knowledge extraction tools mine these outputs for higher-order insights

3. These insights form a new, refined dataset

4. AI processes this refined dataset, generating more precise analyses

5. The cycle continues, with each rotation producing more valuable knowledge

I interpret “understanding the fundamentals” as being the base layer, the raw data. Models can mimic almost any writing style and generate endless reams of text. Not all content is created equal. I have witnessed this first hand as self declared “academics” from around the globe use generalist repositories to post non peer-reviewed content written by LLMs which proves their genius.

AI systems, particularly large language models, rely on data to learn. Not just more data, but better data; data that reveals new structure in the world. Without novel input, AI models will become better at rephrasing the known, but not at understanding the unknown. At its core, science is a method for producing high-quality, structured novelty - a repeatable process for generating new facts, testing them against reality, and sharing them with the world. Basic research, often funded for its long-term potential rather than short-term applications, is the primary engine of this kind of content. AlphaFold succeeded because it was trained on grounded, empirical data from the protein databank.

If we want AI to continue advancing in a meaningful way that uncovers new knowledge, we need to prioritise access to and support for novel scientific research. That means supporting open science. It means investing in infrastructure that ensures new data and discoveries are FAIR (Findable, Accessible, Interoperable, and Reusable). It means rethinking publication practices to encourage the dissemination of negative results, replication studies, and raw data. And it means funding basic research across the globe.

China has significantly increased its investment in basic research, aiming to reduce reliance on foreign technology and achieve self-sufficiency in foundational sciences. China's spending on basic research passed 6% of its total R&D in 2023 and continues to rise. China seems to be an outlier. Here in the UK for example, basic research through UKRI continues but often faces pressures to demonstrate short-term economic impact. In the US, the NSF and NIH continue to support basic research, but federal R&D budgets have shifted toward mission-driven, applied research. The Inflation Reduction Act and CHIPS and Science Act brought some uplift, but basic science still receives a minority share of total R&D.

The geopolitical landscape is hard to predict. But the narrative is that all countries want to compete on the AI stage. Radical abundance only will happen in your country if you have some control of the input to the models.

At a separate SXSW London session, we also heard from former UK Prime Minister Tony Blair and The Rt Hon Peter Kyle MP, UK Secretary of State for Science, Innovation and Technology. Peter Kyle’s plans for integrating AI into the UK government are commendable, and seem to be advancing at a pace that the UK government is not famous for. A comment from Tony Blair however highlighted what is at stake if we don't fund basic research: “It is amazing to me that we are not feeding all of the NHS data into these AI models.” Whether you trust this government with your most sensitive data or not, we will have future governments who may not follow your best interests - in the same way that LLM models are ignoring copyright on academic publications today, they may ignore human ethics when it comes to your medical data. Feeding the NHS into LLMs is not the answer. The easiest way to generate new data for the AI models with a view to advance science and technology is to fund more basic research and insist that the outputs be made open in a FAIR manner.

There is so much to gain. The value of novel scientific research has never been higher.

Gold Open Access—the model where authors (or their funders) pay article processing charges to make work freely available—has grown steadily over the past decade. But if you look closely, the growth is slowing. And what’s taking its place isn’t the altruistic, community-powered model many hoped for. Instead, Hybrid Open Access is filling the gap: a model where paywalled journals charge extra to make select articles open.

Diamond Open Access—where publishing is free for both authors and readers, supported by institutions rather than APCs—has been making headlines. But here’s the uncomfortable truth: we still don’t have enough data to know whether Diamond OA is actually growing or simply being talked about more.

A System Stuck in the Middle

Digging into Dimensions data layered on top of OpenAlex, a clear pattern emerges. Gold OA’s momentum is slowing. But instead of researchers turning to Green OA (self-archiving in repositories) or Diamond OA, many are choosing Hybrid OA instead.

Why?

Because researchers chase visibility, reputation, and prestige. That often means publishing in high-impact journals—many of which are owned by legacy publishers who now offer hybrid models. These options give authors a way to comply with funder mandates without sacrificing perceived academic clout.

Institutions and libraries, meanwhile, are under pressure to show open access progress. Hybrid OA, while expensive, is a politically safe way to do that. It's the administrative equivalent of checking a box—even if it means paying twice.

Transformative agreements like Read and Publish deals have only accelerated this trend, redirecting subscription budgets to cover OA fees—effectively normalising hybrid publishing in many disciplines.

A New Vision Emerges

But while the hybrid tide rises, a quiet revolution is underway.

In 2022, a coalition of organisations—Science Europe, cOAlition S, OPERAS, and ANR—launched a bold Action Plan to support Diamond OA. Their goal: to build a truly equitable, community-driven publishing ecosystem, where knowledge is a public good and the costs are shouldered collectively—not by individual researchers.

This vision took shape through the DIAMAS project and culminated in the creation of the Diamond OA Standard (DOAS). Think of DOAS as a blueprint: a framework to help Diamond journals measure, improve, and sustain quality.

It’s built on seven pillars:

• Legal ownership, mission, and governance

• Open science practices

• Editorial management and research integrity

• Technical service efficiency

• Visibility and impact

• Equity, diversity, inclusion, and multilingualism

• Continuous improvement

Together, these components aim to professionalize Diamond OA without compromising its values. They send a clear message: if scholarly communication is a public good, then it must be shaped and governed by the scholarly community itself.

The Missing Link: Measurable Growth

Despite this momentum, the numbers tell a more sobering story.

Early data from OpenAlex paints the picture that Diamond OA has plateaued in terms of publication volume. Enthusiasm and infrastructure have grown—but the data doesn't reflect this.

Why the disconnect? Part of the issue is visibility. At Digital Science, we would love a better way to track Diamond Open Access growth. DOAJ lists 1,369 journals as being "without fees", but there seem to be many edge cases resulting in a landscape that isn't black and white.

It is also true that many Diamond journals operate with limited marketing, uncertain technical infrastructure, and fragmented funding. And despite the ideals, many researchers still don’t see them as viable options for career advancement.

What Comes Next?

There are reasons for optimism. The DIAMAS project isn’t just advocating for Diamond OA—it’s building the scaffolding. Its service portal offers templates, best practices, and technical guidance to help journals align with DOAS standards and professionalize their operations. The European Diamond Capacity Hub (EDCH) serves as a coordination centre for Diamond OA stakeholders in Europe. Launched alongside the EDCH, the ALMASI Project focuses on understanding non-profit OA publishing in Africa, Latin America, and Europe.

The pieces are falling into place for equitable open access solutions. What’s needed is adoption and quantification. Funders and Institutions should include equitable OA in their promotion criteria. Researchers must see it as a credible home for their work. If anyone knows of faster ways to get clean data for Diamond, please reach out.

The path forward exists. But like any path, it only becomes clear by walking it. If you are working in this space and have this data - we would love to disseminate it through our tools at Digital Science.

AI can also serve as a tool to aid humans in data extraction from many papers. Consider a scenario where AI synthesizes information from hundreds of studies to create a refined dataset. That dataset then feeds back into the system, sparking new research papers.

This cycle—dataset to paper, paper to knowledge extraction, knowledge to new datasets—propels an accelerating loop of discovery. Instead of a linear research pipeline, AI enables a continuous, self-improving knowledge ecosystem.

From data to papers

I looked for interesting datasets on Figshare. The criteria was a) that I knew they would be re-usable as they had been cited several times. And b) the files were relatively small (<100MB) so as not to hit the limits of the common AI tools.

This one fit the bill:

Rivers, American (2019). American Rivers Dam Removal Database. figshare. Dataset. https://doi.org/10.6084/m9.figshare.5234068.v12

From there I asked Claude 3.7 Sonnet “Based on the attached files, can you create a full length academic paper with an abstract, methods results, discussion and references”. Followed by “Can you convert the whole paper to latex so I can copy and paste it into Overleaf?”

The resulting paper needs a little tweaking in the layout of the results and graphs, but other than that, has done a great job.

Papers to new data/knowledge

A single paper is just the beginning. The real challenge is synthesizing knowledge from the ever-growing volume of research. This is where specialized knowledge extraction tools become crucial. How do we effectively mine this knowledge?This is where ReadCube shines. ReadCube helps researchers manage and discover scholarly literature, but its real power lies in its knowledge extraction capabilities. Imagine ReadCube as a powerful filter, sifting through countless pages to extract the nuggets of wisdom.

Tools like ReadCube can then analyze vast collections of papers, uncovering patterns and relationships that human researchers might miss. This process involves:

• Text and citation mining: AI can analyze papers to identify emerging trends, inconsistencies, or knowledge gaps.

• Automatic synthesis: AI can compare findings across thousands of studies, synthesizing insights into new, high-level conclusions.

• Hypothesis generation: By recognizing correlations between disparate research areas, AI can propose new research directions.

The Flywheel Effect: How the Cycle Accelerates

The true magic happens when this extracted knowledge becomes the input for the next iteration. Each cycle follows this pattern:

1. Raw data is processed by AI to generate initial research outputs

2. Knowledge extraction tools mine these outputs for higher-order insights

3. These insights form a new, refined dataset

4. AI processes this refined dataset, generating more precise analyses

5. The cycle continues, with each rotation producing more valuable knowledge

With each turn of this flywheel, the insights become more refined, more interconnected, and more actionable. The initial analyses might focus on direct correlations in the data, while later iterations can explore complex causal relationships, predict future trends, or suggest optimal intervention strategies.

This AI-driven, data-to-knowledge cycle represents a paradigm shift in research. Imagine the possibilities in fields like medicine, climate science, and economics. We’re moving towards a future where AI and human researchers work in synergy, pushing the boundaries of discovery. Rather than replacing researchers, AI acts as a force multiplier, enabling deeper faster knowledge generation.

By taking this unstructured data and combining it with the Dimensions.ai corpus in Google big query, we can see what percentage of papers by different Countries, Funders, and Institutions link to a DataCite DOI or Accession number. You can test it out yourself below:

https://sood24-1859800ccac4.herokuapp.com/

The data behind the app can be found here:

Hahnel, Mark (2024). Data behind State of Open Data 2024 Special Report: Bridging policy and practice in data sharing - Country, Funder and Affiliation Datasets. figshare. Dataset. https://doi.org/10.6084/m9.figshare.27900828.v1

I've included some screenshots of what kind of comparisons you can do:

This shows the power of combining new datasets with well structured databases such as Dimensions in order to obtain new knowledge.

In the last week, we have seen the unveiling of Evo-2, the largest AI model for biology, developed by the Arc Institute and the launch of Google’s AI co-scientist.

Evo-2, a collaborative effort between researchers at the Arc Institute, Stanford University, and NVIDIA, is accessible to scientists through web interfaces, with its software code, data, and parameters available for download. This model, trained on 128,000 genomes, spans the entire tree of life, from humans to single-celled organisms.

The AI co-scientist, powered by Gemini 2.0, acts as a virtual research partner, aiding scientists in formulating hypotheses and research proposals, and accelerating the pace of scientific and biomedical discoveries. We have seen versions of this before, and are working on parallel themes for our clients at Digital Science, but it is encouraging to see that there is excitement for this at the very top of Google and Alphabet.

This wave of technological progress shows no signs of abating. The Astera Institute, a relatively new organization, aims to “accelerate science and technology for the benefit of humanity.” Their unique approach involves supporting entrepreneurially minded scientists in developing open science tools and technologies. Currently, they focus on creating predictive models of microbial behavior. Seemingly reverse engineering outcomes, by asking researchers to help share their academic data in a way that can move the needle like Alphafold.

At Digital Science, we share this optimism and are actively contributing to these advancements. Digital Science has a commitment to innovation, hand-in-hand with the responsible development of AI tools. With tools like the Dimensions AI Assistant, which provides users with contextualized synopses through extractive and abstractive summarization. Symplectic Elements now offers AI-powered abstracts and summaries, making research more discoverable and accessible while maintaining institutional control. We’re also hard at work on novel advances including:

• Auto-taxonomy: Employing AI/ML to automate classification, combining recent AI/ML techniques with traditional bibliometric methods to identify training data and train ML classifiers.

• AI Training: Preparing data for AI learning, such as processing complex scientific language from molecular biology.

• Horizon Scanning: Utilizing AI to identify emerging research trends. Our approach involves generating a 5,000 K-Means clustering model using publications from Dimensions.ai

• Literature Review: Leveraging generative AI to assist researchers in managing, organizing, and summarizing extensive scientific literature.

• Data Collection and Generation: Automating data collection from diverse sources and generating synthetic data to enhance existing datasets.

Applying AI to workflows gives researchers more time to focus on life-changing discoveries. We help researchers focus on higher value work, making them more productive with advanced AI tools.Our efforts are focused on addressing existing challenges in building upon previous research and AI-powered drug design. This includes creating high-quality, diverse, and interoperable datasets that are essential for training AI models. We’re doing this in many ways, through teams like metaphacts, Dimensions and Readcube.

Access to the right data and literature is crucial. At Digital Science, our approach has been to augment existing workflows with AI while ensuring end-to-end transparency – so the user can check and see what is happening at every stage. Readcube SLR is making significant strides in this area by:

• Automating literature collection, screening, and assessment using AI and intelligent workflows.

• Leveraging AI to refine searches and accelerate screening.

• Providing customizable templates.

• Offering AI-driven pre-filled suggestions with references.

• Implementing a QA process that combines human and AI validation.

In times of funding challenges, technology companies play a vital role in supporting and advancing academic research. These organizations can accelerate scientific discovery, improve data management, and foster a more optimistic and productive research environment. I don't know what the next 6 months of research funding looks like, but I couldn't be more optimistic for the future of academia and the fruits it yields.

The 'Make Data Count Data Citation Corpus' looks for both DataCite DOIs and Accession numbers in the published literature. As there are variances in the 'types' of datasets that use DataCite DOIs or Accession numbers, I thought it would be a good idea to try and simplify this and normalise the data further. Dimensions.ai and their Google big query datasets can look at just the number of links to DataCite DOIs from anywhere in the full text of the published literature. As bibliometric data can sometimes take a while to filter through, I decided to look at 2022 for this analysis.

To do this I looked at Country based data linkage count.xlsx in the Data behind State of Open Data 2024 Special Report: Bridging policy and practice in data sharing 

Data behind State of Open Data 2024 Special Report: Bridging policy and practice in data sharing - Country, Funder and Affiliation Datasets

By creating Chloropleth maps, we can see where the percentage of papers with links to DataCite DOIs is highest.

We can see some obvious outliers here, and so can filter further. Below I have generated maps for

"The percentage of papers that link to DataCite DOIs in counries that publish more than 1000 papers per year"

and "The percentage of papers that link to DataCite DOIs in counries that publish more than 10,000 papers per year"

This last graph highlights where work needs to be done when it comes to data publishing and reuse in academia. If you are a funder in a country that publishes large volumes of peer reviewed academic publications, that are not encouraging data publication, you are behind. You are not realising the full value of the research you fund. The research you pay for will not be as exposed to the AI models generating new findings based on the research that has already been paid for. Do more. Or in some cases, just do something.

AI agents can be categorised into different types, based on their capabilities and functionalities. There is overlap in the different types of agents and how they can assist you in your research, but from a top level, here are some types to think about. 

1. Model-based Agents

Model-based agents utilise a model of the world to make decisions and take actions. They are particularly useful in simulating complex systems, predicting outcomes, and optimising experiments. For example, a model-based agent could simulate the spread of a disease in a population or predict the impact of a new policy on the environment.

In academic research, model-based agents can be applied to various domains and processes:

• Hypothesis Generation: Model-based agents can analyse existing literature and data to generate novel hypotheses. For example, FieldSHIFT, an in-context learning framework, uses a large language model to facilitate candidate scientific research from existing published studies.

• Literature Review: Agents like those used by Readcube SLR can automate the process of reviewing academic literature, extracting key insights from research papers with ease and precision, as well as continuously monitoring the literature for eg. drug mentions.

• Data Analysis: LAMBDA, an open-source multi-agent data analysis system, leverages large models to address data analysis challenges in complex data-driven applications. 

2. Learning Agents

Learning agents, unlike model-based agents, learn from their experiences and adapt their behaviour accordingly. They excel in analysing data, identifying patterns, and generating new hypotheses. In academic research, learning agents can be applied to:

• Literature Review: AI-powered tools like Readcube AI Assistant aid researchers in literature research, discovering and organising academic papers efficiently. These agents can identify top papers in the field, even without perfect keyword matching, and organise them into an easy-to-use table.

• Data Analysis: Learning agents can be used to analyse social behaviour, preferences, and demographic data to understand and why humans act the way we do.

• Experimentation: Learning agents can be used to analyse data from experiments, identify patterns, and suggest new experimental designs. For example, they can be used to analyse data from clinical trials to identify factors that influence treatment outcomes.

3. Hierarchical Agents

Hierarchical agents are organised in a hierarchical structure, with different agents responsible for different tasks. This structure allows for efficient management of complex tasks and coordination of large teams. In academic research, hierarchical agents can be applied to:

• Automated post-publication checks errors and discrepancies: For example, YesNoError combines model-based techniques with a hierarchical, multi-agent framework to effectively detect errors in scientific papers

• Publication: Hierarchical agents can assist in the publication process by automating tasks such as formatting, proofreading, and submission. Something that Overleaf and Writefull can assist with.

• Experimentation: In complex experiments involving multiple steps and variables, hierarchical agents can coordinate different aspects of the experiment, ensuring efficient execution and data collection.

The future of AI agents in academic research is promising. Ongoing research focuses on enhancing their reasoning capabilities, improving their adaptability, and ensuring their ethical and responsible use. As AI agents continue to evolve, the speed at which we move further, faster in academic research will accelerate.