This is an article dated June 2026 but has been re-published here.

There is a huge market for Intelligent Document Processing (IDP), and it is projected to grow to somewhere between $20 billion and $90 billion by the early 2030s, depending on which analyst you ask.

Probably driven by companies paying $15–25 per invoice in manual handling costs.

Because I stay close to the tech world, I have watched a wave of small, specialized OCR vision models ship over the past year, mostly from Chinese organizations.

These are not the only new options, though. Document-parsing tools such as LlamaParse and Docling have also come onto the scene.

These new document-parsing models and pipelines now share the space with old-timers such as Tesseract and AWS Textract Structured. Depending on what they are being used for, the price tag for companies can currently range from $0 to around $65 per 1,000 processed pages.

So, I felt it would be useful to properly test 14 of these engines, old and new, on a range of messy documents, including handwritten notes, tables, legacy financial documents, scanned invoices, receipts, charts, old newspapers, and tax forms, to see how they compete.

The idea was to test how they handled two things: text recovery and the ability to preserve useful table structure.

The main questions I wanted answered were: do you really need to pay $65 per 1,000 structured pages, or can you cut that down to a fraction? And do the new specialized OCR models outperform the general ones?

This article is for anyone working in AI engineering, but also for business people who need to understand how to approach this space and optimize their own setup.

TL;DR

OCR is not really a search for the best engine. It is a routing and evaluation problem, and the answer depends on your documents.

Cheap, old tools can handle clean documents. General vision models are better when documents become unpredictable. Expensive structured extraction can be necessary, but there are now cheaper options that perform just as well.

Specialized models trained specifically for OCR do well enough for the price, but can fail sharply outside their distribution, which is impossible to see from benchmarks alone.

The key thing to carry with you is that using one engine for every document can force you to pay far more than necessary, while new technology does not always outperform older, proven tools.

Let’s cover how we got to this conclusion though.

Let’s run through the mechanics first

For anyone new to OCR, or anyone who needs a refresher, I’ll give a brief introduction first.

Then, let’s cover the test setup so you understand the engines, the types of documents I chose, and the metrics I used to decide how well each engine performed.

Explain the OCR space to me

OCR, or Optical Character Recognition, is how a machine turns a picture into machine-readable text. It is simple in principle and, for easier documents, mostly solved. It becomes harder when things get more human.

To give you a quick overview, older OCR systems found text on a page, sliced it into characters, and matched each one against a library of known shapes. Tesseract has done this since the 1980s.

Modern OCR, however, including newer versions of Tesseract, usually uses a neural network that looks at the whole page at once and outputs the document as text. So, if your document is a clean PDF or a high-quality scan in a standard font, OCR is mostly a solved problem.

It stops being solved the moment things get messier: photographed receipts, handwritten notes, weird graphs and charts, dense financial tables, or scanned tax forms and loan applications.

Companies need this done well for obvious reasons, as every downstream system acts on the output. The better OCR gets, the more paperwork becomes something a system can reason over instead of something a human has to read by hand.

There is also the fact that if we feed AI systems badly parsed documents, everything that comes after will be difficult to trust.

This raises the question I wanted to test: can small open-source models actually do the work that expensive APIs charge for, or should we look to general vision models to handle OCR too?

The docs, the engines, and the metrics

This experiment needs to cover a few things before we move on to the results: specifically what engines we used, what docs we tested with, and how we decided who won.

For the engines, I wanted a lineup that covered all the choices I talked about, this meant: old and new, open and closed, local and cloud, specialized and general.

Tesseract became the classical choice. It runs locally and is very fast. Then I added two document-parsing pipelines: Docling and Marker. Docling is slower but runs on CPU, Marker is open-weight but needing a GPU to run fast, which shows up later in the price.

Then for the new wave of specialized open OCR models: GLM-OCR, PaddleOCR-VL, DeepSeek-OCR, and MinerU 2.5 (a borderline case, really a pipeline with a VLM inside). I picked them off OpenDataLab’s OmniDocBench leaderboard, where they ranked first, second, fourth, and fifth.

Note for PaddleOCR-VL, we ran the two-part system pipeline.

I hosted them on Modal and served the applicable ones with vLLM, batching to speed things up. I counted the scale-up time when measuring latency later.

I also added one closed purpose-built model, Mistral OCR, which I’d heard good things about.

On the open side, I used Qwen3-VL (8B, from Alibaba), also hosted on Modal with the rest of the smaller models. I should flag that I gave it a plain transcription prompt rather than the optimized serving setup it was designed for, so I may not have given it a fair shot.

On the closed side, for the general models, I picked Gemini Flash 3.1 Lite (currently first on the IDP Leaderboard, the western counterpart built on OmniDocBench v1.5) and Claude Sonnet 4.6, at sixth.

For the cloud document services: LlamaParse and AWS Textract, in both its text and structured forms. Structured Textract can do far more than I asked of it. I only tested its text accuracy across the board and its table extraction against eight of the other engines.

Let’s turn to the documents. I picked seventeen document types that were either easy, medium, or hard. Ninety-three files in all.

Easy was the stuff OCR mostly solved years ago: clean invoices and receipts. Medium came largely from the OmniAI OCR Benchmark dataset: bank statements, medical notes, photographed receipts, shipping documents, tax forms.

Hard was chosen when things turned more difficult: charts, forms, handwritten notes, weirdly scanned financial tables, legal papers, newspapers, and old legacy reports.

Some docs were really quite difficult, such as the legacy scanned docs you see below, and this was just because I was curious if some could actually do it well.

Some of these images came with gold ground truth and some didn’t, and the ground truth I did have wasn’t always consistent, some files labeled correctly, some not, which is why we should briefly cover the metrics too.

Since every engine emits different markup, the usual scoring didn’t quite fit. One might pick Precision and Recall for a case like this.

Precision looks at how many of the OCR output’s words actually match in the ground truth (GT) while Recall measures how many times each GT word was captured.

Precision would punish engines that emit markdown structure the GT doesn’t contain, furthermore the GT sometimes skipped labels entirely which would punish the engine unfairly. Recall would measure the words but punish the frequency.

So, I added on a third metric called Coverage.

I just wanted to measure how much of the ground truth shows up somewhere in the engine’s output. It isn’t perfect, but it tells me whether an engine caught most of what mattered, without penalizing it for gaps that were the ground truth’s fault rather than the engine’s.

For the documents with no gold ground truth at all, I fell back on an LLM judge, with Gemini 3 Pro as the base model as the judge, and anyone who’s used an LLM judge knows this is fickle business.

What this experiment showed

Let’s briefly run through the results, look at which engines work best where, and then cover the other oddities from this test that are worth learning from.

Let’s first map out the results

We mapped every document against the Coverage metric to build a scatter chart, and tracked latency on a separate chart. The thing a generalized chart can’t tell you though is that the engines failed in different ways.

The bubble graph showed that most engines fall somewhere in the middle top, with two outliers on both sides of it.

Gemini Flash and Textract Text did very well across the board with some edge cases. Some of the open source models like GLM and Qwen3 VL did just below the APIs but what stood out is the Coverage for GLM while being fast (it’s a 0.9B model after all).

Sonnet performed the highest but also with a steeper price tag.

When we also mapped latency, some of the models turned out to be very slow, but again most wound up somewhere in the middle or to the left (if small and open source).

Tesseract, Claude Sonnet 4.6, and Docling stand out. Tesseract was incredibly fast compared to all other engines. It should be your go-to for easier documents. The specialized models with 0.9B — 3B params were quite fast when optimized for the correct GPU running on Modal.

These graphs generalise across all the documents, but I did separate the results based on the type and difficulty level.

To start with the easy docs. On invoices, every engine did well, Tesseract especially. Receipts knocked everyone down a little.

The one outlier was Docling, which struggled across a lot of the categories, even the easy ones.

When I looked into the Docling failures I found things like Ifjointreturn instead of “joint return,” and worse, strings like City,wrostffielfouaveaoreignadresalcomletacesb. DeepSeek also missed key details here like invoice number and date, which is why its number sits low.

Textract was the best engine on a lot of the medium types, along with Claude Sonnet 4.6 and Mistral OCR.

On the harder types, Gemini Flash started rising, beating Textract on forms and handwritten notes, matching it elsewhere. It did remarkably well everywhere. Tesseract and Docling failed hard on handwritten, and forms were tough for them too.

For the docs with no ground truth (newspapers, legal, reports, some scanned legacy documents) we used an LLM judge. These are genuinely hard, so it’s no surprise almost everyone failed on the reports and newspapers.

Except Gemini Flash that did reasonably well everywhere. Mistral OCR also did well for newspapers. Gemini Flash won everywhere with the judge, though we used Gemini Pro as the judge so take that with a grain of salt (but I did double check myself).

Before rounding off: I also ran 8 engines against Textract Structured to see how they did on financial tables, extracting an HTML table.

I used Textract Structured’s output as the ground truth for TEDS (Tree Edit Distance Similarity) and scored Claude Sonnet 4.6, LlamaParse, Mistral OCR, Gemini Flash, Marker, MinerU, DeepSeek-OCR, and Docling against it.

Mistral OCR, and LlamaParse, and Marker did very well while being much cheaper. I also ran it through an LLM judge, and the winners were the same three (even before Textract Structured) though I’d want to build that test better before I fully trust it.

Now, let’s talk about what it costs to scale this up, and what would make sense where.

Let’s spell it out: what does best where

Let’s run through what it costs to scale up with these engines, and then based on these docs what you would chose where.

First, the cost of using these engines vary wildly, as you saw before. Sometimes it helps to see the cost not just for one document, but thousands up to a million.

We’re self-hosting on Modal, so these costs come from actual usage there.

We’ve calculated that the GLM OCR 0.9B model does well enough on an A10G while running qwen3_vl, mineru, deepseek_ocr, marker, and paddleocr_vl on an A100.

You can obviously run locally, but my computer wouldn’t allow it and I didn’t want to try it and this gives us numbers to work with.

For the docs we have here, Gemini Flash 3.1-Lite is a clear winner but so is GLM OCR for that price and latency. Mistral OCR did well on structured tables while staying cheap. GLM OCR did very well when we look at latency too (considering we’re self hosting).

Claude Sonnet 4.6 did well too, but it’s slow and expensive comparatively.

Docling is so very slow on my laptop. I’m sure there are ways to speed it up, but it also failed in ways that make it inherently unstable (still a small test though).

Textract is a stable choice, but structured buys you almost no additional text accuracy so if you’re paying that steep markup for structured output, make sure you actually use it. I’m guessing it’s a pretty good business model for them.

So, in general for this very small test: for clean, high-volume print, just use Tesseract. For general heterogeneous production, go Gemini Flash or GLM OCR. For a cost-floor with table structure, test Mistral OCR. For high-stakes docs, route to Sonnet or a larger model.

Since everyone did well in different ways you’ll have to contact me for specifics.

Let me just quickly talk about some things that stood out after doing this experiment.

What this whole experiment taught me

A handful of things surfaced from this that are worth pulling out on their own.

First, if you want to understand how a model or engine will do on your docs, the only way is to test on those docs, you can’t rely on benchmarks to tell you. This was the number one insight this showed.

OCR usefulness depends on your own document mix, layouts, languages, scans, tables, handwriting, and failure tolerance.

Do not pay for structure if you don’t need it. I wonder how many are using certain APIs or models for a reason they can’t justify. Map the cost to understand what you are losing by not using the correct engine for the documents.

Lastly, the failure modes told me more than the averages.

DeepSeek OCR has chart blindness and empty outputs on some docs. While Docling has character errors, word-merging, and column misalignment all stacking together. Tesseract did fine on clean docs (as mentioned) but failed on photo/handwriting altogether outputting garbage.

Caveats to consider

Before we round up, let me cover how this test is ultimately imperfect by naming the issues in the GT, the metrics used, and the sample size.

I covered this in one of the section above, but the ground truth differs between documents depending on the dataset where they were found. In general tokenization artifacts can make correct OCR look worse than it is.

Most engines have different formats, some return plain text, some markdown, some HTML/rich markdown and it’s hard to generalize across all.

We are using Coverage, and then also some other metrics, but these aren’t perfect. Coverage won’t charge the engine if it outputs too much text or the structure of it is off. Though I did find that for the engines that failed, they did so at the start or mid-way through rather than at the end.

This means it’s useful for ranking but not a perfect way to score.

LLM judges are not neutral truth: I’ve covered this in the past, but they are biased and very prompt sensitive.

Then I just need to say that this test is interesting but not that big, the sample size is way too small to use this as a factual study. But, I don’t fully trust these metrics nor the judge so it was the only way for me to be able to double check the results on my own without this turning into a year long project.

So, this test is useful for direction and getting a sense of what works, but for getting a sense of your use case, you need to run it through with your specific docs.

Lastly, latency and reproducibility is unstable. Serverless cold starts make timing noisy, and API models can silently change over time.

Like always with these articles, it takes quite a bit to do an experiment like this, but I do it because I’m genuinely curious.

What it looks like though is that OCR seems to be a routing problem, and perhaps an evaluation problem. Classify your docs and run them through several engines, then try to build a decent router and validator in your pipeline to escalate failures and then log the costs.

But we can say there are now new options for harder documents that can be just as fast and cheaper than before.

❤

All datasets used in this benchmark are publicly available and sourced from HuggingFace. Licenses include MIT, CC-BY-4.0, and fair-use frameworks (UCSF Industry Documents Library) covering research, scholarship, and education. No source documents are reproduced — datasets were used solely as evaluation inputs to measure OCR engine performance.

This is an article dated February 2026 but has been re-published here.

Last year I wrote an article about overengineering a RAG system, adding fancy things like query optimization, detailed chunking with neighbors and keys, along with expanding the context.

The argument against this kind of work is that for some of these add-ons, you still end up paying 40–50% more in latency and cost.

So after some back and forth, I decided to test two pipelines, one with query optimization and neighbor expansion, and one without. Let’s call it a naive vs. a complex pipeline (even if both use good engineering practices).

After running a few evals on a synthetically created dataset to test both of these pipelines, I couldn’t see any differences in results. They performed exactly the same.

The issue was that the dataset was just too synthetic.

So, we ran it on a more real dataset with questions grabbed from humans. This turned out to show something entirely different. The ‘complex’ pipeline won by a large enough margin on most tests.

But although it won on the metrics we had set out to test, such as groundedness, faithfulness, answer quality, we started witnessing new issues like over-synthetication that we hadn’t predicted before.

Thus, we understood that both had different failure modes.

This article will take you through the experiment, the results, and what we can learn from it. Great if you’re just starting to learn how to do evals or if you’re just curious about what we found here.

We’ll go through the setup, the experiment design, three different evaluation runs with different datasets, how to understand the results, and the cost/benefit tradeoff.

Please note that this experiment is using reference-free metrics and LLM judges, which you always have to be cautious about. You can see the entire breakdown in this excel document.

If you feel confused at any time, there are two articles, here and here, that came before this one, though this one should stand on its own.

The intro

People continue to add complexity to their RAG pipelines, and there is a reason for it. The overall design is flawed, so we keep patching on fixes to make something that is more robust.

Most people have introduced hybrid search, BM25 and semantic, along with re-rankers. This has become standard practice. But there are more complex features you can add.

The pipeline we’re testing here introduces two new features, query optimization and neighbor expansion, and tests their efficiency.

Query optimization rewrites your question (or query) into something that will more likely match information in the database. Such as “I want you to find out about Laura Palmer” might get re-written to “Laura Palmer.”

As for neighbor expansion, this refers to the ability to expand up and down in a document when a chunk has been matched. This should help when the answer is spread across multiple chunks in a section.

There is a reason we add these features, and if you want to know exactly how they work you can read this article, but both can backfire (which is what we’ll discuss here).

We’re using LLM judges and different datasets to evaluate automated metrics like faithfulness, along with A/B tests on quality, to see how the metrics move and change for each.

The introduction will walk through the setup and the experiment design.

The setup

Let’s first run through the setup, briefly covering detailed chunking and neighbor expansion, and what I define as complex versus naive for the purpose of this article.

The pipeline I’ve run here uses very detailed chunking methods, if you’ve read my previous article.

This means parsing the PDFs correctly, respecting document structure, using smart merging logic, intelligent boundary detection, numeric fragment handling, and document-level context (i.e. applying headings for each chunk).

I decided not to budge on this part, though this is obviously the hardest part of building a retrieval pipeline.

When processing, it also splits the sections and then references the chunk neighbors in the metadata as you saw above. This allows us to expand the content so the LLM can see where it comes from.

For this test, we use the same chunks, but we remove query optimization and context expansion for the naive pipeline to see if the fancy add-ons are actually doing any good.

This means that we only send in on average 6 chunks for the “naive” pipeline and an average of 55 chunks for the “complex” pipeline. The chunks themselves are quite small, cut off after each paragraph.

I should also mention that the use case for this was scientific RAG papers. This is a semi-difficult case, and as such, for easier use cases some of these features may not apply (but we’ll get to that later too).

You can see the papers this pipeline has ingested here, and read about the use case here.

To conclude: the setup uses the same chunking, the same reranker, and the same LLM. The only difference is optimizing the queries and expanding the chunks to neighbors.

The experiment design

We have three different datasets that were run through several automated metrics, then through a head-to-head judge, along with examining the outputs to validate and understand the results.

We do this because each one can catch a different way the system can lie to you, and no single dataset or metric exposes all failure modes.

I started by creating a script that would generate around 256 questions from the ingested corpus. You can see this dataset here.

This can be a good way to validate that your pipeline works, but if it’s too clean it can give you a false sense of security.

Note that I did not specify how the questions should be generated. This means I did not ask it to generate questions that an entire section could answer, or that only a single chunk could answer.

The second dataset was also generated from the corpus, but I intentionally asked the LLM to generate messy questions like “what are the plz three types of reward functions used in eviomni?”

The third dataset, and the most important one, was the random dataset.

I asked an AI agent to research different RAG questions people had online, such as “best rag eval benchmarks and why?” and “when does using titles/abstracts beat full text retrieval.”

Remember, the pipeline had only ingested around 150 scientific papers from September/October that mentioned RAG. So we don’t know if the ingested documents even had the answers.

To run the first evals, I used automated metrics such as faithfulness (does the answer stay grounded in the context) and answer relevancy (does it answer the query) from RAGAS.

I also added a few metrics from DeepEval to look at context relevance, structure, and hallucinations.

We ran both pipelines through the different datasets, and then through all of these metrics.

Then I added another head-to-head judge to A/B test the quality of each pipeline for each dataset. This judge did not see the context, only the question, the answer, and the automated metrics.

Why not include the context in the evaluation?

Because you can’t overload these judges with too many variables. This is also why evals can be difficult to do. You need to understand the one key metric you want to measure for each one.

I should note that this can be an unreliable way to test systems. If the hallucination score is mostly vibes, but we remove a data point because of it before sending it into the next judge that tests quality, we can end up with highly unreliable data once we start aggregating but alas.

For the final part, we looked at semantic similarity between the answers and examined the ones with the largest differences, along with cases where one pipeline clearly won over the other.

Let’s now turn to running the experiment.

Running the experiment

Since we have several different datasets, we need to go through the results of each. The first two datasets proved pretty lackluster, but they did show us something, so it’s worth covering them.

The random dataset showed the most interesting results so far. This will be the main focus, and I’ll dig into the results a bit to show where it failed and where it succeeded.

Remember, I’m trying to reflect reality here, which is often messier than people want it to be. You’ll see all the results in this sheet.

Clean questions from the corpus

The clean corpus dataset showed pretty identical results on all metrics. The judge seemed to prefer one over the other based on shallow preferences, but it showed us the issue with relying on synthetic datasets.

The first run was on the corpus dataset. Remember the clean questions that had been generated from the docs the pipeline had ingested.

The results of the automated metrics were eerily similar.

Even context relevance was pretty much the same, ~0.95–0.97 for both. I had to double check it several times to confirm. As I’ve had great success with using context expansion, the results made me a bit uneasy.

It’s quite obvious though in hindsight, the questions are already well formatted for retrieval, and one passage may answer the question.

I did have the thought on why context relevance didn’t decrease for the expanded pipeline if one passage was good enough. This was because the extra contexts come from the same section as the seed chunks, making them semantically related and not considered “irrelevant” by RAGAS.

The A/B test for quality we ran it through had similar results.

For the cases where naive won, the judge liked the answer’s conciseness, clarity, and focus. It penalized the complex pipeline for more peripheral details (edge cases, extra citations) that weren’t directly asked for.

When complex won, it liked the completeness/comprehensiveness of the answer over the naive one. This meant having specific numbers/metrics, step-by-step mechanisms, and “why” explanations, not just “what.”

Nevertheless, these results didn’t point to any failures. This was more a preference thing rather than about pure quality differences, both did exceptionally well.

So what did we learn from this? In a perfect world, you don’t need any fancy RAG add-ons, and using a test set from the corpus is highly unreliable.

Messy questions from the corpus

Next up we tested the second dataset, which showed similar results as the first one since it had been synthetically generated, but it started moving in another direction, which was interesting.

Remember that I introduced the messier questions generated from the corpus earlier. This dataset was generated the same way as the first one, but with messy phrasing (“can u explain like how plz…”).

The results from the automated metrics showed that the results were still very similar, though context relevance started to drop in the complex one while faithfulness started to rise slightly.

For the ones that failed the metrics, there were a few RAGAS false positives.

But there were also some failures for questions that had been formatted without specificity in the synthetic dataset, such as “how many posts tbh were used for dataset?” or “how many datasets did they test on?”

There were some questions that the query optimizer helped by removing noisy input. But I realized too late that the questions that had been generated were too directed at specific passages.

This meant that pushing them in as they were did well on the retrieval side. I.e., questions with specific names in them (like “how does CLAUSE compare…”) matched documents fine, and the query optimizer just made things worse.

There were times when the query optimization failed completely because of how the questions had been phrased.

Such as the question: “how does the btw pre-check phase in ac-rag work & why is it important?” where direct search found the AC-RAG paper directly, as the question had been generated from there.

Running it through the A/B judge, the results favored the advanced pipeline a lot more than they had for the first corpus.

The judge favored naive’s conciseness and brevity, while it favored the complex pipeline for completeness and comprehensiveness.

The reason we see the increase in wins for the complex pipeline is that the judge increasingly chose “complete but verbose” over “brief but potentially missing aspects” this time around.

This is when I had the thought how useless answer quality is as a metric. These LLM judges run on vibes sometimes.

In this run, I didn’t think the answers were different enough to warrant the difference in results. So remember, using a synthetic dataset like this can give you some intel, but it can be quite unreliable.

Random questions dataset

Lastly, we’ll go through the results from the random dataset, which showed a lot more interesting results. Metrics started to move with a higher margin here, which gave us something to dig into.

Up to this point I had nothing to show for this, but this last dataset finally gave me something interesting to dig into.

See the results from the random dataset below.

On random questions, we actually saw a drop in faithfulness and answer relevancy for the naive baseline.

Context relevance was still higher for naive, along with structure, but this we had already established for the complex pipeline in the previous article.

Noise will inevitably happen for the complex one, as we’re talking about 10x more chunks. Citation structure may be harder for the model when the context increases (or the judge has trouble judging the full context).

The A/B judge, though, gave it a very high score compared to the other datasets.

I ran it twice to check, and each time it favored the complex one over the naive one by a huge margin.

Why the change? This time there were a lot of questions that one passage couldn’t answer on its own.

Specifically, the complex pipeline did well on tradeoff and comparison questions. The judge reasoned “more complete/comprehensive” compared to the naive pipeline.

An example was the question “what are pros and cons of hybrid vs knowledge-graph RAG for vague queries?” Naive had many unsupported claims (missing GraphRAG, HybridRAG, EM/F1 metrics).

At this point, I needed to understand why it won and why naive lost. This would give me intel on where the fancy features were actually helping.

Looking into the results

Now, without digging into the results, you can’t fully know why something is winning. Since the random dataset showed the most interesting results, this is where I decided to put my focus.

First, the judge has real issues evaluating the fuller context. This is why I could never create a judge to evaluate each context against the other. It would prefer naive because it’s cognitively easier to judge. This is what made this so hard.

Nevertheless, we can pinpoint some of the real failures.

Even though the hallucination metric showed decent results, when digging into it, we could see that the naive pipeline fabricated information more often.

We could locate this by looking at the low faithfulness scores.

To give you an example, for the question “how do I test prompt injection risks if the bad text is inside retrieved PDFs?” the naive pipeline filled in gaps in the context to provide the answer.

Question: How do I test prompt injection risks if the bad text is inside retrieved PDFs?
Naive Response: Lists standard prompt-injection testing steps (PoisonedRAG, adaptive instructions, multihop poisoning) but synthesizes a generic evaluation recipe that is not fully supported by the specific retrieved sections and implicitly fills gaps with prior knowledge.
Complex Response: Derives testing steps directly from the retrieved experiment sections and threat models, including multihop-triggered attacks, single-text generation bias measurement, adaptive prompt attacks, and success-rate reporting, staying within what the cited papers actually describe.
Faithfulness: Naive: 0.0 | Complex: 0.83
What Happened: Unlike the naive answer, it is not inventing attacks, metrics, or techniques out of thin air. PoisonedRAG, trigger-based attacks, Hotflip-style perturbations, multihop attacks, ASR, DACC/FPR/FNR, PC1–PC3 all appear in the provided documents. However, the complex pipeline is subtly overstepping and has a case of scope inflation.

The expanded content added the missing evaluation metrics, which bumped up the faithfulness score by 87%.

Nevertheless, the complex pipeline was subtly overstepping and had a case of scope inflation. This could be an issue with the LLM generator, where we need to tune it to make sure that each claim is explicitly tied to a paper and to mark cross-paper synthesis as such.

For the question “how do I benchmark prompts that force the model to list contradictions explicitly?” naive again has very few metrics and thus invents metrics, reverses findings, and collapses task boundaries.

Question: How do I benchmark prompts that force the model to list contradictions explicitly?
Naive Response: Mentions MAGIC by name and vaguely gestures at “conflicts” and “benchmarking,” but lacks concrete mechanics. No clear description of conflict generation, no separation of detection vs localization, no actual evaluation protocol. It fills gaps by inventing generic-sounding steps that are not grounded in the provided contexts.
Complex Response: Explicitly aligns with the MAGIC paper’s methodology. Describes KG-based conflict generation, single-hop vs multi-hop and 1 vs N conflicts, subgraph-level few-shot prompting, stepwise prompting (detect then localize), and the actual ID/LOC metrics used across multiple runs. Also correctly incorporates PC1–PC3 as auxiliary prompt components and explains their role, consistent with the cited sections.
Faithfulness: Naive: 0.35 | Complex: 0.73
What Happened: The complex pipeline has far more surface area, but most of it is anchored to actual sections of the MAGIC paper and related prompt-component work. In short: the naive answer hallucinates by necessity due to missing context, while the complex answer is verbose but materially supported. It over-synthesizes and over-prescribes, but mostly stays within the factual envelope. The higher faithfulness score is doing its job, even if it offends human patience.

For complex, though, it over-synthesizes and over-prescribes, but stays within the factual information.

This pattern shows up in multiple examples. The naive pipeline lacks enough information for some of these questions, so it falls back to prior knowledge and pattern completion. Whereas the complex pipeline over-synthesizes under false coherence.

Essentially, naive fails by making things up, and complex fails by saying true things too broadly.

This test was more about figuring out if these fancy features help, but it did point to us needing to work on claim scoping: forcing the model to say “Paper A shows X; Paper B shows Y,” and so on.

You can dig into a few of these questions in the sheet here.

Before we move on to the cost/latency analysis, we can try to isolate the query optimizer as well.

How much did the query optimizer help?

Since I did not test each part of the pipeline for each run, we had to look at different things to estimate whether the query optimizer was helping or hurting.

First, we looked at the seed chunk overlap for the complex vs naive pipeline, which showed 8.3% semantic overlap in the random pipeline, versus more than 50% overlap for the corpus pipeline.

We already know that the full pipeline won on the random dataset, and now we could also see that it surfaced different documents because of the query optimizer.

Most documents were different, so I couldn’t isolate whether the quality degraded when there was little overlap.

We also asked a judge to estimate the quality of the optimized queries compared to the original ones, in terms of preserving intent and being diverse enough, and it won with an 8% margin.

A question that it excelled on was “why is everyone saying RAG doesn’t scale? how are people fixing that?”

Orginal: why is everyone saying RAG doesn’t scale? how are people fixing that?
Optimized (1): RAG scalability challenges (hybrid)
Optimized (2): Solutions for RAG scalability (hybrid)

Whereas a question that naive did well on its own was “what retrieval settings help reduce needle-in-a-haystack,” and other questions that were very well formatted from the start.

We could reasonably deduce, though, that multi-questions and messier questions did better with the optimizer, as long as they were not domain specific.

The optimizer was overkill for well formatted questions.

It also did badly when the question would already be understood by the underlying documents, in cases where someone asks something domain specific that the query optimizer wouldn’t understand.

You can look through a few examples in the sheet.

This teaches us how important it is to make sure that the optimizer is tuned well to the questions that users will ask. If your users keep asking with domain specific jargon that the optimizer is ignoring or filtering out, it won’t perform well.

We can see here that it is rescuing some questions and failing others at the same time, so it would need work for this use case.

Let’s discuss it

I’ve now overloaded you with a lot of data, so now it’s time to go through the cost/latency tradeoff, discuss what we can and cannot conclude, and the limitations of this experiment.

The cost/latency tradeoff

When looking at the cost and latency tradeoffs, the goal here is not to argue whether the results are “worth it”, but to put concrete numbers on what these features cost and their benefits.

I have collected the estimates here in the same sheet.

The cost of running this pipeline is very slim. We’re talking $0.00396 per run, and this does not include caching. Removing the query optimizer and neighbor expansion decreases costs by 41%.

It’s not more than that because token inputs, the thing that increases with added context, are quite cheap.

What actually costs money in this pipeline is the re-ranker from Cohere, which both the naive and the full pipeline use.

For the naive pipeline, the re-ranker accounts for 70% of the entire cost. So it’s worth looking at every part of the pipeline to figure out where you might implement smaller models to cut costs.

Nevertheless, at around 100k questions, you would be paying $400.00 for the full pipeline and $280.00 for the naive one.

There is also the case for latency.

We measured a +49% increase in latency with the complex pipeline, which amounts to about 6 seconds, mostly driven by the query optimizer using GPT-5-mini. It’s possible to use a faster and smaller model here.

For neighbor expansion, we measured the average increase to be 2–3 seconds longer. Do note that this doesn’t scale linearly.

4.4x more input tokens only added 24% more time.

Remember, you can see the entire breakdown in the sheet here.

What this shows is that the cost difference is real but not extreme, while the latency difference is much more noticeable. Most of the money is still spent on re-ranking, not on adding context.

What we can conclude

Let’s focus on what worked, what failed, and why. We see that neighbor expansion may pull it’s weight when questions are diffuse, but each pipeline has it’s own failure modes.

The clearest finding from this experiment is that neighbor expansion earns its keep when retrieval gets hard and one chunk can’t answer the question.

We did a test in the previous article that looked at how much of the answer was generated from the expanded chunks, and on clean corpus questions, only 22% of the answer content came from expanded neighbors.

We also saw that the A/B results here in this article showed a tie.

On messy questions, this rose to 30% with a 10-point margin for the A/B test. On random questions, it hit 41% (used from the context) with a 44-point margin for the A/B test.

What’s happening underneath is a difference in failure modes. When naive fails, it fails by omission. The LLM doesn’t have enough context, so it either gives an incomplete answer or fabricates information to fill the gaps.

We saw this clearly in the prompt injection example, where naive scored 0.0 on faithfulness because it invented an evaluation recipe that wasn’t in the documents.

When complex fails, it fails by inflation. It has so much context that the LLM over-synthesizes and makes claims broader than any single source supports. But at least those claims are grounded in something real.

The faithfulness scores reflect this asymmetry. Naive bottoms out at 0.0 or 0.35, while complex’s worst cases still land around 0.73.

The query optimizer is harder to call. It helped on 38% of questions, hurt on 27%, and made no difference on 35%. The wins were dramatic when they happened, rescuing questions like “why is everyone saying RAG doesn’t scale?” where direct search returned nothing.

But the losses were equally dramatic when the user’s phrasing already matched the corpus vocabulary and the optimizer introduced drift.

This probably suggests you’d want to tune the optimizer carefully to your users, or find a way to detect when reformulation is likely to help versus hurt.

On cost and latency, the numbers weren’t where I expected. Adding 10x more chunks only increased generation time by 24% because reading tokens is parallelized while writing them is sequential.

The real cost driver is the reranker, at 70% of the naive pipeline’s total.

The query optimizer contributes the most latency, at nearly 3 seconds per question. If you’re optimizing for speed, that’s where to look first, along with the reranker.

So more context doesn’t necessarily mean chaos, but it does mean you need to control the LLM to a larger degree. When the question doesn’t need the complexity, the naive pipeline will rule, but once questions become diffuse, the more complex pipeline starts to pull its weight.

Let’s talk limitations

I have to cover the main limitations of the experiment and what we should be careful about when interpreting the results.

The obvious one is that LLM judges run on vibes.

The metrics moved in the right direction across datasets, but I wouldn’t trust the absolute numbers enough to set production thresholds on them.

The messy corpus showed a 10-point margin for complex, but honestly the answers weren’t different enough to warrant that gap. It it could be noise.

I also didn’t isolate what happens when the docs genuinely can’t answer the question.

The random dataset included questions where we didn’t know if the papers had relevant content, but I treated all 66 the same.

I did though hunt through the examples, but it’s still possible some of the complex pipeline’s wins came from being better at admitting ignorance rather than better at finding information.

Finally, I tested two features together, query optimization and neighbor expansion, without fully isolating each one’s contribution. The seed overlap analysis gave us some signal on the optimizer, but a cleaner experiment would test them independently.

For now, we know the combination helps on hard questions and that the cost is 41% more per query. Whether that tradeoff makes sense depends entirely on what your users are actually asking.

Notes

I think we can conclude from this article that doing evals is hard, it’s even harder to put an experiment like this on paper.

I wish I could give you a clean answer, but it’s complicated.

I would personally say though that fabrication is worse than being overly verbose. But still if your corpus is incredibly clean and each answer usually points to a specific chunk, the neighbor expansion is overkill.

This then just tells you that these fancy features are insurance of a flawed system.

Remember you can see the full breakdown and numbers here.

❤

This is an article dated January 2026 but has been re-published here.

This is a follow-up to Building an Overengineered Retrieval System. That one was about building the entire system. This one is about doing the evals for it.

In the previous article, I went through different parts of a RAG pipeline: chunking the data properly, query optimization, retrieval (semantic, BM25, or hybrid search), re-ranking, expanding chunks to neighbors, building the context, and then generation with an LLM.

One of the questions I got was: does expanding chunks to neighbors actually improve answers, or does it just add noise and make it harder for the model to stay grounded?

So that’s what we’ll test here. We’ll run some basic evaluations and look at metrics like faithfulness, answer relevancy, context relevance, and hallucination rate, and compare results across different models and datasets.

Some of the LLM-as-a-judge metrics we’ll run here

I’ve collected most of the results here and here already, but we’ll go through them too.

I usually include an introductory section before getting into the results. If evals aren’t new to you, skip straight to the results.

Why we perform evals

Evals are about pressure-testing the system on a larger (and more targeted) corpus than your favorite ten questions, and making sure that changes you push don’t change the quality of the system.

Changes in data, prompts, or models can affect performance without you noticing it.

You may also need to show your team the general performance of the system you’ve built before being allowed to test it on real users.

But before you do this, you need to decide what to test. What does a successful system look like to you?

If you care about multi-hop, you need questions that actually require multi-hop. If you care about Q&A quality and proper citations, you need to test for that. Otherwise, you end up evaluating the wrong thing.

This is a bit like investigative work: you test something, try to understand the results, and then build better tests.

To do this well, you should ideally aim to build a golden set, often from user logs. This isn’t always possible, so in situations like this, we build synthetic datasets. This may not be the best approach, since it’s clearly biased but you may need to start somewhere.

For this article, I’ve created three different datasets so we can discuss it.

The first one is created from the ingested corpus, the second one uses messy user questions from the corpus, and third one has random questions on RAG that haven’t been generated from the corpus at all.

You’ll be able to see how these datasets give us different results on the metrics, but that they all mean different things.

What to think about

I’m not going to go through everything there is to think about here, because doing evals well is pretty difficult.

There are a few things you need to keep in mind.

LLM judges are biased, cherry-picking questions is a problem, gold answers are best if you have them, and structuring a larger dataset can help you understand where and how the system is failing.

If you’ve read the eval metrics article, you’ve already seen the idea of LLM-as-a-judge. It can be useful, but it’s not inherently reliable because it has baked-in preferences and blind spots.

There are things that will drive you a bit mad, like a judge punishing an answer that’s grounded in the corpus but not explicitly stated in the retrieved chunks, or judging the same answer differently depending on how the question is phrased.

You’ll realize this later, when you dig into failing questions to understand why.

Another thing to keep in mind is not to cherry-pick questions, even if the temptation is there. The goal is to get close to what users are actually asking, identify the issues, and continuously update the dataset based on where the system seems to fail.

It’s easy to get nice numbers if you mostly test “easy” questions, but then the eval becomes useless.

Ideally, you want not just real user questions but also gold answers.

So even if you can “bypass” references by using an LLM judge, having correct answers is still ideal. That’s when you can use the LLM to judge whether the output matches the gold answer, instead of asking it to judge the answer with vibes.

Sample size matters too.

Too small and the results aren’t reliable. Too big and it’s easy to miss smaller problems. If you have enough data, you can tag questions by topic, phrasing (pessimistic, typical), and type (short, long, messy) to see what breaks where.

I’ve seen recommendations to start with something like 200–1,000 real queries with gold answers if you want this to resemble a real evaluation setup.

Since this entire exercise is hypothetical, and the system has ingested documents to demo the idea of expanding to neighbors, some of the eval datasets here are synthetically generated. That makes them less reliable, but there are still things we can learn from them.

Deciding on metrics & datasets

This section is about two things: which metrics I’m using to evaluate the pipeline, and how I’m using them across datasets to see if neighbor expansion seems to help.

First, if you haven’t read about evals for LLM systems at all, go read this article. It gives you a taxonomy of the different metrics out there (RAG included).

Since I’m lazy for this, I needed reference-free metrics, but this will also limit us to what we can actually test. We can have the judge look at the context, the question, and the generated answer.

A few metrics that can help here are faithfulness (is the answer grounded in the provided context), answer relevancy (does it actually answer the question), context precision (how much of the context is just noise), and hallucination rate (how many claims are actually backed up by the provided context).

Since we want to figure out if seed expansion is useful, and without building two different pipelines, we can do one simple comparison: ask the judge to look at the seed chunks vs. the final expanded context and score how much of the answer comes from each for the faithfulness metric.

If grounding improves when the judge sees the expanded context, that’s at least evidence that the model is using the expanded chunks and it’s not just noise. We would need more testing, though, to say for sure which is the winner.

Finally, the datasets matter as much as the metrics.

If you’ve read the over-engineered RAG article, you know that all the docs that have been ingested are scientific articles that mention RAG. So all the questions that we create here need to be about RAG.

I have generated three different datasets with a different RAG flavor.

The first is based on the ingested corpus, going through each scientific article and writing two questions each that it can answer.

The second is doing the same but providing messy questions like: “how does k2 btw rag improve answer fetching compared to naive rag, like what’s the similarity scores in terms of q3?” This messy user questions dataset could tell us if stating things differently would skew the results.

The third dataset is based on 66 random RAG questions found online.

This means that these questions may not have answers in the corpus (the ingested RAG articles are just from September to October 2025, so we don’t know exactly what they contain).

To conclude, the first two will evaluate how well the pipeline behaves, whether it can answer questions on the documents it has, and the third one tells us what it is missing and how it behaves on questions that it might not be able to answer.

Though this is a bit simplified, as the first questions may be structured on sections and the random ones may be better answered by seed chunks.

Setting up to run the evals

To run the evals, you first need to run the pipeline on every question, for every model, and store the results.

If you don’t store everything you need, you can’t debug later. You want to be able to go from a low score back to the exact answer, the exact retrieved context, and the exact model settings.

I also wanted to compare models, because people assume “bigger model = better answers,” and that’s not always true, especially for easier tasks. So I’m running the same pipeline across GPT-5-mini, GPT-5.1, and GPT-5.2, for several datasets.

Once that’s done, I build the eval layer on top of those stored outputs. I used RAGAS for the standard metrics and DeepEval for the custom ones.

You can obviously build the tests manually, but it’s much easier this way.

I love how seamless DeepEval is, though it’s harder to debug if you find issues with the judge later.

A few specifics before I dig into the results: the pipeline runs with no context cap, the judge model is gpt-4o-mini, and we use n=3 for RAGAS and n=1 for the custom judges.

Since neighbor expansion is the whole point of this pipeline, we also run this check: for faithfulness, we score grounding against the seed chunks and against the full expanded context, to see if there’s a difference.

Running the evals

Let’s run the evals for the different datasets, metrics, and models to see how the pipeline is doing and how we can interpret the results.

You’ll find the full results here and here (especially if you dislike my childish sketches).

We can start with the results from the dataset generated by the corpus. You’ll see a fun sketch below with the numbers.

The table above shows the first RAGAS metrics. Faithfulness (does it stay grounded in the context provided) and answer relevancy (does it answer the question) are very high at around 0.95.

The models seem to perform pretty much the same.

This is to be expected, as we’re basically giving it questions that it should be able to answer with the documents. If these showed low numbers, there would be something severely off in the RAG pipeline.

It also gives us back seed faithfulness, where the judge is estimating how grounded the answer is to the seed chunks. This one is showing lower numbers than the full context faithfulness, 12–18 points across the different models.

This means that the chunk neighborhood expansion increases support coverage for an answer’s claims (higher judged faithfulness when the judge has full context).

In fewer words: we can say that the LLM is using the full context, not just the seed chunks, when generating its answer.

What we can’t judge though is if the seed-only answer would have been just as good. This will require us to run two pipelines and compare the same metrics and datasets for each.

Now let’s look at these next metrics (for the same dataset).

I would have estimated that context relevance would decrease here, as it’s looking at the full context that pulls in up to 10 different chunk neighbors for the section.

A reason for this may be that the questions generated are based on sections, which means that added context helps to answer them.

Structure citations (i.e. does it cite its claims correctly) looks alright, but hallucination is high, which is good (1 means no made-up claims in the answer).

Now you’ll see that the different models provide very little difference in terms of performance.

Yes, this is quite an easy Q&A task. But it does demonstrate that the additional size of the model may not be needed for everything, and the added context expansion may be able to act as a buffer for the smaller models.

Now let’s look at the results if we change the dataset to those messy user questions instead.

Here we compare messy with normal questions both using just GPT-5-mini

The table above shows us messy versus normal questions running GPT-5-mini.

We see a few drops in points, but they still stay high, though without isolating the outliers here we can’t say why.

But the median looks lower for faithfulness when only judging with the seed chunks for the messy user questions by an additional 7 points, which is interesting.

This could just be a result of using LLM judges that can fluctuate a few points up and down for each run (yes this happens).

Let’s now turn to the third dataset, which will be able to tell us a lot more. If we run the evals on it we get the results below.

We see all around worse numbers which is expected, the corpus that has been ingested probably can’t answer all of these questions so well. This will help us point to where we have missing information.

Faithfulness stays high though still for the full context runs. Here the difference from the seed-only runs are a lot higher, which means the added expansion is being used more in the answer.

Something that was strange here was how GPT-5.2 consistently did worse for answer relevance across two different runs. This can be a metric thing, or it can be a model thing where it answers more cautiously than before, thus getting a lower score.

This also tells you why it’s so important to test these new models on your own pipelines before adding them in.

Let’s check the results of the additional metrics for this dataset too.

Context relevance is very low, so the judge thinks there is a lot of noise in the context that does not directly answer the question for this dataset.

The context relevance metric rewards retrieving a high proportion of directly relevant chunks, but seed+expand intentionally adds neighbor chunks (definitions, adjacent sections, captions) that should improve grounding.

Unfortunately, there may be some noise that comes with that.

Both structure citations and hallucination score worse here, probably because it’s harder to stay grounded if you’re not provided any relevant information to use.

Now this was a first build for this pipeline. Based on these results, you can improve it until you drive those numbers up. Maybe change the metrics to be more custom to your pipeline.

Both datasets show different things: the corpus-based one shows the pipeline works, the random one shows there is missing information and that there is a lot more noise to navigate around.

What to examine

So what do you do now that you have all these scores? You can look at regressions, worst offenders, and do some random sampling.

Like I said before, this whole exercise is a bit like investigative work. But, there are three checks you can start with.

First is regressions: check the stuff that used to pass and now fails following changes.

Second is checking the worst offenders (bottom ~5–10 data points) to see what is happening. It could be the metric, the judge, it could be the retrieval, and it could be the model.

Below is one of the worst ones for full faithfulness and answer relevancy.

“question”: “how often does entity linking mess up acronyms and lookalikes, and how do i test that”,
“answer”: “The documents do not report a direct number for “how often” entity linking fails specifically on acronyms or look‑alike names.\n\nWhat we do see is that:\n- Even when structure is correct, LLMs often link to the wrong property or entity (semantic inaccuracies) in SPARQL generation over a scholarly KG [2]. This is the same failure mode you’d see when an acronym (e.g., NMF, ACE) is mapped to the wrong node.\n- In medical KGs, entity linking relies on exact keywords and then semantic similarity of embeddings when no exact match exists [3]. This last step explicitly allows linking to a merely “semantically related” concept rather than the exact one, which is where lookalikes and acronym collisions can occur [3].\n- In conversational entity‑centric systems, the largest single error category is coreference/entity confusion (27% of failures) [4], showing that confusing similar mentions is a major practical problem even outside strict EL.\n\nHow to test this in your setting, using patterns from the papers:\n1) Build a targeted test set\n   - Create a small benchmark that stresses acronyms and visually/lexically similar entities, analogous to how MATE is used to stress cross‑modal entity linking [1].\n   - Include positive pairs (correct acronym→entity) and hard negatives (different entities sharing acronym or near‑identical names).\n\n2) Run your full EL pipeline\n   - Use the same identifier / keyword / embedding‑based linking strategy described in MedKGent [3].\n\n3) Manually and/or LLM‑assisted evaluation\n   - Have humans or an LLM score whether each link is correct vs. incorrect, similar to the triple‑validity rubric used in MedKGent [3] and the SPARQL error audit in FIRESPARQL [2].\n\n4) Analyze error types\n   - Break down errors like the semantic vs. structural analysis in [2] and the error table in [4], but with categories “acronym collision,” “lookalike surface form,” etc.\n\nThis will give you an empirical error rate for your own stack, focused specifically on acronyms and lookalikes.”,
“full_faithfulness”: 0.24,
“seed_faithfulness”: 0.8260869565000001,
“answer_relevancy”: 0.0,
“context_relevance”: 0.208549739206933,
“context_relevance_reason”: “The context provided does not directly address the user’s question about how often entity linking messes up acronyms and lookalikes, nor does it offer methods for testing that. While it discusses entity linking and its evolution, it lacks specific information on the issues related to acronyms and lookalikes, which is the core of the user’s inquiry.”,
“hallucination_score”: 0.6572611409640697,
“hallucination_reason”: “The response accurately identifies that the documents do not provide a specific frequency for how often entity linking fails with acronyms or lookalikes, which aligns with the input query. It also discusses relevant issues such as semantic inaccuracies and coreference confusion, which are pertinent to the topic. However, it lacks direct references to specific claims made in the context, such as the limitations of traditional EL methods or the role of exact keywords in medical KGs, which could have strengthened the response further.”,
“full_contexts”: [”Entity Linking\n\nEntity Linking (EL) has evolved from text-only methods to Multimodal Entity Linking (MEL), and more recently to Cross-Modal Entity Linking (CMEL), which supports crossmodal reasoning. Traditional EL methods associate textual entities with their corresponding entries in a knowledge base, but overlook non-textual information (Shen, Wang, and Han 2015; Shen et al. 2023). MEL extends EL by incorporating visual [truncated]]

Here we can see the corpus didn’t actually contain direct evidence for what was asked but the seed chunks were more contained so faithfulness got a higher score when only providing the seed chunks.

Answer relevancy was punished unusually low for not being able to answer the direct question.

This one was an outlier though and in general full faithfulness is higher than seed faithfulness.

You can also check a good one to understand how it estimates the ones that passes.

“question”: “What retrieval settings help reduce needle-in-a-haystack issues for medical QA?”,
“answer”: “To reduce “needle‑in‑a‑haystack” problems in medical QA, the retrieval setup should focus on bringing back a small, very focused set of highly relevant items instead of large, noisy context.\n\n1. Control context size and granularity\n- Use abstracts or snippets rather than full articles; long, unfocused context diluted model performance in biomedical QA, so reranking down to a smaller, high‑quality subset was necessary for good answers [1].\n- Limit the number of documents passed to the model (e.g., rerank from thousands down to a few hundred or less) to avoid disorienting the model with too much text [1].\n\n2. Use multi‑stage retrieval\n- Start with high‑recall lexical retrieval (e.g., BM25 over PubMed) to get a broad set of candidates (up to thousands) [1].\n- Apply a strong semantic reranker to pick only the top slice for RAG (e.g., top 300) [1].\n- If evidence is missing, fall back to broader web‑or multi‑source search over trusted medical sites to fill gaps, rather than over‑stuffing the first pass with everything [1].\n\n3. Improve query quality\n- Use an LLM to rewrite or expand queries (synonyms, related terms, less rare jargon) when initial recall is poor, boosting the chance that the key paper is retrieved without over‑expanding every query by default [1].\n\n4. Use structured, corpus‑aware retrieval\n- Retrieve from structured medical knowledge sources such as knowledge graphs and prepend only the most relevant subgraph, which consistently improved accuracy across several medical QA benchmarks [3].\n- In multimodal/heterogeneous setups, use corpus‑specific retrieval for different knowledge sources so each corpus is searched with methods tuned to its content, instead of one generic retriever over everything [2].”,
“full_faithfulness”: 1.0,
“seed_faithfulness”: 0.8636363636000001,
“answer_relevancy”: 0.9135841092,
“context_relevance”: 0.8976322813389481,
“context_relevance_reason”: “The context passages provide comprehensive insights into retrieval settings that can mitigate needle-in-a-haystack issues in medical QA. Specifically, the discussion on the integration of LLMs for information retrieval, the use of semantic reranking, and the multi-stage retrieval approach directly addresses the user’s question. The emphasis on maintaining relevance while expanding query coverage and the mention of ensemble methods highlight effective strategies for improving retrieval accuracy in complex biomedical queries. However, while the information is highly relevant, a more explicit connection to specific ‘needle-in-a-haystack’ challenges could enhance clarity.”,
“hallucination_score”: 0.8893376167284271,
“full_contexts”: [”Abstract\n\nBiomedical question answering (QA) poses significant challenges due to the need for precise interpretation of specialized knowledge drawn from a vast, complex, and rapidly evolving corpus. In this work, we explore how large language models (LLMs) [truncated] ]

Here, the extra (full) context actually contains the missing evidence which helped and therefore got a higher score than the seeds.

Lastly, check a few random ones and see how it is evaluating these.

“question”: “how do i measure if i\u2019m getting diverse sources instead of 10 chunks from the same pdf”, 
“answer”: “You can treat \u201cdiverse sources\u201d as something you measure directly, not just eyeball.\n\n1. Simple source-diversity score (like in the paper)\nIn Document 1, they define a retrieval diversity metric as the fraction of unique review IDs among the top\u201110 chunks: Diversity = (number of distinct IDs) / 10 [1].\nYou can copy this idea for documents:\n- Give every PDF (or document) a unique ID.\n- When you retrieve K chunks (say K=10), look at which document each chunk came from.\n- Compute: Diversity = (number of distinct document IDs) / K.\n  \u2022 If all 10 chunks come from the same PDF, Diversity = 1 / 10 = 0.1 (very low).\n  \u2022 If all 10 chunks come from different PDFs, Diversity = 10 / 10 = 1.0 (perfect diversity).\n\nYou can log this number for each query to monitor whether you\u2019re repeatedly pulling from the same file or actually spreading across your corpus.\n\n2. Combine with relevance\nIn Document 1, they also track how close the chunks are to the query using an average cosine similarity between the query embedding and each chunk [1]. In practice you want both:\n- High similarity (on\u2011topic).\n- High diversity (many different source IDs).\n\n3. Optional: richer diversity later\nDocument 2 shows that diversity can also be measured at different levels (lexical, semantic, ingredient) when you have multiple outputs for one input [2]. By analogy, once you\u2019re happy with source diversity, you could later add:\n- Semantic diversity between chunks (are they saying different things?).\nBut the easiest starting point is the per-query \u2018distinct document IDs / K\u2019 metric.”,
“full_faithfulness”: 0.8823529412000001, 
“seed_faithfulness”: 0.5294117647000001,
“answer_relevancy”: 0.8588673985, 
“context_relevance”: 0.5303662744007874,
“context_relevance_reason”: “The context passages provide relevant information about measuring diversity in retrieval systems, particularly in the context of recipe adaptation and LLMs. However, while some passages discuss diversity metrics and retrieval methods, they do not directly address the user’s specific question about measuring diverse sources versus multiple chunks from the same PDF. The relevance of the context is somewhat indirect, leading to a moderate score.”,
“hallucination_score”: 0.7209711030557213,
“hallucination_reason”: “The response effectively outlines a method for measuring source diversity by introducing a simple source-diversity score and providing a clear formula. It aligns well with the context, which discusses retrieval diversity metrics. However, while it mentions combining relevance with diversity, it does not explicitly connect this to the context’s focus on average cosine similarity, which could enhance the completeness of the answer. Overall, the claims are mostly supported, with minor gaps in direct references to the context.”
“full_context”: [”D. Question and Answering (QA)\n\nFor retrieval of reviews, we sampled five Spotify-centric queries and retrieved the top K = 10 review chunks for each. We measured two unsupervised metrics... [truncated]

This above is interesting as you see that the evaluator is taking a reasonable generalization and treats it as “kinda supported” or “meh.”

Evaluating this item above with another LLM, it said that it thought the context relevance comment was a bit whiny.

But as you see, low scores don’t have to mean that the system is bad. You have to examine why they are low and also why they are high to understand how the judge works and why the pipeline is failing.

A good example is context precision for this.

Context precision is measuring how much of the retrieved context was useful. If you’re doing neighbor expansion, you will almost always pull in some irrelevant text, so context precision will look worse, especially if the corpus can’t answer the question in the first place.

The question is whether the extra context actually helps grounding (faithfulness / hallucination rate) enough to be worth the noise.

Some ending notes

Okay, some notes before I round this off.

Testing seeds here is clearly biased, and it doesn’t tell us whether they were actually useful on their own. We’d have to build two different pipelines and compare them side by side to say that properly.

I’ll do this in the future, and I’ll be honest about it.

I should note that the system has very few docs in the pipeline: only about 150 PDF files along with some Excel files, which is a few thousand pages.

But I have to demo this in public, and this was the only way.

Remember we used only metrics on the generation side here, looking at the context that was retrieved. If the context retrieved is lying or has conflicting information, these metrics may not show it, you have to measure that before.

Furthermore many teams also build their own custom metrics, that’s unique to their pipeline and to what they want to test, there are pitfalls to this, so I’m not sure I would recommend it, but you should always test what is useful to your application.

The last thing to note is LLM judge bias. I’m using OpenAI models both for the RAG pipeline and for the evaluator. You should be fine as long as you make sure it’s not the same model, but ideally it’s good to use different providers for the judges.

If you’ve gotten this far, I hope you liked it and that it gave you some intel into how to work with evals.

I hope you learned that running evals with LLMs as judges is risky and that smaller models do fine when you serve information up front.

As for this use case, we can say that the fancy add-ons doesn’t seem to cause that much issues, but without testing two different pipelines against each other it’s hard to understand which one wins. So stay tuned for the article that will test both against each other.

Remember you can look at the full results here and here.

❤

This is an article dated December 2025 but has been re-published here.

When you ask ChatGPT or Claude to “search the web,” it isn’t just answering from its training data. It’s calling a separate search system.

Most people know that part.

What’s less clear is how much traditional search engines matter and how much has been built on top of them.

All of it isn’t fully public, so I’m doing some mental deduction here. But we can use different hints from looking at larger systems to build a useful mental model.

We’ll go through query optimization, how search engines are used for discovery, chunking content, “on-the-fly” retrieval, and how you could potentially reverse-engineer a system like this to build a “GEO [Generative Engine Optimization] scoring system.”

If you’re familiar with RAG, some of this will be repetition, but it can still be useful to see how larger systems split the pipeline into a discovery phase and a retrieval phase.

If you’re short on time, you can read the TL;DR.

TL;DR

Web search in these AI chatbots is likely a two-part process. The first part leans on traditional search engines to find and rank candidate docs. In the second part, they fetch the content from those URLs and pull out the most relevant passages using passage-level retrieval.

The big change (from traditional SEO) is query rewriting and passage-level chunking, which let lower-ranked pages outrank higher ones if their specific paragraphs match the question better.

The technical process

The companies behind Claude and ChatGPT aren’t fully transparent about how their web search systems work within the UI chat, but we can infer a lot by piecing things together.

We know they lean on search engines to find candidates, at this scale, it would be absurd not to. We also know that what the LLM actually sees are pieces of text (chunks or passages) when grounding their answer.

This strongly hints at some kind of embedding-based retrieval over those chunks rather than over full pages.

This process has several parts, so we’ll go through it step by step.

Query re-writing & fan-out

First, we’ll look at how the system cleans up human queries and expands them. We’ll cover the rewrite step, the fan-out step, and why this matters for SEO.

I think this part might be the most transparent, and the one most people seem to agree on online.

The query optimization step is about taking a human query and turning it into something more precise. For example, “please search for those red shoes we talked about earlier” becomes “brown-red Nike sneakers.”

The fan-out step, on the other hand, is about generating additional rewrites. So if a user asks about hiking routes near me, the system might try things like “beginner hikes near Stockholm,” “day hikes near Stockholm public transport,” or “family-friendly trails near Stockholm.”

This is different from just using synonyms, which traditional search engines are already optimized for.

If this is the first time you’re hearing about it and you’re unconvinced, take a look at Google’s own docs on AI query fan-out or do a bit of digging around query rewriting.

To what extent this actually works, we can’t know. They may not fan it out that much and just work with a single query, then send additional ones down the pipeline if the results are lackluster.

What we can say is that it’s probably not a big model doing this part. If you look at the research, Ye et al. explicitly use an LLM to generate strong rewrites, then distill that into a smaller rewriter to avoid latency and cost overhead.

As for what this part of the pipeline means, for the business and SEO people out there, it means those human queries you’ve been optimizing for are getting transformed into more robotic, document-shaped ones.

SEO, as I understand it, used to care a lot about matching the exact long-tail phrase in titles and headings. If someone searched for “best running shoes for bad knees,” you’d stick to that exact string.

What you need to care about now is also entities, attributes, and relationships.

So, if a user asks for “something for dry skin,” the rewrites might include things like “moisturizer,” “occlusive,” “humectant,” “ceramides,” “fragrance-free,” “avoid alcohols” and not just “how would I find a good product for dry skin.”

But let’s be clear so there’s no confusion: we can’t see the internal rewrites themselves, so these are just examples.

If you’re interested in this part, you can dig deeper. I bet there are plenty of papers out there on how to do this well.

Let’s move on to what these optimized queries are actually used for.

Using search engines (for doc level discovery)

It’s pretty common knowledge by now that, to get up-to-date answers, most AI bots rely on traditional search engines. That’s not the whole story, but it does cut the web down to something smaller to work with.

I’m assuming the full web is too large, too noisy, and too fast-changing for an LLM pipeline to pull raw content directly. So by using already established search engines, you get a way to narrow the universe.

If you look at larger RAG pipelines that work with millions of documents, they do something similar. I.e. using a filter of some sort to decide which documents are important and worth further processing.

For this part, we do have proof.

Both OpenAI and Anthropic have said they use third-party search engines like Bing and Brave, alongside their own crawlers.

Perplexity may have built out this part on their own by now, but in the beginning, they would have done the same.

We also have to consider that traditional search engines like Google and Bing have already solved the hardest problems. They’re an established technology that handles things like language detection, authority scoring, domain trust, spam filtering, recency, and so on.

Throwing all of that away to embed the entire web yourself seems unlikely.

However, we don’t know how many results they actually fetch per query, whether it’s just the top 20 or 30. One unofficial article compared citations from ChatGPT and Bing, looked at the ranking order, and found that some came from as far down as 22nd place.

If true, this suggests you need to aim for top-20-ish visibility.

Furthermore, we also don’t know what other metrics they use to decide what surfaces from there. This article argues that AI engines heavily favor earned media rather than official sites or socials.

Still, the search engine’s job (whether it’s fully third-party or a mix) is discovery. It ranks the URL based on authority and keywords.

It might include a snippet of information, but that alone won’t be enough to answer the question. Unless of course it’s a very easy question like “who is the CEO of X?”

But for deeper questions, if the model relied only on the snippet, plus the title and URL, it would likely hallucinate the details. It’s not enough context.

So this pushes us toward a two-stage architecture, where a retrieval step is baked in (which we’ll get to soon).

What does this mean in terms of SEO?

It means you still need to rank high in traditional search engines to be included in that initial batch of documents that gets processed. So, yes, classic SEO still matters.

But it may also mean you need to think about potential new metrics they might be using to rank those results.

Nevertheless, this stage is all about narrowing the universe to a few pages worth digging into, using established search tech plus internal knobs. Everything else (the “it returns passages of information” part) comes after this step, using standard retrieval techniques.

Crawl, chunk & retrieve

Now let’s move on to what happens when the system has identified a handful of interesting URLs.

Once a small set of URLs passes the first filter, the pipeline is fairly straightforward: crawl the page, break it into pieces, embed those pieces, retrieve the ones that match the query, and then re-rank them. This is what’s called retrieval.

I call it on-the-fly here because the system only embeds chunks once a URL becomes a candidate, then it caches those embeddings for reuse. This part might be new if you’re already familiar with retrieval.

To crawl the page, it seem like they use their own crawlers. For OpenAI, this is OAI-SearchBot, which should then fect the raw HTML so it can be processed.

Crawlers don’t typically execute JavaScript. They most likely rely on server-rendered HTML, so the same SEO rules apply: content needs to be accessible.

Once the HTML is fetched, the content has to be turned into something searchable.

If you’re new to this, it might feel like the AI “scans the document,” but that’s not what happens. Scanning entire pages per query would be too slow and too expensive.

Instead, pages are split into passages, usually guided by HTML structure: headings, paragraphs, lists, section breaks, that kind of thing. These are called chunks in the context of retrieval.

Each chunk becomes a small, self-contained unit. Token-wise, you can see from Perplexity UI citations that chunks are on the order of tens of tokens, maybe around 150, not 1,000. That’s about 110–120 words.

After chunking, those units are embedded using both sparse and dense vectors. This enables the system to run hybrid search and match a query both semantically and by keyword.

Once a popular page has been chunked and embedded, those embeddings are probably cached. No one is re-embedding the same StackOverflow answer thousands of times a day.

This is obviously why the system feels so fast, probably the hot 95–98 % of the web that actually gets cited is already embedded, and cached aggressively.

We don’t know to what extent though and how much they pre-embed to make sure the system runs fast for popular queries.

Nevertheless, now the system needs to figure out which chunks matter. It uses the embeddings for each chunk of text to compute a score for both semantic and keyword matching.

If you’re new to semantic search, in short, it means the system searches for meaning instead of exact words. So a query like “symptoms of iron deficiency” and “signs your body is low on iron” would still land near each other in embedding space. You can read more on embeddings here if you’re keen to learn how it works.

After this it picks the chunks with the highest scores. This can be anything from 10 to 50 top-scoring chunks.

Most mature systems also add in a re-ranker (cross-encoder) to process those top chunks again, doing another round of ranking. This is the “fix the retrieval mess” stage, because unfortunately retrieval isn’t always completely reliable for a lot of reasons.

Although they say nothing about using a cross-encoder, Perplexity is one of the few that documents their retrieval process openly.

Their Search API says they “divide documents into fine-grained units” and score those units individually so they can return the “most relevant snippets already ranked.”

What does this all mean for SEO? If the system is doing retrieval like this, your page isn’t treated as one big blob.

It’s broken into pieces (often paragraph or heading level) and those pieces are what get scored. The full page matters during discovery, but once retrieval begins, it’s the chunks that matter.

That means each chunk needs to answer the user’s question.

It also means that if your important information isn’t contained inside a single chunk, the system can lose context. Retrieval isn’t magic. The model never sees your full page.

Note, this does depend on their chunking strategy, but I’m assuming they don’t do anything too fancy here.

So now we’ve covered the retrieval stage: where the system crawls pages, chops them into units, embeds those units, and then uses hybrid retrieval and re-ranking to pull out only the passages that can answer the user’s question.

Doing another pass & handing over chunks to LLM

Now let’s move on to what happens after the retrieval part, including the “continuing to search” feature and handing the chunks to the main LLM.

Once the system has identified a few high-ranking chunks, it has to decide whether they’re good enough or if it needs to keep searching. This decision is almost certainly made by a small controller model, not the main LLM.

I’m guessing here, but if the material looks thin or off-topic, it may run another round of retrieval. If it looks solid, it can hand those chunks over to the LLM.

At some point, that handoff happens. The selected passages, along with some metadata, are passed to the main LLM.

The model reads all the provided chunks and picks whichever one best supports the answer it wants to generate.

It does not mechanically follow the retriever’s order though. So there’s no guarantee the LLM will use the “top” chunk. It may prefer a lower-ranked passage simply because it’s clearer, more self-contained, or closer to the phrasing needed for the answer.

So just like us, it decides what to take in and what to ignore. And even if your chunk scores the highest, there’s no assurance it will be the first one mentioned.

What to think about

This system isn’t really a black box. It’s a system people have built to hand the LLMs the right information to answer a user’s question.

If what I’ve inferred here is true, then it finds candidates, splits documents into units, searches and ranks those units, and then hands them over to an LLM to summarize.

From this we can also figure out what we need to think about when creating content for it.

Traditional SEO still matters a lot, because this system leans on the old one. Things like having a proper sitemap, easily rendered content, proper headings, domain authority, and accurate last-modified tags are all important for your content to be sorted correctly.

As I pointed out, they may be mixing search engines with their own technology to decide which URLs get picked, which is worth keeping in mind.

But if they use retrieval on top of it, then paragraph level relevance is the new leverage point.

This means answer-in-one-chunk design might rule. (Just don’t do it in a way that feels weird, maybe a TL;DR.) And remember to use the right vocabulary: entities, attributes, relationships, like we talked about in the query optimization section.

How to build a “GEO Scoring System” (for fun)

To figure out how well your content will do, we’ll have to simulate the hostile environment your content will live in. So let’s try to reverse engineer this pipeline.

Note, this is non-trivial, as we don’t know the internal metrics they use nor is this system public, so think of this as a rough blueprint.

The idea is to create a pipeline that can do query rewrite, discovery, retrieval, re-ranking and an LLM judge, and then see where you end up compared to your competitors for different topics.

You begin with a few topics like “hybrid retrieval for enterprise RAG” or “LLM evaluation with LLM-as-judge,” and then build a system that generates natural queries around them.

Then you pass those queries through an LLM rewrite step, because these systems often reformulate the user query before retrieval. Those rewritten queries are what you actually push through the pipeline.

The first check is visibility. For each query, look at the top 20–30 results across Brave, Google and Bing. Note whether your page appears and where it sits relative to competitors.

At the same time, collect domain-level authority metrics (Moz DA, Ahrefs DR, etc.) so you can fold that in later, since these systems probably still lean heavily on those signals.

If your page appears in these first results, you move on to the retrieval part.

Fetch your page and the competing pages, clean the HTML, split them into chunks, embed those chunks, and build a small hybrid retrieval setup that combines semantic and keyword matching. Add a re-ranking step.

Somewhere here you also inject the authority signal, because higher-authority domains realistically get scored higher (even though we don’t know exactly how much).

Once you have the top chunks, you add the final layer: an LLM-as-a-judge. Being in the top five doesn’t guarantee citation, so you simulate the last step by handing the LLM a few of the top-scored chunks (with some metadata) and see which one it cites first.

When you run this for your pages and competitors, you see where you win or lose: the search layer, the retrieval layer or the LLM layer.

But remember this is still a rough sketch, but it gives you something to start with if you want to build a similar system.

This article focused on the mechanics rather than the strategy side of SEO/GEO, which I get won’t be for everyone.

The goal was to map the flow from a user query to the final answer and show that the AI search tool isn’t some opaque force.

Even if parts of the system aren’t public, we can still infer a reasonable sketch of what’s happening. What’s clear so far is that the AI web search doesn’t replace traditional search engines. It just layers retrieval on top of them.

There’s still more to figure out before saying what actually matters in practice. Here I’ve mostly gone through the technical pipeline but if this is new stuff I hoped it explain it well.

❤

This is an article dated August 2025 but has been re-published here.

Agentic Evals are mostly about testing your LLM application to make sure it performs consistently.

It’s not the most exciting topic, but more and more companies are paying attention. So it’s worth digging into which metrics to track to actually measure that performance.

It also helps to have proper evals in place anytime you push changes, to make sure things don’t go haywire.

So, for this article I’ve done some research on common metrics for multi-turn chatbots, RAG, and agentic applications.

I’ve also included a quick review of frameworks like DeepEval, RAGAS, and OpenAI’s Evals library, so you know when to pick what.

This article is split in two. If you’re new, Part 1 talks a bit about traditional metrics like BLEU and ROUGE, touches on LLM benchmarks, and introduces the idea of using an LLM as a judge in evals.

If this isn’t new to you, you can skip this. Part 2 digs into evaluations of different kinds of LLM applications.

What we did before

If you’re well versed in how we evaluate NLP tasks and how public benchmarks work, you can skip this first part.

If you’re not, it’s good to know what the earlier metrics like accuracy and BLEU were originally used for and how they work, along with understanding how we test for public benchmarks like MMLU.

Evaluating NLP tasks

When we evaluate traditional NLP tasks such as classification, translation, summarization, and so on, we turn to traditional metrics like accuracy, precision, F1, BLEU, and ROUGE.

These metrics are still used today, but mostly when the model produces a single, easily comparable “right” answer.

Take classification, for example, where the task is to assign each text a single label. To test this, we can use accuracy by comparing the label assigned by the model to the reference label in the eval dataset to see if it got it right.

It’s very clear-cut: if it assigns the wrong label, it gets a 0; if it assigns the correct label, it gets a 1.

This means if we build a classifier for a spam dataset with 1,000 emails, and the model labels 910 of them correctly, the accuracy would be 0.91.

For text classification, we often also use F1, precision, and recall.

When it comes to NLP tasks like summarization and machine translation, people often use ROUGE and BLEU to see how closely the model’s translation or summary lines up with a reference text.

Both scores count overlapping n-grams, and while the direction of the comparison is different, essentially it just means the more shared word chunks, the higher the score.

This is pretty simplistic, since if the outputs use different wording, it will score low.

All of these metrics work best when there’s a single right answer to a response and are often not the right choice for the LLM applications we build today.

LLM benchmarks

If you’ve watched the news, you’ve probably seen that every time a new version of a large language model gets released, it follows a few benchmarks: MMLU Pro, GPQA, or Big-Bench.

These are generic evals for which the proper term is more “benchmark.”

Although there’s a variety of other evaluations done for each model, including for toxicity, hallucination, and bias, the ones that get most of the attention are more like exams or leaderboards.

Datasets like MMLU are multiple-choice and have been around for quite some time. I’ve actually skimmed through it before and seen how messy it is.

Some questions and answers are quite ambiguous, which makes me think that LLM providers will try to train their models on these datasets just to make sure they get them right.

This creates some fear in the general public that most LLMs are just overfitting when they do well on these benchmarks and why there’s a need for newer datasets and independent evaluations.

LLM scorers

To run evaluations on these datasets, you can usually use accuracy and unit tests. However, what’s different now is the addition of something called LLM-as-a-judge.

To benchmark the models, teams will mostly use traditional methods.

So as long as it’s multiple choice or there’s just one right answer, there’s no need for anything else but to compare the answer to the reference for an exact match.

This is the case for datasets such as MMLU and GPQA, which have multiple choice answers.

For the coding tests (HumanEval, SWE-Bench), the grader can simply run the model’s patch or function. If every test passes, the problem counts as solved, and vice versa.

However, as you can imagine, if the questions are ambiguous or open-ended, the answers may fluctuate. This gap led to the rise of “LLM-as-a-judge,” where a large language model like GPT-4 scores the answers.

MT-Bench is one of the benchmarks that uses LLMs as scorers, as it feeds GPT-4 two competing multi-turn answers and asks which one is better.

Chatbot Arena, which use human raters, I think now scales up by also incorporating the use of an LLM-as-a-judge.

For transparency, you can also use semantic rulers such as BERTScore to compare for semantic similarity. I’m glossing over what’s out there to keep it condensed.

So, teams may still use overlap metrics like BLEU or ROUGE for quick sanity checks, or rely on exact-match parsing when possible, but what’s new is to have another large language model judge the output.

What we do with LLM apps

The primary thing that changes now is that we’re not just testing the LLM itself but the entire system.

When we can, we still use programmatic methods to evaluate, just like before.

For more nuanced outputs, we can start with something cheap and deterministic like BLEU or ROUGE to look at n-gram overlap, but most modern frameworks out there will now use LLM scorers to evaluate.

There are three areas worth talking about: how to evaluate multi-turn conversations, RAG, and agents, in terms of how it’s done and what kinds of metrics we can turn to.

You can see below the massive amount of metrics that have already been defined in these three areas.

We’ll talk about all of these briefly before moving on to the different frameworks that help us out.

Multi-turn conversations

The first part of this is about building evals for multi-turn conversations, the ones we see in chatbots.

When we interact with chatbots, we want the conversation to feel natural, professional, and for it to remember the right bits. We want it to stay on topic throughout the conversation and actually answer the thing we asked.

There are quite a few standard metrics people track here. The first we can talk about are Relevancy/Coherence and Completeness.

Relevancy is a metric that should track if the LLM appropriately addresses the user’s query and stays on topic, whereas Completeness is high if the final outcome actually addresses the user’s goal.

That is, if we can track satisfaction across the entire conversation, we can also track whether it really does “reduce support costs” and increase trust, along with providing high “self-service rates.”

The second part, is Knowledge Retention and Reliability.

That is: does it remember key details from the conversation, and can we trust it not to get “lost”? It’s not just enough that it remembers details. It also needs to be able to correct itself.

This is something we see in vibe coding tools. They forget the mistakes they’ve made and then keep making them. We should be tracking this as low Reliability or Stability.

The third part we can track is Role Adherence and Prompt Alignment. This tracks whether the LLM sticks to the role it’s been given and whether it follows the instructions in the system prompt.

Next are metrics around safety, such as Hallucination and Bias/Toxicity.

Hallucination is important to track but also quite difficult. People may try to set up web search to evaluate the output, or they split the output into different claims that are evaluated by a larger model (LLM-as-a-judge style).

There are also other methods, such as SelfCheckGPT, which checks the model’s consistency by calling it several times on the same prompt to see if it sticks to its original answer and how many times it diverges.

For Bias/Toxicity, you can use other NLP methods, such as a fine-tuned classifier.

Other metrics you may want to track could be custom to your application, for example, code correctness, security vulnerabilities, JSON correctness, and so on.

As for how to do the evaluations, you don’t always have to use an LLM, although in most of these cases the standard solutions do.

In cases where we can extract the correct answer, such as parsing JSON, we naturally don’t need to use an LLM. As I said earlier, many LLM providers also benchmark with unit tests for code-related metrics.

It goes without saying that LLMs used to judge aren’t always super reliable, just like the applications they measure, but I don’t have any numbers for you here, so you’ll have to hunt for that on your own.

Retrieval Augmented Generation (RAG)

To continue building on what we can track for multi-turn conversations, we can turn to what we need to measure when using RAG.

With RAG systems, we need to split the process into two: measuring retrieval and generation metrics separately.

The first part to measure is retrieval and whether the documents that are fetched are the correct ones for the query.

If we get low scores on the retrieval side, we can tune the system by setting up better chunking strategies, changing the embedding model, adding techniques such as hybrid search and re-ranking, filtering with metadata, and similar approaches.

To measure retrieval, we can use older metrics that rely on a curated dataset, or we can use reference-free methods that use an LLM as a judge.

I need to mention the classic IR metrics first because they were the first on the scene. For these, we need “gold” answers, where we set up a query and then rank each document for that particular query.

Although you can use an LLM to build these datasets, we don’t use an LLM to measure, since we already have scores in the dataset to compare against.

The most well-known IR metrics are Precision@k, Recall@k, and Hit@k.

These measure the amount of relevant documents fetched, how many relevant documents were retrieved based on the gold reference answers, and whether at least one relevant document made it into the results.

The newer frameworks such as RAGAS and DeepEval introduce reference-free, LLM-judge style metrics like Context Recall and Context Precision.

These count how many of the truly relevant chunks made it into the top K list based on the query, using an LLM to judge.

That is, based on the query, did the system actually return any relevant documents, or are there too many irrelevant ones to answer the question properly?

To build datasets for evaluating retrieval, you can mine questions from real logs and then use a human to curate them.

You can also use dataset generators with the help of an LLM, which exist in most frameworks or as standalone tools like YourBench.

If you were to set up your own dataset generator using an LLM, you could do something like below.

# Prompt to generate questions
qa_generate_prompt_tmpl = “”“\
Context information is below.

---------------------
{context_str}
---------------------

Given the context information and no prior knowledge
generate only {num} questions and {num} answers based on the above context.

...
“”“

If we turn to the generation part of the RAG system, we are now measuring how well it answers the question using the provided docs.

If this part isn’t performing well, we can adjust the prompt, tweak the model settings (temperature, etc.), replace the model entirely, or fine-tune it for domain expertise. We can also force it to “reason” using CoT-style loops, check for self-consistency, and so on.

For this part, RAGAS is useful with its metrics: Answer Relevancy, Faithfulness, and Noise Sensitivity.

These metrics ask whether the answer actually addresses the user’s question, whether every claim in the answer is supported by the retrieved docs, and whether a bit of irrelevant context throws the model off course.

If we look at RAGAS, what they likely do for the first metric is ask the LLM to “Rate from 0 to 1 how directly this answer addresses the question,” providing it with the question, answer, and retrieved context. This returns a raw 0–1 score that can be used to compute averages.

So, to conclude we split the system into two to evaluate, and although you can use methods that rely on the IR metrics you can also use reference free methods that rely on an LLM to score.

The last thing we need to cover is how agents are expanding the set of metrics we now need to track, beyond what we’ve already covered.

Agents

With agents, we’re not just looking at the output, the conversation, and the context.

Now we’re also evaluating how it “moves”: whether it can complete a task or workflow, how effectively it does so, and whether it calls the right tools at the right time.

Frameworks will call these metrics differently, but essentially the top two you want to track are Task Completion and Tool Correctness.

For tracking tool usage, we want to know if the correct tool was used for the user’s query.

We do need some kind of gold script with ground truth built in to test each run, but you can author that once and then use it each time you make changes.

For Task Completion, the evaluation is to read the entire trace and the goal, and return a number between 0 and 1 with a rationale. This should measure how effective the agent is at accomplishing the task.

For agents, you’ll still need to test other things we’ve already covered, depending on your application.

I just have to note: even if there are quite a few defined metrics available, your use case will differ, so it’s worth registering what the common ones are but don’t assume they are the best ones to track for your app.

Next, let’s turn to get an overview of the popular frameworks out there that can help you out.

Frameworks that help you out

There are quite a few frameworks that help you out with evals, but I want to talk about a few popular ones: RAGAS, DeepEval, OpenAI’s and MLFlow’s Evals, and break down what they’re good at and when to use what.

You can find the full list of different eval frameworks I’ve found in this repository.

You can also use quite a few framework-specific eval systems, such as LlamaIndex, especially for quick prototyping.

OpenAI and MLFlow’s Evals are add-ons rather than stand-alone frameworks, whereas RAGAS was primarily built as a metric library for evaluating RAG applications (although they offer other metrics as well).

DeepEval is possibly the most comprehensive evaluation library out of all of them.

However, it’s important to mention that they all offer the ability to run evals on your own dataset, work for multi-turn, RAG, and agents in some way or another, support LLM-as-a-judge, allow setting up custom metrics, and are CI-friendly.

They differ, as mentioned, in how comprehensive they are.

MLFlow was primarily built to evaluate traditional ML pipelines, so the number of metrics they offer is lower for LLM-based apps. OpenAI is a very lightweight solution that expects you to set up your own metrics, although they provide an example library to help you get started.

RAGAS provides quite a few metrics and integrates with LangChain so you can run them easily.

DeepEval offers a lot out of the box, including the RAGAS metrics.

Comparison between frameworks, see the Github repo here | Image by author

If we look at the metrics being offered, we can get a sense of how extensive these solutions are.

It’s worth noting that the ones offering metrics don’t always follow a standard in naming. They may mean the same thing but call it something different.

For example, faithfulness in one may mean the same as groundedness in another. Answer relevancy may be the same as response relevance, and so on.

This creates a lot of unnecessary confusion and complexity around evaluating systems in general.

Nevertheless, DeepEval stands out with over 40 metrics available and also offers a framework called G-Eval, which helps you set up custom metrics quickly making it the fastest way from idea to a runnable metric.

OpenAI’s Evals framework is better suited when you want bespoke logic, not when you just need a quick judge.

According to the DeepEval team, custom metrics are what developers set up the most, so don’t get stuck on who offers what metric. Your use case will be unique, and so will how you evaluate it.

So, which should you use for what situation?

Use RAGAS when you need specialized metrics for RAG pipelines with minimal setup. Pick DeepEval when you want a complete, out-of-the-box eval suite.

MLFlow is a good choice if you’re already invested in MLFlow or prefer built-in tracking and UI features. OpenAI’s Evals framework is the most barebones, so it’s best if you’re tied into OpenAI infrastructure and want flexibility.

Lastly, DeepEval also provides red teaming via their DeepTeam framework, which automates adversarial testing of LLM systems. There are other frameworks out there that do this too, although perhaps not as extensively.

I’ll have to do something on adversarial testing of LLM systems and prompt injections in the future. It’s an interesting topic.

The dataset business is lucrative business which is why it’s great that we’re now at this point where we can use other LLMs to annotate data, or score tests.

However, LLM judges aren’t magic and the evals you’ll set up you’ll probably find a bit flaky, just as with any other LLM application you build. According to the world wide web, most teams and companies sample-audit with humans every few weeks to stay real.

The metrics you set up for your app will likely be custom, so even though I’ve now put you through hearing about quite many you’ll probably build something on your own.

It’s good to know what the standard ones are though.

Hopefully it proved educational anyhow.

If you liked this one, be sure to read some of my other articles on agentic frameworks or building advanced RAG agents.

❤

This is an article dated February 2024 but has been re-published here.

Looking at the image at the top, you can at least figure out that I’ll be talking about using these smaller, fine-tuned NLP models in an application.

My last article was about fine-tuning smaller natural language models for specific use cases. Specifically, I talked about fine-tuning a BART model for keyword extraction. I also open-sourced it for anyone who wanted to use it.

For this piece, though, I will go into more application building with these smaller NLP models. The models I have used here are two models that were fine-tuned by me, and one was obtained from the Hugging Face hub.

When I show people the finished application, they find that it’s a bit of an odd concept like some kind of Google Trends for the tech space coupled with search.

The idea though is that it will crawl certain tech websites and then using NLPs it will breakdown the crawl-able content to analyze sentiment, extract keywords, organize content by category.

Example process for processing three titles using three different NLP models | Image by author

The end results will allow you to see what keywords are ‘trending’ and in what capacity for each category, or as a whole.

To demonstrate an example, check the graph below using the category Tools & Services.

Graphing keywords per category to demonstrate the application | Image by author

As you see from above we can deduce that Copilot is having a bad week based on the increase in mentions along with a majority of negative sentiment.

It will then allow you to get sources for the keywords you find interesting, so you can navigate to each source or analyze the content altogether. So in this case, it would be interesting to check out the urls where people are talking about Copilot and to read what they have to say there.

The application is primarily an API that I’m continuously building — so it is by no means done yet — but I have been playing with a frontend for it as well. I call it Safron.

The frontend gives you a fun way to click on a keyword within a category to see what has been said about it from the previous day.

You can go test it out; it works well as is for now and updates data daily.

Clicking on the keyword ‘FTX’ that was trending gives you a summary and sources | safron.io

If you decide to play around with the application, decide on your category and then click on the keyword to get a summary from AI about what is going on. You’ll be able to see the sources it is using as well. If it is in blue, the system says it is ‘trending.’

If you want to play around with the API instead — to build news bots — see this article. You can fetch trending and top keywords for up to one week back in time. It always updates in the mornings for the previous day.

It would be possible to have it process in real-time, but it would also be costly to keep the NLP models running 24/7. I pay very little to keep the system running as it is now.

How is it possible to build something like this, where these NLP models will do the brunt of the work for me? I’ll go through a brief introduction on the application itself, and then go into the technical bits.

It’s not groundbreaking stuff but interesting if you’re looking for inspiration and if you’re building something similar.

Introduction

Was it hard to build? Is it expensive to maintain? Why did I build it?

It wasn’t easy to build, mostly because there is so much data you need to clean and maintain. I do though believe that advancements in ML have made it possible to accomplish something like this faster than before.

It’s interesting to analyze public opinion and behavior, and since I work within tech, it made sense to build it.

When you build something you find interesting, you tend to improve at it, and then you think of new, innovative ways to continue working with it. So, in a way, sometimes you don’t need to have a clear goal because the process itself will produce something worthwhile.

Is it expensive to build and run?

The main reason for fine-tuning these smaller open-source models was to keep costs low. I went into some detail regarding costs using LLMs versus SLMs in my last piece. The total cost hasn’t gone over $80 so far, if I exclude the purchase of the domain name.

This price includes data extraction, database storage, hosting, fine-tuning the open-source NLP models, API, and cloud hosting. Now, if traffic increases, that’s a separate problem.

The biggest cost was using GPT-4 Turbo to help process the initial datasets for the smaller models. But there are open source options you can look into.

The question then becomes, how expensive would something like this have been if I had used a larger language model? Calling an API up to 15,000 times per day or hosting a very large model would have been very costly.

I did the maths on this in another article, where I looked at the cost of calling an API for GPT-4 and Claude as comparison using zero-shot with a hefty prompt template. You can read more on this here.

So, why did I build it?

You can achieve a lot with structured data like this.

Just attaching yourself to keywords and categories for product and community research is one approach, but also being able to use the API to feed into AI agents can keep you updated without having to sift through a lot of texts on a daily basis.

A clear application of this is to see which AI models are being discussed on a day to day basis, and to what extent. That is, how often do we hear that a new model is being introduced and it’s groundbreaking? By filtering the data with the category AI models, you can identify the top mentioned AI models during the week and which ones have an unusual count.

You can then tap into the sources on what people are saying, having an LLM summarizing for you.

You can see how this can be done below.

Sorting weekly data based on AI Models as Category (Note: this is an outdated UI)

You’ll see Mistral and Code Llama were mentioned significantly more than usual in the graph above — this was some week in January 2024. Mistral kept gaining ground, but that’s not always the case.

If mentions and engagement sharply increases from one period to another it will show up as trending by the API.

The incredible market share that ChatGPT holds, however, is amazing in comparison to these other models. It makes you think about how much ground it has already covered.

To check it out yourself, filter with the Category for AI models & assistants. You can filter by 11 categories such as companies, websites, tools, and platforms.

The UI has changed since this piece was written but you can still sort the keywords based on category and the click on the keyword to see the sources.

I’ll be honest, the model that does this categorization for me was pretty hard to create. I had to settle for a hybrid approach at first that I later changed to a model I built that understood context better. Today, it runs on its own but it took quite awhile to build the dataset for it.

Another useful application I discovered is using AI agents with the API to summarize the issues people are experiencing with certain keywords. This is great for generating regular reports but also for product research.

See an example using data for last week — in January 2024 — for the word “Copilot” with GPT-4.

Or a report for the word “Unity” that ChatGPT has summarized.

It would be straightforward enough to tap into these keywords by polling the API daily, let an LLM process it, and then have the reports sent to your email or your preferred medium.

This is what I mean by working with structured data. It can help the LLM provide you with accurate reports.

Going through this data, I’ve observed that there is a vast number of tools that you never hear about. If you follow the trends, you’ll notice that most people talk about the same tools, companies, and individuals on a daily basis.

It’s quite rare for something to immediately hit the mainstream track after being introduced.

The key will be, once the system has accumulated months of data, to begin to use this data to create better ML processes to understand how and when something is gaining traction.

Technical Process

I won’t delve into extensive detail about web crawling or the architecture behind building something like this, there should be a ton of content on this already.

Nevertheless, I store long-term data in BigQuery and host cached data in MongoDB. To construct the front-facing API, it made sense to create temporary collections in MongoDB using more complex SQL queries in BigQuery on a daily basis through automated scripts. This approach allows me to keep BigQuery query costs down to zero.

I do pay for a MongoDB dedicated cluster, which is a temporary solution as I aim to keep costs extremely low. This setup is feasible, but in this scenario, it would be beneficial to cache the table data on the frontend to reduce the number of requests. Something I decided to skip for now because I have a bit of spaghetti code going on.

The point I wanted to make with this application was to demonstrate it could be done at virtually no cost. I think I’ve managed to convey that message.

Let’s dig into the parts where I’ve used natural language processing though.

Natural Language Processing

For everyone new to natural language processing, transformer models were introduced in 2017. Transformer models understood the nuances of natural language much better than previous models.

With Google’s release of BERT in 2018, the practice of model fine-tuning became more common. BERT and other base models introduced the concept that you could take a pre-trained model and fine-tune it on a smaller dataset for specific tasks, achieving good results.

Platforms like HuggingFace have made it really easy to work with advanced NLP, allowing anyone to engage in the work I’m doing here and to apply NLP for various use cases.

One of the models I’m using someone else has built, the others I fine-tuned myself. If you’re keen to read about how to fine-tune these models, check out this article or this.

Sentiment, Keyword Extraction & Categorization

Under the hood, Safron is powered by three NLP models.

I wanted to build six models, but to maintain my sanity, I’ve stuck with using three in total. The first model I’m using is, unsurprisingly, the tech keyword extractor that I built some time ago.

I’ve detailed how I built this in a previous article. This model is important as it has the ability to identify important words automatically, which would have been very messy otherwise.

Keyword extraction is one of those things people think is very straightforward. Just think about how you would build a system to recognize what is a company, what is a person and what is a tool without the system understanding natural language.

After fine-tuning this model, I went back to fine-tune it again with subpar results. It makes me wonder if, in the future, there will be jobs only dedicated to sorting datasets for language models. It’s tough work and you do tend to give up pretty easily. GPT-4 can provide some help, but it’s not consistent enough without the use of human supervision.

The second model I used was the sentiment analyzer. I used someone else’s model here. This model can categorize a text as neutral, positive, or negative, assigning it a specific score.

Having used this for some time now, it’s rare to encounter many positive texts online; they are usually negative or neutral, especially within the tech sector. If I rebuild it, I will build it to understand context better for the exact keyword and perhaps with emotions rather than sentiment.

So, how do I use these models? The idea is to take the collected crawled data that had been previously processed and stored, and then applying these two models to the texts on a daily basis in batches.

Crawlable titles gets processed with NLPs to break it down into keywords

These keywords are stored and then linked back to their original content in another data table. I collect all the data, aggregate it, and send it to MongoDB, making it easily accessible from there.

It’s important to note that since the application applies sentiment analysis to texts as a whole, the analysis of the keywords is contextual. However, if a keyword consistently shows a high amount of negative sentiment, it usually indicates negative public opinion.

Tesla is one of those companies that have had a rough time lately.

ChatGPT summary of data for keyword ‘Tesla’ picked up by Safron— somewhere in January 2024 | Image by author

I would have attached an LLM to the application so instead of giving you the sources it gives you a nice summary.

This is something you can do now, via safron.io, as I utilize a very cheap LLM to do the last part of summarizing the texts per keyword.

Let’s get back on track though, I wasn’t quite done with extracting keywords and sentiment but I wanted to be able to categorize the words that were coming in. Having thousands of keywords in front of me on a day to day basis was too difficult to analyze by just looking at it, so I embarked on fine-tuning an encoder model for classification.

I opted for a larger RoBERTa model for this task because I needed it to be trained on as much data as possible. The goal was for it to identify each keyword and then assign a category from 12 different labels.

I used a hybrid approach that uses both a database and the NLP model to determine the category, though it sometimes makes mistakes. Later though, rebuilt the model to understand the keyword along with the text as context, this one does really well. I was able to achieve 99% accuraccy across 12 different labels last I gave it a go.

It does sometimes though make mistakes, but that can be rectified later by re-training the model with more quality data.

If you want to work with text classification, see this article.

Hosting Smaller NLP Models

You can boot up a smaller model using Colab, but using Hugging Face’s free pipeline to interact with the models would take about 2 hours to process up to 2,000 texts with a 400M parameter model.

I process up to 10,000 texts per day, which means it would need to run for about 8 hours daily. Hosting the models on a small GPU is incredibly efficient for these smaller models; by utilizing an Nvidia Tesla T4 GPU, I can process 10,000 texts in 30 minutes, batching 20–40 texts at a time.

Currently, I’ve only tried using Hugging Face’s inference endpoints and Replicate to host these models, mostly because it was straightforward enough to set up. However, it’s possible to just containerize them and expose them via an API endpoint on your own.

Last Notes

Any expert with experience in this field would probably take one look at this project and acknowledge that it’s not so simple. It’s not the NLPs that are difficult to create, it’s the entire data cleaning and data sorting that is the biggest hurdle. The creation of the algorithms and the API that calculates what is trending is another huge job.

The model for extracting tech terms, for example, will inevitably extract variations like “Google Gemini” or “Gemini Pro” or “Google’s Gemini.”

Cleaning this up is such work. My hat off to data engineers, honestly.

You can’t simply instruct these smaller models; they operate based on the data they’ve been trained on. You’ll need additional processing to be able to connect the words correctly, and even then some will fall off.

The categorization model that decides which label a keyword should be set as uses a pre-trained RoBERTA, which was last trained in 2019. It hasn’t seen many of these tools and companies that it is tasked to categorize. It will make assumptions for the correct category but you need a vast set of data to train it on for it to be able to perform.

Another prevalent issue was that these models can sometimes react quite strongly to data it hasn’t been trained on. For instance, my keyword model would crash the GPU whenever a Chinese character would slip in, simply because it hadn’t seen this kind of language before.

The keyword model also has issues in extracting words from comments, since it wasn’t trained on comments. This is something I will introduce in a later model, but if you’re new to building these models make sure you introduce the data you want processed early.

Building the model yourself means you understand the underlying data it was built on and can learn how to work with it effectively.

When you account for all this work, you also need to account for the work of creating the algorithms and the API that calculates what is trending.

Nevertheless, I hope this gave you some inspiration to build something of your own. I’m pretty keen to analyze the data that’s currently collected and try to find ways I can use it effectively.

I’ll keep building out the API which will hopefully be able to derive actual insights on the tech scene in time.

❤

This is an article dated December 2023 but has been re-published here.

This article will help you understand how to start working with smaller open source NLP models for specific use cases, why it can be more effective, as well as go through how to fine-tune a base model on your own.

I’ll get really nitty gritty and fine-tune a sequence to sequence model (BART) with a custom dataset for a specific use case — keyword extraction of texts for specific tech terms. The final model you’ll find here.

I’ll keep fine-tuning this model, but for now it works decently. If you’re wondering what good the model will do, I’ll get to that later.

I’ve also organized all material I’ve gathered in this Github repo if you’ll need it. This one has intel on the different base models, tasks and business cases as well as examples of fine-tuned models.

I’d also like to mention that what we’ll embark on here is completely free. Working with smaller open source models is a tad more complicated but a lot cheaper in the long run.

Do you need more convincing? Then go through the introduction section otherwise skip it and go to the training section directly.

Introduction

Large vs Small Models

It’s easy to assume that using the largest and most advanced models out there — such as GPT-4 or GPT-3 — is the most efficient way to work with NLP tasks. I think we can all agree that GPT-4 — and event GPT-3 — excels in various areas. I use it quite often and could very well use it for anything such as sentiment analysis, categorizing documents, and translations.

Although amazing, GPT-4 — and even GPT-3— are quite large models.

GPT, specifically, is a text generation model and is built primarily for content generation, which is amazing for creative writing or marketing content, developing chatbots and whatnot. So, although so advanced that it can do anything at this point, it was not technically made for all NLP tasks.

As a side note, most larger language models out there — open source as well — are text generation, decoder, models.

So what is the difference between a decoder and other models? See a table below for three examples of open source models, their model types and the specific tasks they were trained for.

Go here to check out all the base models for each model architecture.

GPT-2 is an open source model, but obviously the more advanced models by OpenAI are all closed source. BERT and BART is what is interesting to us here, and excel in other areas which we’ll talk about further.

If you’re wondering how these models tasks fits into a specific business case, go here to brainstorm. You’ll see examples of cases based on tasks for encoder, decoder and seq-to-seq models.

If you’re confused, skip it for now. Just know that all of these models, along with GPT, were originally built for different NLP tasks.

When choosing a model you should also consider more than simply performance but also computational resources and cost. When you get introduced to AI with GPT-3, it’s easy to disregard the other choices out there.

Sometimes, smaller, less complex models — fine-tuned for a specific use case — is more efficient in terms of resource usage and speed. It will also be significantly less expensive.

Let’s illustrate this.

Say we’re doing something banal, like analyzing sentiment or summarizing content. We’re scraping certain social media sites or maybe customer service conversations and then using AI to analyze its content, summarizing and giving us a report.

Let’s look at the costs of using a larger closed source model like GPT-4 Turbo, GPT-3.5 Turbo, or Claude 2 if we were to analyze up to 3,000 texts per day with 400 tokens input and 200 tokens out.

Costs to make up to 3,000 API calls per day for up to 31 days — keep in mind, that the larger your system template or prompt template is the more costly every API call will be.

We could obviously try to batch the texts in the same API calls but then our tokens would increase. The idea is though, that using these larger models for something simple is overkill.

I also need to stress, that when we build our own models we try to make them as small as possible because we also have to host them. Hosting can be very expensive.

Let’s look at a few smaller open source fine-tuned models in Hugging Face that does sentiment analysis here. These are fine-tuned models performing on a specific task.

Below is also another sentiment analysis model built on RoBERTa, an encoder model, that helps you classify the sentiment of texts as positive, negative or neutral.

This model is finiteautomata/bertweet-base-sentiment-analysis if you want to try it out on your own.

These are just examples. Go browse your own.

As a side note, when you test the models on the Hugging Face hub, you are using the Inference API which is for testing purposes only. If you want to go ahead connect to the models in Colab with Hugging Face’s pipeline use this script.

There are tons of more models available. Just to demonstrate, here is one to spot fake news, one to classify click bait articles and so on.

This is still very new, so models are a bit here and there and it’s hard to understand the performance of each model. You can look at likes and downloads. You should obviously test them too. But you’ll see more and more fine-tuned models being shared, maybe they’ll be a bit more organized in the future as well.

Nevertheless, the point is that you’ll find a several models that are vastly smaller than GPT-4 or Claude that might do well enough for your specific use case, especially when you account for cost.

As for how to use a Hugging Face model in your application, you can deploy your model using Hugging Face’s Inference Endpoints and let the application scale to 0 when not in use. For a model with 400 million parameters you would pay from $0.5 per hour (when in use) for a small GPU instance. You can also use the model directly in Google Colab or locally with Hugging Face’s pipeline for free.

There are also tools such as Replicate and Modal that offer cloud hosting with pricing per second of use. The other option is to containerize a model on your own and expose them via an API endpoint, using AWS for hosting.

So, Hugging Face is a great hub for finding and sharing open source models and datasets. As for hosting them, the future looks really bright. But the key here is how to actually train these smaller base models for a specific use case.

Hugging Face has simplified this for us too in various ways giving us an accessible training API, pre-built tokenizers and dynamic padding collators. What this means is that they will give us the correct tools to easily fine-tune these models without extensive machine learning knowledge.

Just as a side note about the transformers architecture, transformers have made it much easier for computers to understand and interact with human language. It has a unique structure that allows it to process words in relation to all other words in a sentence, rather than one at a time sequentially. This has made it possible to understand context better than previous models.

Transformers has therefore improved the output quality in general, which is something you’ve seen if you’ve tried ChatGPT, that offers fine-tuned versions of GPT-3 and GPT-4. I won’t go further than that as there is a lot of information out there you can scout. Check the Hugging Face learning hub, they have some great resources.

Instead, I’ll get right into working with a specific use case.

Use Case: Keyword Extraction

I came to Hugging Face because I was keen to build a fine-tuned model that would extract specific tech terms (i.e. keywords) from texts. I think we forget how well these NLP models can do at things we used libraries such as NLTK for before.

See an example of me trying to extract keywords from a text using NLTK, spaCy and KeyBERT below.

None of these libraries did what I need it to do. If I would aggregate thousands of texts the words that would be at the top would be ‘want’ ‘make’ and ‘said.’

You’ll see me testing KeyBERT above as well which is a library that is provided by Hugging Face and built on an enconder model (BERT). Unfortunately, it didn’t work well enough for my use case.

Here the keywords I would be looking for.

Bubble.io, Shopify, Analytics

To be able to do this, I would need something very specific most libraries wouldn’t be able to provide.

I did find a model that had already been fine-tuned in Hugging Face and that did a decent job. Check out the results from this model below.

The Inference API endpoint will be slow to load for the first API call — be patient

Better but not good enough.

If I test a few more texts with the fine-tuned BART model I found above I would get these results.

I needed something cleaner than this. The words returned weren’t always isolated, correct nor relevant.

So let’s check out the fine-tuned model I built below. I’ll demonstrate it with the same texts I used above.

See how it returns just a few relevant keywords making sure to separate names such as Docker? This is what I needed. By being able to extract the correct keywords, I can do analysis on thousands of texts.

The model I’ve built is not perfect as I’ve only fine-tuned it once. As for trying the model yourself, it is open-source so you may use it too.

To build this model I had 50,000 titles of different sizes I’ve accessed from various social media platforms using their public API endpoints. I was too lazy to process all of it as a first go but I did a trial of 3,000 texts along with 8,500 texts. The first trial run did alright with 3,000 data points but it wasn’t good enough so I went back to process 5,500 more rows.

I did use GPT-4 as help to transform the text to keywords. I dished out $30 for 10,000 texts, batched by 10 for every API call. Here is the repository for the script if you need to build your own dataset.

You need to make sure that the keywords are exactly the keywords you want generated so I had to manually go through 8,500 rows. As anyone will tell you, data is the biggest factor for fine-tuning. What you put in will come out.

It’s good to know, that quality trumps quantity always. You want quality data first, then if you can you add on to it.

I’ve shared my processed dataset of 8,500 rows here so check it out or use it yourself.

The dataset I had created to build the tech-keyword-extractor using the fields ‘text’ and ‘keywords’

This dataset has several fields but the ones I’ve added are keywords, topic and summary. Go nuts with that one if you want to build a different model, i.e. you can build a model that gives you the topic or a 3 word summary of the text rather than keywords.

To process my dataset from Hugging Face to Colab check out this script. We’ll go through processing this one later too though when fine-tuning the model.

Process

Let’s go through the process to to build this keyword model. If you have another use case in mind for a sequence to sequence model, this process should work the same.

The parts will be as follows.

• Explore models & tasks

• Set up your dataset

• Process the data

• Train the model & test it

• Push the model to Hugging Face

Some Notes

If you’re not comfortable with code what so ever, I would suggest giving Hugging Face’s AutoTrain a go. I think the first process is free but otherwise you’ll have do dish out some cash to continue to use this service.

Remember to make sure your dataset is on-point regardless.

If you want to do this manually with their trainer API manually, read on. This will be completely free.

Also use this Hugging Face GPT if you run into issues. Great for weird questions you don’t want anyone to see. I build one for anything I do nowadays.

Training

Models & Tasks

First, it’s good to just explain a bit about encoder vs decoder NLP models, especially when working with smaller models. There should be a ton of information you can scout on this too, so I’ll keep it concise.

The fundamental difference is that an encoder model will generally generate condensed outputs from larger inputs whereas decoder models will generally expand or generate data from a smaller input.

A very simplified visual of decoder vs encoder inputs and outputs

The technical explanation is that a decoder will look at the next sequence, i.e. what word comes next, while an encoder model will look at the text in its entirety to understand the meaning of the entire input.

A decoder model is more concerned with generating the next sentence while an encoder model focuses on analyzing and derive meaning from the entire text.

A decoder model — like GPT — thus excels in tasks like text generation, where you start with a small amount of information and need to create a more extensive coherent output. An encoder model, on the other hand, is trained to understand the context or the big picture, which is great for tasks where you want to classify or extract content.

Sequence-to-sequence models is a mix of encoder and decoder, where the encoder processes the input and the decoder generates the output. The first transformer model that was introduced in the “Attention is All You Need” paper was a sequence to sequence model.

I think of sequence to sequence (Seq2Seq) models as models that will be able to provide condensed outputs that are a bit more coherent compared to encoder models. I.e. it will excel in summarization which provides a more condensed version of a text but it is still able to maintain context.

A question you may have at this point, is why are the bigger models mostly decoder-only models? Could be because of computational efficiency. But I don’t have all the answers, so I would suggest you go and do some research on your own there.

It’s also worth noting, that the architecture was split up into decoder and encoder later and today’s larger LLMs can obviously handle many tasks — i.e. classification, summarization and so on — because of their massive scale. However, these larger LLMs might still not provide the specialized performance of all models in all scenarios.

To understand how you can find pre-trained models for each model type, look at the table below. This demonstrates a few different smaller pre-trained base models.

This list is not exhaustive of pretrained models you can use — go through this repo to see more.

The bigger models usually perform better for more complex tasks, so be a bit strategic on how you use them. With big, for the smaller models, I mean around 400–500M parameters.

If you are a bit lost, remember to look at the model types here. Also check different business cases for the different models here.

For my case, I will try BART as a base model and will use Summarization as the task for a model that will extract tech terms and names from texts.

Maybe this is an odd choice as an encoder model seem to be the better choice for keyword extraction. With a seq-to-seq model I may end up with summary keywords, rather than just the keywords. I.e. it may create its own. But I’m ok with this.

I will also use a fairly large pre-trained model of 400 million parameters. I would suggest you build one with a smaller model to see how it does, as the hosting costs increase with the size of the model.

Obtain the Dataset

After you’ve decided on your model and your task, next is to obtain a dataset.

The dataset will look different based on the model you will be training. For me, as I’m training a Seq2Seq model (BART-large) as mentioned above, I will have a text and a target — the keywords — in my dataset.

As for the size of it, maybe 3000 texts will do well enough, but ideally you’ll need at least 8,000 examples. Either you create your own dataset or you use an already created dataset that has been shared by someone else.

My dataset is 8,500 texts. You’ll find it here so you can use the same data. It has several rows so you can pick another field such as summary or topic as your target instead.

Another option is to browse the Hugging Face hub for a dataset you can use. Here you’ll have large datasets as options which is great if you’re just getting started.

If you’re building this dataset from scratch, see this script on generating your new dataset with GPT-4. This will be faster. But even with help, creating your own data is hard work. I manually checked my dataset and it took me two days to properly go through it.

Once you’ve decided on your dataset, you’ll have to make sure to split your data into a training, testing and validation set. The split usually is 80% of data for training, 10% validation and 10% testing depending on how small or large your dataset is.

To create a dataset dict with the appropriate datasets using either a custom CSV file or import a dataset from Hugging Face in Colab, see these cook books. Most datasets in Hugging Face are already processed with the appropriate sets, but if they are incomplete you can use these cook books as a guide.

My dataset already has the proper sets set up so no need to tweak it. You’ll see what this looks like when we import it into a Colab notebook.

Shop Around for a Base Model

I’ve seen many ‘shop’ around for a base model before they decide which one to take. You can fine-tune a base model directly or a model that has already been fine-tuned and fine-tune it further. It needs to be a transformer model though.

I have never liked the results from fine-tuning an already fine-tuned model but it is possible. This is when it has been fine-tuned for a specific task not pre-trained.

You can either test a bit in Hugging Face directly using the Inference API or open up a Colab notebook and use their pipeline.

Process the Dataset

Let’s start.

If I am working with custom data, I can get it from my Google Drive. If you’re doing the same you would start a bit differently. Here you would first import the file and then create a dataset dict with the training, validating and test sets. Here is the cook book for this.

What we’ll do here is just import my dataset from Hugging Face so we can use it.

Open up a new Colab notebook. The entire script for this process you’ll find here if you want it generated for you.

If you want to go through the process, we’ll start by importing all the dependencies we need.

!pip install -U datasets
!pip install -U accelerate
!pip install -U transformers
!pip install -U huggingface_hub

Then we’ll import the dataset we’ll be using. Here it’s your choice what dataset you’ll import.

from datasets import load_dataset

dataset = load_dataset(”ilsilfverskiold/tech-keywords-topics-summary”)
dataset

This one will already have a training, validation and test set setup for you. It should log this if you run it.

DatasetDict({
    train: Dataset({
        features: [’id’, ‘source’, ‘text’, ‘timestamp’, ‘reactions’, ‘engagement’, ‘url’, ‘text_length’, ‘keywords’, ‘topic’, ‘summary’, ‘__index_level_0__’],
        num_rows: 7196
    })
    validation: Dataset({
        features: [’id’, ‘source’, ‘text’, ‘timestamp’, ‘reactions’, ‘engagement’, ‘url’, ‘text_length’, ‘keywords’, ‘topic’, ‘summary’, ‘__index_level_0__’],
        num_rows: 635
    })
    test: Dataset({
        features: [’id’, ‘source’, ‘text’, ‘timestamp’, ‘reactions’, ‘engagement’, ‘url’, ‘text_length’, ‘keywords’, ‘topic’, ‘summary’, ‘__index_level_0__’],
        num_rows: 635
    })
})

The validation and testing sets are really low. I found it did alright but ideally you’d want all around more data.

If you’re importing a dataset without a specific set such as validation and testing, see this cook book.

We can map out some examples from the dataset to see what it looks like.

def show_samples(dataset, num_samples=3, seed=42):
    sample = dataset[”train”].shuffle(seed=seed).select(range(num_samples))
    for example in sample:
        print(f”\n’>> Text: {example[’text’]}’”)
        print(f”’>> Keywords: {example[’keywords’]}’”)


show_samples(dataset)

The result is this.

‘>> Text: Driverless car users will not be prosecuted for fatal crashes in UK’
‘>> Keywords: Driverless Cars, Legal Issues, UK’

‘>> Text: Google is embedding inaudible watermarks right into its AI generated music -’
‘>> Keywords: Google, AI Music, Watermarks, Audio Technology’

‘>> Text: What are your thoughts on Nextjs performance? Do you agree with this chart? - ( by 10up where Nextjs appears lower than WordPress on core vitals. Couldn’t post the image here due to community rules. But appreciate any other studies and thought you have on this matter.’
‘>> Keywords: Next.js, Performance, 10up, WordPress’

Here I’m using text and keywords as fields. You may decide to use other fields though.

Next we’ll get the tokenizer for the base-model we’re using so we can check if our texts are too long to be processed as a single input. A tokenizer converts text into tokens, allowing models to understand language. BART has a 1024 token limit for each input.

from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

model_name = ‘facebook/bart-large’ # go smaller if you can
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSeq2SeqLM.from_pretrained(model_name)

texts = dataset[’train’][’text’]

# Tokenize all texts and find the maximum length (max for BART is 1024 tokens)
max_token_length = max(len(tokenizer.encode(text, truncation=True)) for text in texts)
print(f”The longest text is {max_token_length} tokens long.”)

I’m way below this token limit though, but I suppose it’s good practice to check.

Now we’ll preprocess this data and convert both the input text and the target (i.e. the keywords) into a format suitable for training a sequence-to-sequence model.

def get_feature(batch):
  encodings = tokenizer(batch[’text’], text_target=batch[’keywords’],
                        max_length=1024, truncation=True)

  encodings = {’input_ids’: encodings[’input_ids’],
               ‘attention_mask’: encodings[’attention_mask’],
               ‘labels’: encodings[’labels’]}

  return encodings

dataset_pt = dataset.map(get_feature, batched=True)
dataset_pt

Remember data preprocessing functions would look different if you are using a model with a different architecture, such as an encoder-only or decoder-only model. This one is designed to handle both the input text and the target outputs (in this case, keywords).

This dataset_pt will now log a few new fields, input_ids, attention_mask and labels.

DatasetDict({
    train: Dataset({
        features: [’id’, ‘source’, ‘text’, ‘timestamp’, ‘reactions’, ‘engagement’, ‘url’, ‘text_length’, ‘keywords’, ‘topic’, ‘summary’, ‘__index_level_0__’, ‘input_ids’, ‘attention_mask’, ‘labels’],
        num_rows: 7196
    })
    validation: Dataset({
        features: [’id’, ‘source’, ‘text’, ‘timestamp’, ‘reactions’, ‘engagement’, ‘url’, ‘text_length’, ‘keywords’, ‘topic’, ‘summary’, ‘__index_level_0__’, ‘input_ids’, ‘attention_mask’, ‘labels’],
        num_rows: 635
    })
    test: Dataset({
        features: [’id’, ‘source’, ‘text’, ‘timestamp’, ‘reactions’, ‘engagement’, ‘url’, ‘text_length’, ‘keywords’, ‘topic’, ‘summary’, ‘__index_level_0__’, ‘input_ids’, ‘attention_mask’, ‘labels’],
        num_rows: 635
    })
})

We will specify that these new fields are the ones that should be returned.

columns = [’input_ids’, ‘labels’, ‘attention_mask’]
dataset_pt.set_format(type=’torch’, columns=columns)

The last thing we’ll do before we start to train the model is get the data collator for a sequence to sequence model.

The data collator is responsible for dynamically padding the batches to the maximum length in each batch which is crucial for efficient training of transformer models like BART or T5.

from transformers import DataCollatorForSeq2Seq

data_collator = DataCollatorForSeq2Seq(tokenizer, model=model)

Padding will look different depending on the type of model you use.

When looking at the preprocessing function and data collator, see it as us trying to translate human readable text into something a computer will understand.

Models are designed differently and that’s why it may look different if you were to use an encoder-only or decoder-only model.

Training

We should be ready to start training the model. We’re using the Trainer API which abstracts a lot of complexity here and makes it easy for us to fine-tune transformer models.

Make sure you switch your runtime on Colab to use at least T4. If you’re on a pro plan you can use the V100 but then also increase the batch size to at least 8.

from transformers import TrainingArguments, Trainer

training_args = TrainingArguments(
    output_dir = ‘bart_tech_keywords’, # rename to what you want it to be called
    num_train_epochs=3, # your choice
    warmup_steps = 500,
    per_device_train_batch_size=4, 
    per_device_eval_batch_size=4,
    weight_decay = 0.01,
    logging_steps = 10,
    evaluation_strategy = ‘steps’,
    eval_steps=50, 
    save_steps=1e6,
    gradient_accumulation_steps=16 
)

trainer = Trainer(model=model, args=training_args, tokenizer=tokenizer, data_collator=data_collator,
                  train_dataset = dataset_pt[’train’], eval_dataset = dataset_pt[’validation’])

trainer.train()

If you want to read more to understand the parameters go to the Trainer API docs. It depends a bit on what gear you are training this on.

If you’re using a large dataset here it may take up to two hours, but if you’re using this small dataset of 8,500 rows then this will be a 10 minute ordeal.

Once you start the process, what we’re looking for here is overfitting. The training loss should consistently decrease, whereas the validation loss may fluctuate but should decrease at the end.

Overfitting means that the model is fitting too well to our data which means it may not be able to generate unique answers. This will enable it to only generate output of data it has already been given which we do not want.

Ask the Hugging Face GPT about your parameters and results, it can help you interpret them.

Once it is done, save the model. Make sure you set the name you want.

trainer.save_model(’tech-keywords-extractor’)

Testing

I did not set up any evaluation metrics in this run, which is not recommended, see my other article for help in setting this up though and to understand what means what.

For this model, I’ll just test it manually to see how it does on our test set.

from transformers import pipeline

pipe = pipeline(’summarization’, model=’tech-keywords-extractor’)

test_text=dataset[’test’][0][’text’]
keywords = dataset[’test’][0][’keywords’]
print(”the text: “, text_test)
print(”generated keywords: “, pipe(test_text))
print(”orginal keywords : “,keywords)

You can also iterate over several examples so you can look them over one by one to see how they’re doing.

for i in range(0, 50):
    text_test = dataset[’test’][i][’text’]
    keywords = dataset[’test’][i][’keywords’]
    print(”text: “, text_test)
    print(”generated keywords: “, pipe(text_test)[0][’summary_text’])
    print(”original keywords: “, keywords)

Here it is good to do some more proper testing but I didn’t. I was decently satisfied.

Push to Hugging Face

So if you’re ready you can push it to the Hugging Face hub to save it for future use.

To log in you’ll need to navigate to Hugging Face and your account Settings to find Access Tokens.

Once you’ve found it, create a new write token and copy it. Use it when they ask you for your token.

!huggingface-cli login

After you’ve logged in you can simply set your username and the path you want your model to be pushed to. This will create a new model repository for you.

# you would replace your own name here
# you do not need to create a repository beforehand
trainer.push_to_hub(”your_hugging_face_username/tech-keywords-extractor”)

We’re done.

If you’ve got stuck along the way, here is the full Colab notebook for what we’ve done here. I didn’t run the whole thing because I already pushed my model once.

❤

This is an article dated September 2024 but has been re-published here.

Embeddings are a cornerstone of natural language processing. You can do quite a lot with embeddings, but one of the more popular uses is semantic search used in retrieval applications.

Although the entire tech community is abuzz with understanding how knowledge graph retrieval pipelines work, using standard vector retrieval isn’t out of style.

You’ll find multiple articles showing you how to filter out irrelevant results from semantic searches, something we’ll also be focusing on here using techniques such as clustering and re-ranking.

The main focus of this article, though, is to compare open source and closed source embedding models of various sizes.

We will compare up to 9 different embedding models that are high on the MTEB leaderboard. This will give you an idea of how a large versus a small model can perform and what the costs would be as you scale.

If you’ve ever used OpenAI’s models to generate embeddings, you’ve probably been curious to see if they are competitive enough.

Just as a quick recap on embeddings, if they’re new to you: when we create embeddings — an array of vectors — for each text, this translates into something a computer can understand.

Specifically for semantic search, we compare the embeddings for different texts to see how much semantic similarity there is between them. This allows us to use a kind of fuzzy search with a query — i.e., searching for relationships — rather than an exact keyword match.

I will go through embeddings and how they work in the introduction section, especially focusing on how we calculate semantic similarity.

I always use a custom case when I write, and this time is no different. I got this idea from a consultancy owner who was asking if he could create an application that would match job descriptions to LinkedIn profiles.

If we were doing this for real, we would be using millions of user profiles, but for this piece, I have created 6,900 synthetic LinkedIn profiles.

You can find the dataset here.

This is generally an easy use case as we’re not chunking up documents in a large file; the profiles will fit into one chunk each. The domain is not a difficult one, as it is easy for us to understand if a model is finding the right relationships.

But it will give you an idea of how to think about solving a similar problem.

If you want to go straight to experimenting with the different models with the LinkedIn dataset, you can scroll past the introduction.

Introduction

For this article, as we have so few profiles, we can use clustering as a kind of unsupervised classification method for the entire dataset.

See an illustration of what clustering looks like below.

Clustering will also let us understand how the different models perceives connected relationships.

Depending on the model, we can then isolate the correct group before performing semantic search within the cluster.

This should allow us to filter out irrelevant results, such as the model confusing product managers with product marketing managers.

You can add re-ranking with an LLM as a last step to make sure the top results wind up on top.

To make things very easy and less price-y, I have already added the embeddings for each model we’ll be evaluating in this dataset, I have also created embeddings for our queries, i.e., our anonymous job descriptions.

Remember if you want to go straight to experimenting you can scroll down to the the use case, although don’t skip the economics part.

Embeddings

I mentioned that embeddings are numerical representations of texts that capture their meaning, allowing computers to process and understand natural language.

With the more modern transformer models, these models can understand the entire context and thus understand several meanings of words and sentences — something that just wasn’t true a few years ago.

We can actually visualize embeddings on a graph by representing them as points in geometric space. Semantic relationships between embeddings thus translate into geometric closeness.

Different models are built for different tasks, but most of the larger ones are generalist enough to perform various tasks, such as retrieval, clustering, and classification.

Semantic search, used in retrieval, uses this closeness on the graph to figure out where a query would match with the other embeddings, i.e., it computes the distance of the embeddings on the graph.

To calculate this similarity between embeddings in semantic search, several methods are used, but cosine similarity is the most popular.

The model we use will directly affect the results you get from performing semantic search and this is a result of how it has been trained.

It matters what datasets, objectives, and architectures models are trained with, as it will influence how well it understand and link various texts.

Clustering, on the other hand, organizes data into groups (or clusters) where items are more similar to each other than to those in other groups. It is better at identifying and matching similarities between embeddings, allowing us to effectively isolate the group.

This process allows us to first filter out any irrelevant matches before performing semantic search and thus acts as a noise reduction tactic.

This is the idea, at least.

Not all models will be able to use clustering in the way we need them to; some will be better at it and some worse based on how they have been built.

Embedding Models

So, how do you know which model to pick? In comes the MTEB leaderboard that ranks embedding models based on their performance across various tasks.

I have picked out a few of these models that we’ll test for this, from the more popular models from OpenAI to compare with a fine-tuned Mistral-7B and smaller newer models such as Mxbai from Mixedbread AI.

They have all been released in the last two years, more or less.

If you’re new to open source models, you may be surprised to see that many open source models rank quite highly. If you’re not new to trying these models, it may still be interesting to see which one did best for this task.

Look at the table below to see the size, max tokens, and the ranking for retrieval and clustering for each model.

Many have used Ada-002 from OpenAI, and which is at the bottom of our list with respect to all the other models. OpenAI have though released text-embed-3 in both small and large sizes that perform better and is cheaper as well.

So, you may ask yourself, why would someone use a commercial model when they can just use an open-source model that’s high on the leaderboard?

Economics of Open Source Models

Using an open source model certainly sounds good and is the preferred privacy choice. Many are high on the leaderboard, but you do need to consider the economics of hosting a model versus using an API.

I looked at the cost of hosting both smaller (around 350M) and larger (7B) open source models on a GPU versus paying per token for a few popular commercial models.

The assumption here is that each text is 400 tokens, thus a 334M model will be able to process up to 75–90 texts per second on a single L4 GPU, and a 7B model will process around 30-40 texts per second with a single A100 but maybe more.

As you’ll observe, using a model like text-embed-3-large or ada-002 will really add up once you start to embed millions of texts. This is not including storage.

If you’re an enterprise client and you’re looking into Nvidia’s embedding models, such as nv-embed-v1, they offer quite a good API that you can tap into. I’ve used it to test a few of these models.

Using a small model though, less than 500 parameters, is certainly the most sound choice. If you can go with a smaller open source model, you should do so as you can slash your compute costs by up to 90%.

I also went ahead and calculated the processing times for smaller and larger models, if you were to host them on a single GPU.

Calling an API will also take time, and they have inference limits, so regardless of what choice you make, you’ll have to consider the amount of time it takes to fully embed an entire dataset.

For the open source models, you can always use more GPUs to process, but it gives you an idea of how using something smaller may be more energy-efficient.

It’s always good to test a few to see which model performs well for your task though as well, which is what we will do in a bit.

Quantization

As you saw above, using larger models, such as those around 7B, is still quite expensive and energy-intensive. We only calculated costs for 2.5 million embeddings, but once you start to scale further, it may be worthwhile to look into quantization.

Quantization compresses a model by using fewer bits to represent its data, which decreases the model’s size. The idea is that using quantization techniques like 4-bit and 8-bit quantization will help to run larger models on hardware that would normally not be able to handle such large models.

There have been a few people who have tried to measure the performance decrease of various metrics on quantized models; I think the last one I saw argued for a 12% overall drop in performance.

I will have to write a bit about this in the future, I love to look at the economics and performance cuts of these things.

The Use Case

I don’t know about you, but I like to test different models rather than just look at metrics. This gives me a sense of how smaller and larger models can interpret relationships between texts.

The dataset with the synthetic LinkedIn profiles you can find here, along with the dataset with our job descriptions that should be matched.

The Colab notebook we will work in you can find here.

Importing the Data

You need to open the notebook to follow along, but once you have done so, you should see that we’re importing two datasets from Hugging Face.

# Synthetic LinkedIn profiles with the embeddings
dataset = load_dataset(”ilsilfverskiold/linkedin_profiles_synthetic”)
profiles = dataset[’train’]

# Anonymous job descriptions with embeddings
dataset = load_dataset(”ilsilfverskiold/linkedin_recruitment_questions_embedded”)
applications = dataset[’train’]

These two datasets will allow us to compare the different embedding models for the 6,900 LinkedIn profiles that have been generated.

The synthetic data is, well, synthetic, so take it with a grain of salt. It was created with Llama 3.1 and it does suffer from some great alignment where it describes profiles using words such as ‘results-driven,’ ‘seasoned,’ and ‘dedicated.’

The embeddings have already been added, which you’ll see if you look into the ‘profiles.’

# profiles dataset
Dataset({
    features: [...,’embeddings_nv-embed-v1’, ‘embeddings_nv-embedqa-e5-v5’, ‘embeddings_bge-m3’, ‘embeddings_arctic-embed-l’, ‘embeddings_mistral-7b-v2’, ‘embeddings_gte-large-en-v1.5’, ‘embeddings_text-embedding-ada-002’, ‘embeddings_text-embedding-3-small’, ‘embeddings_voyage-3’, ‘embeddings_mxbai-embed-large-v1 ‘],
    num_rows: 6904
})

Ps. embeddings_gte-large-en-v1.5 does not work. I tried to host it but failed to set all the embeddings for it so do not use it.

From here, you need to decide on the job description you are interested in matching to the profiles.

Look at the code below; I have picked the second application, but you can set another number.

application = applications[1] # deciding on the second application - a product marketing manager position 
application_text = application[’natural_language’]
print(”application we’re looking for: “,application_text)

Check the dataset directly on Hugging Face if that is easier.

From here, you can decide which embedding model you’d like to work with. I have already tested most of them, so I’ll use embeddings_mxbai-embed-large-v1 for this run.

This is the 334M open-source model that ranked quite high on the leaderboard for both retrieval and clustering if you scroll up to the table I used earlier.

If you want to try a different model, you simply set another one. Look into the dataset mentioned above to see which ones you have access to.

# Get the query embeddings for an embedding model - in here we’re picking mxbai-embed-large-v1
query_embedding_vector = np.array(application[’embeddings_mxbai-embed-large-v1’])

embeddings_list = [np.array(emb) for emb in profiles[’embeddings_mxbai-embed-large-v1 ‘]] # note the extra space
texts = profiles[’text’]

Semantic Search

We can try to perform semantic search before we try to cluster; this allows us to see how it can do before adding in anything else.

To calculate the semantic similarity between the profiles and our query — the job application — we run the code below.

# Let’s first try to calculate the cosine similarity (without clustering)
def cosine_similarity(a, b):
    a = np.array(a)
    b = np.array(b)
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

similarities = []
for idx, emb in enumerate(embeddings_list):
    sim = cosine_similarity(query_embedding_vector, emb)
    similarities.append(sim)

Then, we can display the similarity score by sorting the highest on top, limiting the display to the first 30 results.

results = list(zip(range(1, len(texts) + 1), similarities, texts))
sorted_results = sorted(results, key=lambda x: x[1], reverse=True)

# Let’s display the results as well
print(”\nSimilarity Results (sorted from highest to lowest):”)
for idx, sim, text in sorted_results[:30]:  # adjust if you want to show more
    percentage = (sim + 1) / 2 * 100
    text_preview = ‘ ‘.join(text.split()[:10])
    print(f”Text {idx} similarity: {percentage:.2f}% - Preview: {text_preview}...”)

The results will look like something below, but it depends on the application you chose.

Similarity Results (sorted from highest to lowest):
Text 3615 similarity: 89.59% - Preview: Product Marketing Manager | Building Go-to-Market Strategies for Growth Results-driven...
Text 6299 similarity: 89.56% - Preview: Product Marketing Manager | Driving Growth & Customer Engagement Results-driven...
Text 3232 similarity: 89.09% - Preview: Product Marketing Manager | Driving Product Growth through Data-Driven Strategies...
Text 5959 similarity: 88.90% - Preview: Product Marketing Manager | Data-Driven Growth Expert Results-driven Product Marketing...
Text 5635 similarity: 88.84% - Preview: Product Marketing Manager | Driving Growth through Data-Driven Marketing Strategies...
Text 5835 similarity: 88.74% - Preview: Product Marketing Manager | Cloud-Based SaaS Results-driven Product Marketing Manager...
Text 139 similarity: 88.66% - Preview: Product Marketing Manager | Scaling Growth through Data-Driven Strategies Experienced...
Text 6688 similarity: 88.48% - Preview: Product Marketing Manager | Driving Business Growth through Data-Driven Insights...
Text 6405 similarity: 88.27% - Preview: Product Marketing Manager | Scaling SaaS Products for Global Markets...
Text 3439 similarity: 88.11% - Preview: Product Manager | Focused on delivering innovative products that drive...
Text 5958 similarity: 88.00% - Preview: Product Manager Office | Growth Driven by Customer Centricity Highly...
Text 5183 similarity: 87.86% - Preview: Product Marketing Manager | B2B SaaS Experienced Product Marketing Manager...
Text 1329 similarity: 87.81% - Preview: Product Marketing Manager | Scaling Growth for Emerging Tech Startups...
Text 130 similarity: 87.81% - Preview: Product Marketing Manager | Growth Strategies & Launches Results-driven Product...
Text 3423 similarity: 87.78% - Preview: Product Marketing Manager | Scaling B2B SaaS Solutions Experienced Product...
Text 4234 similarity: 87.72% - Preview: Product Manager | Leading Cross-Functional Teams to Drive Business Growth...

Using the mxbai embedding model, along with many others, we can clearly see that the results will return Product Marketing Managers with Product Managers, which is something that we do not want.

Look at the 88.11% — Preview: Product Manager and 88.00% — Preview: Product Manager Office above.

Let’s introduce clustering to see if it can help.

Clustering

First, we set up the clusters from the profile embeddings; here, we need to decide the amount of clusters.

I picked 10.

embeddings_array = np.array(embeddings_list)

num_clusters = 10 # you can pick another number here

kmeans = KMeans(n_clusters=num_clusters, random_state=42)
kmeans.fit(embeddings_array)
cluster_labels = kmeans.labels_

pca = PCA(n_components=2)
reduced_embeddings = pca.fit_transform(embeddings_array)

Then, we need to understand what cluster the query — or job application — will fit into.

# Let’s now see how query fits into the clustering
query_embedding_array = np.array(query_embedding_vector).reshape(1, -1)
reduced_query_embedding = pca.transform(query_embedding_array)

# Let’s also predict which cluster the query would belong to
query_cluster_label = kmeans.predict(query_embedding_array)[0]
print(f”The query belongs to cluster {query_cluster_label}”)

After this, we can visualize the clusters on a 2-dimensional graph — remember that the clusters have been flattened, so they may sit on top of each other.

You can hover over the different embeddings to see the profiles.

We can also isolate the query, our X, on the graph to see the cluster the model thinks it belongs to.

We can clearly see that the model is correctly interpreting the marketing people into one cluster, including SEO specialists and growth hackers in the same cluster, while not including Office Product Manager nor Product Managers.

This is great.

From here, we can now combine both clustering and semantic search to get better results.

Remember to check out the different models and try to see which does better; you’ll see that the bigger models are naturally better at being able to group similar profiles but some smaller models do quite well.

Clustering & Semantic Search

Now that we see that it is able to group the query into the right cluster, we can combine our approach.

# Let’s now do semantic search but only in the correct cluster
cluster_indices = np.where(cluster_labels == query_cluster_label)[0]

cluster_embeddings = embeddings_array[cluster_indices]
cluster_texts = [texts[i] for i in cluster_indices]

similarities_in_cluster = []
for idx, emb in zip(cluster_indices, cluster_embeddings):
    sim = cosine_similarity(query_embedding_vector, emb)
    similarities_in_cluster.append((idx, sim))

similarities_in_cluster.sort(key=lambda x: x[1], reverse=True)

top_n = 40  # adjust this number if you want to display more matches
top_matches = similarities_in_cluster[:top_n]

print(f”\nTop {top_n} similar texts in the same cluster as the query:”)
for idx, sim in top_matches:
    percentage = (sim + 1) / 2 * 100
    text_preview = ‘ ‘.join(texts[idx].split()[:10])
    print(f”Text {idx+1} similarity: {percentage:.2f}% - Preview: {text_preview}...”)

As we can see if we run the code above, the results are now giving back results without Product Manager in them, instead it gives us back Marketing Managers which is a better fit in general.

Top 40 similar texts in the same cluster as the query:
Text 3615 similarity: 89.59% - Preview: Product Marketing Manager | Building Go-to-Market Strategies for Growth Results-driven...
Text 3232 similarity: 89.09% - Preview: Product Marketing Manager | Driving Product Growth through Data-Driven Strategies...
Text 5959 similarity: 88.90% - Preview: Product Marketing Manager | Data-Driven Growth Expert Results-driven Product Marketing...
Text 5635 similarity: 88.84% - Preview: Product Marketing Manager | Driving Growth through Data-Driven Marketing Strategies...
Text 5835 similarity: 88.74% - Preview: Product Marketing Manager | Cloud-Based SaaS Results-driven Product Marketing Manager...
Text 139 similarity: 88.66% - Preview: Product Marketing Manager | Scaling Growth through Data-Driven Strategies Experienced...
Text 6688 similarity: 88.48% - Preview: Product Marketing Manager | Driving Business Growth through Data-Driven Insights...
Text 6405 similarity: 88.27% - Preview: Product Marketing Manager | Scaling SaaS Products for Global Markets...
Text 5183 similarity: 87.86% - Preview: Product Marketing Manager | B2B SaaS Experienced Product Marketing Manager...
Text 1329 similarity: 87.81% - Preview: Product Marketing Manager | Scaling Growth for Emerging Tech Startups...
Text 130 similarity: 87.81% - Preview: Product Marketing Manager | Growth Strategies & Launches Results-driven Product...
Text 3423 similarity: 87.78% - Preview: Product Marketing Manager | Scaling B2B SaaS Solutions Experienced Product...
Text 5945 similarity: 87.63% - Preview: Marketing Manager | Driving Growth through Data-Driven Strategies Results-driven marketing...
Text 2664 similarity: 87.59% - Preview: Product Marketing Manager | Driving Growth & Innovation Results-driven Product...
Text 3368 similarity: 87.54% - Preview: Product Marketing Manager | Scaling Growth through Data-Driven Strategies Highly...
Text 5794 similarity: 87.48% - Preview: Product Marketing Manager | Driving Growth through Data-Driven Insights Results-driven...
Text 5685 similarity: 86.71% - Preview: Performance Marketing Manager | Driving Business Growth through Data-Driven Strategies...
Text 5818 similarity: 86.37% - Preview: Digital Marketing Manager | Driving Business Growth through Data-Driven Strategies...

For a real case, you’d ideally want to filter and do classification on this dataset before performing semantic search.

The idea here is for you to compare the different models, especially smaller ones to the bigger ones, to see how much quality you are willing to sacrifice for faster and cheaper inference.

Don’t go for a bigger model just because, unless you really need it.

If you want to continue to evaluate the models, you can use RAGAs to evaluate how the retrieval application would do based on the different models.

Notes on Model Performance

I needed to pick something here to evaluate performance, so I chose to look at how good the models did at being able to separate product managers and product marketing managers.

The bigger models have more of an ability to get you the correct results before clustering, but all of them had issues at first to separate the two.

Ada-002, possibly being a lot bigger, did well at performing before clustering, whereas OpenAI’s smaller and newer model, text-embed-3-small, did worse.

However, some of the models were also struggling to cluster the profiles correctly as well. Specifically, the fine-tuned 7B Mistral model and E5 did not do well here. This could be a natural consequence of how they were built.

The rest did about the same, for this specific job profile.

I was surprised at how well mxbai performed, being only 335M in size; this goes to show that the bigger models may be overkill for simpler tasks.

This is only an evaluation for this small thing; I suggest you look at other things to evaluate performance for your task.

Nevertheless, we can continue from here and also add on strategies such as re-ranking to give the best results to an LLM to evaluate.

Re-Ranking

There are many strategies to correct for irrelevant results with RAG pipelines; re-ranking is one.

Re-ranking basically means to re-rank the results so the more relevant ones will be on top. Strategies to achieve this can be to use Pairwise Ranking.

To do this, you give a pair to a model, could be an LLM, and ask it to rank the usefulness of two profiles based on the job description.

You’ll have to combine methods for your use case to enable it to perform well.

If you are new to embeddings, I hope you learned something, and if it’s not new, then I hope you got a bit of intel about the economics of using smaller versus larger embedding models, be they open source or commercial.

For the larger LLMs, many closed-source models are taking the lead, this is not true when it comes to embedding models.

Something to take with you is to give a smaller, more computationally efficient model a chance.

❤

This is an article dated October 2025 but has been re-published here.

If you’re just starting to build different agentic systems, one of the interesting areas is the difference between building a single versus multi-agent workflow, or perhaps the difference between working with more flexible vs controlled systems.

This article will help you understand what agentic AI is and how to build agentic systems with LangGraph & LangSmith Studio.

We’ll build a researcher with two different architectures to be able to compare the results to understand which one can do better.

You’ll find the resources we’ll be working with here. Running the systems will mostly be free except for some OpenAI tokens.

As a note, if you want to get an overview of the different open source frameworks out there, check out this article.

The use case

To build something concrete, we’ll be building a research agent for tech that can find what is trending yesterday, or the last week, and then figure out what is news worthy.

Working with summarizing and gathering research is one of those areas that agentic AI can really shine.

This article will be using an API that gathers what people are speaking and sharing in tech, and the agentic system will be tasked with deciding what is important based on our user persona, and then summarizing for us.

The agent won’t be adding any citations in the text, we are looking at how much it covers when we only set up a single agent versus what it covers when we are setting up multiple agents working in sync.

The focus thus is less on the data source and more on the agentic part.

We will build this system first with a simple agent that has access to the different API endpoints, and then we will build out the system to use several teams and more comprehensive tools to see the difference in quality.

Before we start, I always do a review for beginners. If you’re well versed in agentic systems, you can scroll past some of the first sections.

Agentic AI & LLMs

Agentic AI is about programming with natural language. Instead of using rigid, explicit code, you’re instructing large language models (LLMs) to route data and perform actions through plain language to automate tasks.

Using natural language in workflows isn’t new, we’ve used NLP for years to extract and process data.

What’s new is the amount of freedom we can now give language models, allowing them to handle ambiguity and make decisions dynamically.

But just because LLMs can understand nuanced language doesn’t mean they inherently validate facts or maintain data integrity.

I see them primarily as a communication layer (at least right now) that sits on top of structured systems and existing data sources.

I usually explain it like this to non-technical people: they work a bit like we do. If we don’t have access to clean, structured data, we start making things up. Same with LLMs.

So just like us, they do their best with what they’ve got. If we want better output, we need to build systems that give them reliable data or systems to work with.

So, with Agentic systems we integrate ways for them to interact with different data sources, tools and systems.

Now, just because we can use these larger models in more places, doesn’t mean we always should. LLMs shine when interpreting nuanced natural language, think customer service, research, or human-in-the-loop collaboration.

But for structured tasks (like extracting numbers and sending them somewhere) you can to use traditional approaches and automation.

LLMs aren’t inherently better at math than a calculator. So, instead of having an LLM do calculations, you give an LLM access to a calculator.

So whenever you can build parts of a workflow programmatically, that will still be the better option.

Nevertheless, LLMs are great at adapting to messy real-world input and interpreting vague instructions so combining the two can be a great way to build systems.

If you are very new to this, and you’re still confused, go browse some of my other stuff that explain this in more detail. It may make more sense when we build later.

Agentic frameworks and LangGraph

I know a lot of people jump straight to CrewAI or AutoGen for their first agent, but we have a ton of options. For this article I will introduce you to LangGraph.

LangGraph is a graph-based framework built on top of LangChain. I would say it’s more technical and can be complex compared to other frameworks. However, it’s the preferred choice for many developers which is why it is worth building at least something with it.

LangGraph has a lot of abstractions though, where you may want to rebuild some of it just to be able to control and understand it better.

I will not go into detail on LangGraph here, so I decided to build a quick guide for those that need to get a review.

If you want to get an overview of the different frameworks out there, I have done some writing on this here which generated north of 100k readers.

I should note that I prefer no framework at all when I build nowadays but I copy a lot from what I have picked up from different frameworks so it’s still worth it to learn how to work with them.

As for this use case, you’ll be able to run the workflow without coding anything, but if you’re here to learn you may also want to understand how LangGraph works.

Single vs. multi-agent systems

Before we dig into building, let’s just first cover the differences with single vs multi-agent systems.

If you build a system around one LLM and give it a bunch of tools you want it to use, you’re working with a single-agent workflow. It’s fast, and if you’re new to agentic AI, it might seem like the model should just figure things out on its own.

But the thing is these workflows are just another form of system design.

Like any software project, you need to plan the process, define the steps, structure the logic, and decide how each part should behave.

This is where multi-agent workflows come in.

Not all of them are hierarchical or linear though, some are collaborative. Collaborative workflows would then also fall into the more flexible approach that I find more difficult to work with, at least as it is now with the capabilities that exist.

However, collaborative workflows do also break apart different functions into their own modules.

Collaborative workflows are great to start with when you’re just playing around, but they don’t always give you the precision needed for actual tasks.

For the workflow I will build here, I already know how the APIs should be used so it’s my job to guide the system to use it the right way.

We’ll go through comparing a single-agent setup with a hierarchical multi-agent system, where a lead agent delegates tasks across a small team so you can see how they behave in practice.

Building a single agent

To build a single agent we work with one LLM and one system prompt for it, and then we give it access to several tools.

It’s up to the agent to decide which tool to use and when, based on the user’s question.

The challenge with a single agent is control.

No matter how detailed the system prompt is, the model may not follow our requests (this can happen in more controlled environments too). If we give it too many tools or options, there’s a good chance it won’t use all of them or even use the right ones.

There is only so much we can set in the instructions for it to act like we want it to.

To illustrate this, we’ll build the tech news agent that has access to several API endpoints with custom data with several options as parameters in the tools.

It’s up to the agent to decide how many to use and how to setup the final summary.

Remember, I build these workflows using LangGraph. I won’t go into LangGraph in depth here, so if you want to learn the basics to be able to tweak the code, go here (the guide is from April 2025).

You can find the single-agent workflow here. To run it, you’ll need to have a login for LangSmith, where you get access to LangSmith Studio, and python 3 installed.

Once you have a login, you can clone the single agent workflow to your computer.

It will have this structure.

single-agent-workflow/
├─ my_agent/
│  ├─ agent.py
│  ├─ requirements.txt
│  ├─ utils/
│     ├─ nodes.py
│     ├─ state.py
│     └─ tools.py
├─ README.md  
├─ langgraph.json  

Create an .env file and add a Google API key. We’ll be using Gemini for this single agent.

GOOGLE_API_KEY=KEY_HERE

Then set up an environment.

python3.11 -m venv venv_py311

source venv_py311/bin/activate

Install the LangGraph cli.

pip install -U “langgraph-cli[inmem]”

You’ll find the requirements in the my_agent folder and you’ll install these too.

pip install -r my_agent/requirements.txt

Once these have downloaded you can open up the single agent workflow in LangSmith studio.

langgraph dev

This will automatically open up LangSmith Studio (before it was LangGraph Studio).

It’s important that you set up an assistant here as well, as the default will be set to Anthropic and we’re using Gemini for this agent.

To manage assistants, click on the button in the bottom left corner.

Make sure you create a new assistant with ‘gemini’ and click ‘create new assistant’ | Image by author

You’ll see a modal like the one above. Set the model to ‘gemini’ and then click ‘Create New Assistant.’

Once you’re back to the main screen, you can set a start message.

For this message you can tell it what you want, such as what you work with and if you want information for daily, weekly or monthly (see an example below).

{”messages”: [”I’m a tech investor, give me what’s up in tech for the last week”]}

It will look like this in LangSmith Studio.

Once you click submit, the result will be fast, as we’re working with a single agent.

It decides to first check a few trending keywords in 3 different categories and then it checks what people are saying for a few of the trending keywords.

If you want to understand how these tools are created, be sure to check inside the code but it simply uses an API.

The end result looks like this (although it depends when you’ve run it, the information will obviously differ day by day).

Here’s a summary of the trending tech topics from the past week:

Companies:
Apple: Apple is facing scrutiny over its App Store practices, including a UK monopoly case and concerns about attention to detail. There are also reports of reduced iPhone Air production and its potential move to include ads in the Maps app.
Meta: Meta is undergoing layoffs in its AI division, which has generated negative sentiment. There is also discussion around a mod that disables the recording light on Meta’s Ray-Ban glasses, raising privacy concerns.
GM: GM’s decision to ditch Apple CarPlay and Android Auto in future vehicles has sparked controversy and negative reactions from users.
Samsung: Samsung is planning to introduce ads on its smart fridges, drawing criticism. There’s also news about the upcoming Galaxy XR event and a new chief design officer.
[...]

If you want to read the whole thing without booting the agent up yourself, see it here. The result is fine, but as you can see it’s not digging all that deep.

We can of course probe it and keep asking it questions but as you can imagine if we need something more complex it would start to make shortcuts in the workflow.

The key thing is, an agent system isn’t just gonna think the way we expect, we have to actually orchestrate it to do what we want.

This is fine as long as we have a human in the loop, for Q&A, that sort of thing.

For this, if a human asks for one thing and the agent fetches that information, it would work great.

For deep research though we need to build a bit more complex of a system, we can do this with a workflow like system (one part does one thing), or we can try to build a hierarchical system where one agent (or team) is responsible for one thing.

Testing a multi-agent workflow

Building multiagent system is a lot more difficult than building a single agent with access to some tools.

To do this, you need to carefully think about the architecture beforehand and how data should flow between the agents.

The multi-agent workflow I’ll set up here uses two different teams (a research team and an editing team) with several agents under each.

Example architecture of teams/agents in our multi-agent system | Image by author

Every agent has access to a specific set of tools (not too many) and clear instructions.

This limited scope for each agent is great when working with lower level LLMs (i.e. Gemini Flash 2.0), although I always like to use a more advanced LLM for summarizing (in this case we’re using GPT-5 as the summerizer agent).

We’re introducing some new tools, like a research pad that acts as a shared space (one team writes their findings, the other reads from it). The last LLM will read everything that has been researched and edited to make a summary.

An alternative to using a research pad is to store data in a scratchpad in state, isolating short-term memory for each team or agent. But that also means thinking carefully about what each agent’s memory should include.

I also decided to build out the tools a bit more to provide richer data upfront, so the agents don’t have to fetch sources for each keyword individually. Here I’m using normal programmatic logic because I can.

A key thing to remember: if you can use normal programming logic, do it.

The workflow is set up for you here. Before loading it, make sure to add both your OpenAI and Google API keys in an .env file.

GOOGLE_API_KEY=KEY_HERE
OPENAI_API_KEY=KEY_HERE
ANTHROPIC_API_KEY=KEY_HERE

You only need to set the Anthropic key if you will be playing around with changing the agents (but it may give you an error if you don’t set it, in that case be sure to set up an assistant with only Gemini).

You’ll need to do the same as you did with the single agent workflow from here, set up an environment, install the requirements and open up the workflow.

langgraph dev

Once you’ve opened it up, you’ll see that it looks a lot more evolved than our single agent.

In this workflow, the routes (edges) are setup dynamically instead of manually like we did with the single agent. It’ll look more complex if you peek into the code.

To run it you need to give it a message like last time.

I’ve decided to prompt it a bit more here which is a bit of cheating, you can try something simpler.

{”messages”: [”{”messages“: [”I’m an investor and I’m interested in getting an update for what has happened within the week in tech, and what people are talking about (this means categories like companies, people, websites and subjects are interesting). Please also track these specific keywords: AI, Google, Microsoft, and Large Language Models”]}”]}

It’s better to set what you set before so you have something to compare it with but you do you.

Once it starts it will take quite some time, so you can walk away from it and then return in a few minutes.

The tools that the trending keywords agent has access to are more complex from the single agent so they take longer to return data.

In general, these systems take time to gather and process information and that’s just something we need to get used to.

You’ll be able to see the notes it gathers under a folder called ‘notes’ in root later. You’ll find an example here.

The final summary will look something like this:

FINAL RESEARCH SUMMARY
Tech Research Summary

Weekly Tech Investor Brief (week ending Oct 28, 2025)

Key Happenings
Oct 27: ICE signed a $5.7M contract for AI-powered social media surveillance (Reddit: r/technology)
Oct 27: “Windows 10 deadline boosts Mac sales” thread trended, highlighting OS/device churn (Hacker News)
Oct 26: Microsoft 365 Copilot arbitrary data exfiltration via Mermaid diagrams disclosed (Hacker News)
Oct 26: “It’s insulting to read AI-generated blog posts” topped HN, reflecting AI content fatigue (Hacker News)
[...]

Why It Matters

AI demand is moving from hype to operational scrutiny. Government adoption (e.g., ICE’s Oct 27 contract) signals durable budgets for AI monitoring and analytics, but also heightens regulatory and civil liberties overhang—an opening for compliant AI, privacy-tech, and auditing vendors. Enterprise posts on Microsoft 365 Copilot exfiltration and Teams attendance monitoring underscore a near-term buyer focus on security, governance, and employee trust, not just raw AI features.
Platform competition intensified. Google’s Oct 23 Earth AI updates and Google AI Studio “vibe coding” push indicate a bid to reduce time-to-production for AI apps, a likely driver of cloud/TPSU demand and developer lock-in. Meanwhile, community gravitation to open and self-hosted stacks (ComfyUI momentum; S3-compatible storage chatter) reflects a cost-control and control-residency theme—relevant for hybrid vendors and open-core plays.
Consumer backlash is shaping product roadmaps. GM’s removal of CarPlay/Android Auto is trending because it challenges perceived table-stakes features, risking brand equity and sales. Apple’s reported ads in Maps and YouTube’s deepfake measures reflect a broader tension between monetization, safety, and user experience—areas where differentiated policy and design can become competitive moats.
[...]

You can read the whole thing here instead if you want to check it out without running this yourself.

The news will obviously vary depending on when you run the workflow. I ran it the 28th of October so the example report will be for this date.

As for the results, I’ll let you decide for yourself the difference between using a more complex system versus a simple one, and how it gives us more control over the process.

Some ending notes

I’m working with a good data source here. Without that, you’d need to add a lot more error handling, which would slow everything down even more.

Clean and structured data is key. Without it, the LLM won’t perform at its best. Even with solid data, it’s not perfect. You still need to work on the agents to make sure they do what they’re supposed to.

You’ve probably already noticed the system works but it’s not quite there yet. There are still several things that need improvement: parsing the user’s query into a more structured format and adding guardrails so agents always use their tools. We may also want to make sure the system will summarize more effectively to keep the research doc concise.

We need to introduce better error handling, and perhaps “long-term” memory to better understand what the user actually needs. State (short-term memory) is especially important if you want to optimize for performance and cost.

Right now, we’re just pushing every message into state and giving all agents access to it, which isn’t ideal. We really want to separate state between the teams. In this case, it’s something I haven’t done, but you can try it by introducing a scratchpad in the state schema to isolate what each team knows.

Regardless, I hope it was a fun experience to understand the results we can get by building different agentic systems. If you liked it, be sure to like, comment or share it.

❤

This is an article dated May 2024 but has been re-published here.

Text classification models aren’t new, but the bar for how quickly they can be built and how well they perform has improved.

The transformer-based model I will fine-tune here is more than 1000 times smaller than GPT-3.5 Turbo. It will perform consistently better for this use case because it will be specifically trained for it.

The idea is to optimize AI workflows where smaller models excel, particularly in handling redundant tasks where larger models are simply overkill.

I’ve previously talked about this, where I built a slightly larger keyword extractor for tech-focused content using a sequence-to-sequence transformer model. I also went through the different models and what they excelled at.

For this piece, I’m diving into text classification with transformers, where encoder models do well. I’ll train a pre-trained encoder model with binary classes to identify clickbait versus factual articles. However, you may train it for a different use case.

You’ll find the finished model here.

Most organizations use open-source LLMs such as Mistral and Llama to transform their datasets for training, but what I’ll do here is create the training data altogether using Phi-3 via Ollama.

There is always the risk that the model will overfit when using data from a large language model, but in this case, it performed fine, so I’m getting on the artificial data train. However, you will have to be careful and look at the metrics once it is in training.

As for building a text classifier to identify clickbait titles, I think we can agree that some clickbait can be good as it keeps things interesting. I tried the finished model on various titles I made up, and found that having only factual content can be a bit dull.

These issues always seem clear-cut, then you dive into them, and they are more nuanced than you considered. The question that popped into my head was, ‘What’s good clickbait content versus bad clickbait content?’ A platform will probably need a bit of both to keep people reading.

I used the new model on all my own content, and none of my titles were identified as clickbait. I’m not sure if that’s something good or not.

If you’re new to transformer encoder models like BERT, this is a good learning experience. If you are not new to building text classification models with transformers, you might find it interesting to see if synthetic data worked well and to look at my performance metrics for this model.

As we all know, it’s easier to use fake data than to access the real thing.

Introduction

I got inspiration for this piece from Fabian Ridder as he was using ChatGPT to identify clickbait and factual articles to train a model using FastText. I thought this case would be great for a smaller transformer model.

The model we’re building will use synthetic data rather than the real thing, though. The process will be quick, as it will only take about an hour or so to generate data with Phi-3 and a few minutes to train it. The model will be very small, with only 11M parameters.

As we’re using binary classes, i.e., clickbait or factual, we will be able to achieve 99% accuracy. The model will have the ability to interpret nuanced texts much better than FastText though.

The cost of training will be zero, and I have already prepared the dataset that we’ll use for this. However, you may generate your own data for another use case.

If you want to dive into training the model, you can skip the introduction where I provide some information on encoder models and the tasks they excel in.

Encoder Models & What They Excel In

While transformers have introduced amazing capabilities in text generation, they have also offered improvements within other NLP tasks, such as text classification and extraction.

The distinction between model architectures is a bit blurry but it’s useful to understand that different transformer models were originally built for different tasks.

A decoder model takes in a smaller input and outputs a larger text. GPT, which introduced impressive text generation back when, is a decoder model. A decoder primarily focuses on generating the next sentence rather than look at the text as a whole.

While larger language models offer more nuanced capabilities today, decoders were not built for tasks that involve extraction and labeling. For these tasks, we can use encoder models, which take in more input and provide a condensed output.

Encoders excel at extracting information by looking at the entire input at once to create a representation and thus are great at analyzing input data in its entirety.

I won’t go into it any more than this, but there should be a lot of information you can scout on the topic, albeit it can be a bit technical. I do go into it a bit more in this article.

So, what tasks are common with encoders? Some examples are classification — such as sentiment analysis and categorization — named entity recognition (NER), and keyword extraction, among others.

You can try a model that classifies text into twelve different emotions here. You can also look into a model that classifies hate speech as toxic here. Both of these were built with an encoder-only model, in this case, RoBERTa.

There are many base models you can work with; RoBERTa is a newer model that used more data for training and improved on BERT by optimizing its training techniques.

BERT was the first encoder-only transformer model, this one started it all by understanding language context much better than previous models. DistillBERT is a compressed version of BERT.

ALBERT uses some tricks to reduce the number of parameters, making it smaller without significantly losing performance. This is the one I’ll use for this case, as I think it will do well.

DeBERTA is an improved model that better understands word relationships and context. Generally, the bigger models will perform better on complex NLP tasks. However, they can more easily overfit if the training data is not diverse enough.

For this piece, I’m focusing on one task: text classification.

As for how hard is it to build a text classification model, it really depends on what you are asking it to do. When working with binary classes, you can achieve a high accuracy score in most cases. However, it also depends on how complex the use case is.

There are certain benchmarks you can look at to understand how BERT has performed with different open-source datasets.

I reviewed the paper “How to Fine-Tune BERT for Text Classification?” to look at these benchmarks and graphed their accuracy score with the amount of labels they were trained with below.

We see datasets with only two labels do quite well. This is what we call binary labels. What might stand out is the DBpedia dataset, which has 14 classes, yet achieved 98% accuracy as a benchmark, whereas the Yelp Review Full dataset, with only 5 classes, achieved only 70%.

Here’s where complexity comes in: Yelp reviews are very difficult to label, especially when rating stars between 1 and 5. Think about how difficult it is for a human to classify someone else’s text into a specific star rating; it really depends on how the person classifies their own reviews.

If you were to build a text classifier with the Yelp reviews dataset, you would find that 1-star and 5-star reviews are labeled correctly most of the time, but the model would struggle with 2, 3, and 4-star reviews. This is because what one person may classify as a 2-star review, the AI model might interpret as a 3-star review.

The DBpedia dataset on the other hand has texts that are easier to interpret for the model.

When we train a model, we can look at the metrics per label rather than as a whole to understand which labels are underperforming. Nevertheless, if you are working with a complex task, don’t feel discouraged if your metrics aren’t perfect.

Always try it afterwards on new data to see if it works well enough on your use case and keep working on the dataset, or switch the underlying model.

The Economics of Smaller Models

I always have a section on the cost of building and running a model. In any project, you’ll have to weigh resources and efficiency to get an outcome.

If you are just trying things out, then a bigger model with an API endpoint makes sense even though it will be computationally inefficient.

I have been running Claude Haiku to do natural language processing for a project now for a month, extracting category, topics and location from texts. This is for demonstration purposes only, but it makes sense when you want to prototype something for an organization.

However, doing zero-shot with these bigger models, will result in a lot of inconsistency, and some texts have to be disregarded altogether. Sometimes the bigger models will output absolute gibberish, but at the same time, it’s cheaper to run them for such a small project.

With your own models you will also have to host them, that’s why we spend so much time trying to make them smaller. You can naturally run them locally, but you’ll probably want to be able to use them for a development project so you’ll need to keep hosting costs in consideration.

Looking at the picture up top, I have calculated the amount of titles we can process for each instance and compared the same costs for GPT-3.5. I’m aware that it may look a bit messy, but alas it is hard to vizualise.

We can at least deduce that if we are sporadically using GPT-3.5 throughout the day for a small project, it makes sense to use it even though the costs to host the smaller model is quite low.

The breakpoint is when you are consistently processing so much data that surpasses a certain threshold. For this case, this would be when the titles to be processed exceeds 32,000 per day as the cost to keep the instance running 24/7 would equal the same price.

This calculates as if you are keeping the instance running throughout the day, if you are only processing data at certain hours of the day, it makes sense to host and then scale down to zero when it is not in use. Since it’s so small, we can also just containerize it and then host it on ECS or even Lambda for serverless inference.

When using the closed sourced LLMs for zero-shot inference, we would also need to take into account that the model hasn’t been trained for this specific case so we may get inconsistent results. So for redundant tasks where you need consistency, building your own model is a better choice.

It is also worth noting that sometimes you need models that perform on more complex tasks. Here, the cost difference might be steeper for the larger LLMs as you’ll need a better model and a longer prompt template.

Working with Synthetic Data

Transforming data with the use of LLMs isn’t new, if you’re not doing it you should. This is much faster than manually transforming thousands of data points.

I looked at what Orange, the telecom giant, had done via their AI/NLP task force — NEPAL — and they had grabbed data from various places and transformed the raw texts into instruction-like formats using GPT-3.5 and Mixtral to create data that could be used for training.

If you’re keen to read more on this you can look at the session that is provided via Nvidia’s GTC here.

But people are going further than this, using the larger language models to build the entire dataset; this is called synthetic data. It’s a smart way to build smaller specialized models with data that comes from the larger language models but that are cheaper and more efficient to host.

There are concerns at this though, where the quality of synthetic data can be questioned. Relying only on generated data might lead to models that miss nuances or biases inherent in real world data causing it to malfunction when it actually sees it.

However, it is much easier to generate synthetic data than to access the real thing.

Building the Model

I will embark on creating a very simple model here, the model is simply to identify titles as either clickbait or factual. You may build a different text classifier with more labels.

The process is straightforward and I’ll go through the entire process, the cook book we’ll work with is this one.

This tutorial will use this dataset, if you want to build your own dataset be sure to read the first section.

The Dataset

To create a synthetic dataset, we can boot up Ollama locally and run a model we want to use to build the training data. Make sure it is a commercially available model. I chose Phi-3, because it is small and it is very good.

I quite like Javascript, so I used the Ollama JS framework to build a script that could run in the background to produce a CSV file.

This script creates clickbait titles and stores it in a new CSV in your root folder. You need to change the prompt template later to produce an equal amount of titles that are factual.

As I’m using a generative text model, Phi-3, some outputs won’t be usable, but that is to be expected. It will take some time for this to run, so go do something else with your time.

Once you’re finished you can store your finished CSV file with the clickbait and factual tiles in your Google Drive. Remember to set the text and label as fields, where the text is the title and the label is whether it is clickbait or factual.

Since I’ve already prepared the dataset we’ll use, please see this script to upload your custom dataset to HuggingFace.

Looking through the dataset, you’ll see that most clickbait articles that have been generated by Phi-3 has an exclamation mark at the end of it. This is something you want to make sure doesn’t happen, so it’s important to check the work of the LLM generating the data.

Remember that the script I provided you with splits your data into a training, test and validation set. I would recommend to have at least a training and test set for training the model.

If you’ve got your dataset sorted, we can go ahead and fine-tune the model.

Dataset & Model

If you haven’t opened up the cook book, do so here. The first part of this is deciding on your dataset and then your pre-trained model.

from datasets import load_dataset, DatasetDict

dataset = load_dataset(”ilsilfverskiold/clickbait_titles_synthetic_data”)
dataset

model_name = “albert/albert-base-v2”
your_path = “classify-clickbait”

I wen’t through the different models under the introduction section, where ALBERT and DistillBERT are smaller models and BERT and RoBERTa are larger.

For this case, as it’s not overly complex, I will go for ALBERT. I’m sure BERT can do better, but ALBERT is ten times smaller. RoBERTa is too big and may produce some overfitting with this dataset.

Remember, if you’re working with a different language then look for a base model that has been trained on a corpus from at least a similar language.

If you’re working with nordic languages I can recommend KB/bert-base-swedish-cased that I used to create a model for the IPTC newscodes categories.

Prepare The Dataset

Now we need to do a few things for this to work well.

We first convert our labels to a standardized numerical format that the trainer will understand.

from sklearn.preprocessing import LabelEncoder

label_encoder = LabelEncoder()

label_encoder.fit(dataset[’train’][’label’])
def encode_labels(example):
    return {’encoded_label’: label_encoder.transform([example[’label’]])[0]}
for split in dataset:
    dataset[split] = dataset[split].map(encode_labels, batched=False)

Then we need to map the numerical representations back to the actual label names. This is so we can get the actual label names rather than the numerical reps when we do inference with the model.

from transformers import AutoConfig

unique_labels = sorted(list(set(dataset[’train’][’label’])))
id2label = {i: label for i, label in enumerate(unique_labels)}
label2id = {label: i for i, label in enumerate(unique_labels)}

config = AutoConfig.from_pretrained(model_name)
config.id2label = id2label
config.label2id = label2id

# Verify the correct labels
print(”ID to Label Mapping:”, config.id2label)
print(”Label to ID Mapping:”, config.label2id)

After this we’re ready to fetch the pre-trained model and it’s tokenizer. We use the config we set up with the labels when we import the model.

from transformers import AlbertForSequenceClassification, AlbertTokenizer

tokenizer = AlbertTokenizer.from_pretrained(model_name)
model = AlbertForSequenceClassification.from_pretrained(model_name, config=config)

If you’re using a different model such as BERT or RoBERTa, you can use AutoTokenizer and AutoModelForSequenceClassification which will automatically select the correct classes for your specified model.

This next function filters for invalid content and then makes sure the text data is properly tokenized and labeled, preparing the dataset for training.

def filter_invalid_content(example):
    return isinstance(example[’text’], str)

dataset = dataset.filter(filter_invalid_content, batched=False)

def encode_data(batch):
    tokenized_inputs = tokenizer(batch[”text”], padding=True, truncation=True, max_length=256)
    tokenized_inputs[”labels”] = batch[”encoded_label”]
    return tokenized_inputs

dataset_encoded = dataset.map(encode_data, batched=True)
dataset_encoded

dataset_encoded.set_format(type=’torch’, columns=[’input_ids’, ‘attention_mask’, ‘labels’])

We also need to fetch a data collator to handle padding for our inputs.

from transformers import DataCollatorWithPadding

data_collator = DataCollatorWithPadding(tokenizer)

Evaluation Metrics

It’s not required for you to set up any evaluation metrics, such as accuracy, precision, recall or f1. However, you do need at least accuracy to understand how the model is performing.

Accuracy measures the amount of predictions the model got right across all categories. Precision measures how often predictions for a specific category are correct. Recall tells us how well the model can identify all instances within a specific category. The F1 Score is the weighted average of Precision and Recall.

I won’t go into detail on these metrics, but there are many others that write about this. For this case, I’m more interested in how it performs on new real data rather than synthetic data. So, what I look out for are metrics that are too good, indicating that it has overfitted.

We do though set up a function that let us look at the accuracy for each label rather than as an average. This is much more relevant when you have many labels, rather than just two.

from sklearn.preprocessing import LabelEncoderfrom sklearn.metrics import accuracy_score, confusion_matriximport numpy as np

label_encoder = LabelEncoder()
label_encoder.fit(unique_labels)
def per_label_accuracy(y_true, y_pred, labels):
    cm = confusion_matrix(y_true, y_pred, labels=labels)
    correct_predictions = cm.diagonal()
    label_totals = cm.sum(axis=1)
    per_label_acc = np.divide(correct_predictions, label_totals, out=np.zeros_like(correct_predictions, dtype=float), where=label_totals != 0)
    return dict(zip(labels, per_label_acc))

We also set up the general compute metrics function. I am using all of these metrics here because this is general template I have for any text classifier, but you may decide which ones you want.

from sklearn.metrics import accuracy_score, recall_score, precision_score, f1_score

def compute_metrics(pred):
    labels = pred.label_ids
    preds = pred.predictions.argmax(-1)

    decoded_labels = label_encoder.inverse_transform(labels)
    decoded_preds = label_encoder.inverse_transform(preds)

    precision = precision_score(decoded_labels, decoded_preds, average=’weighted’)
    recall = recall_score(decoded_labels, decoded_preds, average=’weighted’)
    f1 = f1_score(decoded_labels, decoded_preds, average=’weighted’)
    acc = accuracy_score(decoded_labels, decoded_preds)

    labels_list = list(label_encoder.classes_)
    per_label_acc = per_label_accuracy(decoded_labels, decoded_preds, labels_list)

    per_label_acc_metrics = {}
    for label, accuracy in per_label_acc.items():
        label_key = f”accuracy_label_{label}”
        per_label_acc_metrics[label_key] = accuracy

    return {
        ‘accuracy’: acc,
        ‘f1’: f1,
        ‘precision’: precision,
        ‘recall’: recall,
        **per_label_acc_metrics
    }

Once you’re decently satisfied, we can move on to setting up the training arguments and the trainer.

Training the Model

Next up we set up our training arguments. Here you can tweak the epochs, batch size and learning rate.

from transformers import Trainer, TrainingArguments

training_args = TrainingArguments(
    output_dir=your_path,
    num_train_epochs=3,
    warmup_steps=500,
    per_device_train_batch_size=16,
    per_device_eval_batch_size=16,
    weight_decay=0.01,
    logging_steps=10,
    evaluation_strategy=’steps’,
    eval_steps=100,
    learning_rate=2e-5,
    save_steps=1000,
    gradient_accumulation_steps=2
)

I chose to go with a learning rate and epochs based on the paper “How to Fine-Tune BERT for Text Classification?” but decreased the batch size.

Now we can go ahead and set up the trainer, with everything we’ve prepared, and run it.

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=dataset_encoded[’train’],
    eval_dataset=dataset_encoded[’test’],
    compute_metrics=compute_metrics,
    tokenizer=tokenizer,
    data_collator=data_collator,
)

trainer.train()

When in training, you need to look out for overfitting. As both the training and evaluation datasets are synthetic, the typical signs of overfitting might be unclear.

Keep an eye on the accuracy and loss for both the training and evaluation datasets. I.e. very low training and validation loss, along with too stellar evaluation metrics could be a sign of overfitting.

But remember binary classes with less complex tasks usually perform well.

You’ll see my results for one run I made below.

As you can see from the training metrics, they are a bit too good. Validation loss is fluctuating as well. This could be a very bad sign so you have to make sure to test the model on real data once it has finished training.

If you’re training a model with several classes, perhaps even with a skewed dataset, don’t worry if the average evaluation metrics aren’t great. Look at the metrics per label.

Evaluating the Model

Once it’s done training, you can run the final evaluation metrics, save the model and then save the state. This will build the metrics for when you push it to the hub for your model page.

trainer.evaluate()
trainer.save_model(your_path)
trainer.save_state()

Now you can run the HuggingFace pipeline in your notebook to test it.

from transformers import pipeline
pipe = pipeline(’text-classification’, model=your_path)

example_titles = [
    “grab an example title”,
    “grab another example title”,
    “and another xample title”
]

for title in example_titles:
    result = pipe(title)
    print(f”Title: {title}”)
    print(f”Output: {result[0][’label’]}”)

Mine did fine on test data, however it missed a few clickbait articles that I personally found to be clickbait. For a production use case, it’s better to build a more diverse dataset (especially with synthetic data) so it can perform well on new real data.

Nevertheless, if you’re not satisfied, then you go back to the dataset, redo it or try with a different model.

If you are wondering, I have indeed gotten stellar results on some runs and less-than-stellar results on other runs with the same data, the same training parameters, and the same seed.

Testing the Model

Before you push the model, you can also test the model against other alternatives.

I asked GPT-3.5 to tell me which titles it thought was clickbait and factual, and it did really well which is to be expected, it is more than 1000x larger than Albert.

We can also compare some titles to what a fine-tuned FastText says versus the fine-tuned transformer encoder model.

Using FastText is very simple and computationally efficient, but it treats words in isolation and lacks deep contextual understanding.

Therefore, FastText doesn’t capture the context and nuances of language as well as a model that is transformer based.

Push to the Hub

If you’re satisfied with your model, you can push it to the HuggingFace hub to store it there.

You simply login with a write token you can find in your HuggingFace account under Settings.

!huggingface-cli login

And then push it.

tokenizer.push_to_hub(”username/classify-clickbait”)
trainer.push_to_hub(”username/classify-clickbait”)

Push the tokenizer just in case, especially if you are working with a version of Albert.

Now you can use it directly from there, mine you’ll find there.

Optimization Techniques

If you want to use a larger model like BERT, you can apply different techniques so you can distill it further after fine-tuning. I didn’t find it that much more successful than just using ALBERT, at least for this case.

BERT on its own though performed much better in general. Although I really like RoBERTa for most cases, it was prone to overfit on this specific dataset either because it was too small, not good enough or too artificial.

For every case you’ll have to estimate how much performance you can sacrifice for efficiency and eventually you learn which models do well in what situation.

Ending Notes

Would the model have performed better if we had used real data? It’s possible, but the accuracy may be lower unless the dataset is meticulously sorted.

This is hard work.

Using synthetic data can get the job done very quickly so you get up a prototype to work with. Synthetic data is much cleaner to work with.

You are also free to work with the larger open source LLMs, so it doesn’t break any rules for people that can’t access high quality data without breaching protocol.

I did not put down time and effort into building this dataset, but in all cases you should make sure you have varied data the model can learn from.

Hopefully this piece was useful and gave you some inspiration and ideas on how to work with the smaller models.

❤

This is an article dated November 2025 but has been re-published here.

One of the things you’ll stumble upon when doing AI engineering work is that there’s no real blueprint to follow.

Yes, for the most basic parts of retrieval (the “R” in RAG), you can chunk documents, use semantic search on a query, re-rank the results, and so on. This part is well known.

But once you start digging into this area, you begin to ask questions like: how can we call a system intelligent if it’s only able to read a few chunks here and there in a document?

So, how do we make sure it has enough information to actually answer intelligently?

Soon, you’ll find yourself going down a rabbit hole, trying to discern what others are doing in their own orgs, because none of this is properly documented, and people are still building their own setups.

This will lead you to implement various optimization strategies: building custom chunkers, rewriting user queries, using different search methods, filtering with metadata, and expanding context to include neighboring chunks.

Hence why I’ve now built a rather bloated retrieval system to show how some of these parts work. So, let’s walk through it so we can see the results of each step, but also to discuss the trade-offs.

To demo this system in public, I’ve embedded 150 recent ArXiv papers (2,250 pages) that mention RAG. This means the system we’re testing here is designed for scientific papers, and all the test queries will be RAG-related.

I have collected the raw outputs for each step for a few queries in this repository, if you want to look at the whole thing in detail.

For the tech stack, I’m using Qdrant and Redis to store data, and Cohere and OpenAI for the LLMs. I do not rely on any framework to build the pipelines (as it makes it harder to debug).

As always, I do a quick review of what we’re doing for beginners, so if RAG is already familiar to you, feel free to skip the first section.

Recap retrieval & RAG

When you work with AI knowledge systems like Copilot (where you feed it your custom docs to answer from) you work with a RAG system.

RAG stands for Retrieval Augmented Generation and is separated into two parts, the retrieval part and the generation part.

Retrieval refers to the process of retrieving information, using keyword and semantic matching, based on a user query. The generation part is where the LLM comes in and answers based on the provided context and the query.

For anyone new to RAG it may seem like a chunky way to build systems. Shouldn’t an LLM do most of the work on its own?

Unfortunately, LLMs are static, and we need to engineer systems so that each time we call on them, we give them everything they need upfront so they can answer the question.

I have written about building RAG bots for Slack before. This one uses standard chunking methods, if you’re keen to get a sense of how people build something simple.

This article goes a step further and tries to rebuild the entire retrieval pipeline without any frameworks, to do some fancy stuff like build a multi-query optimizer, fuse results, and expand the chunks to build better context for the LLM.

As we’ll see though, all of these fancy additions we’ll have to pay for in latency and additional work.

Processing different documents

As with any data engineering problem, your first hurdle will be to architect how to store data. With retrieval, we focus on something called chunking, and how you do it and what you store with it is essential to building a well-engineered system.

When we do retrieval, we search text, and to do that we need to separate the text into different chunks. These pieces of text are what we’ll later search to find a match for a query.

Most simple systems use general chunkers, simply splitting the full text by length, paragraph, or sentence.

But every document is different, so by doing this you risk losing context.

To understand this, you should look at different documents to see how they all follow different structures. You’ll have an HR document with clear section headers, and API docs with unnumbered sections using code blocks and tables.

If you applied the same chunking logic to all of these, you’d risk splitting each text the wrong way. This means that once the LLM gets the chunks of information, it will be incomplete, which may cause it to fail at producing an accurate answer.

Furthermore, for each chunk of information, you also need to think about the data you want it to hold.

Should it contain certain metadata so the system can apply filters? Should it link to similar information so it can connect data? Should it hold context so the LLM understands where the information comes from?

This means the architecture of how you store data becomes the most important part. If you start storing information and later realize it’s not enough, you’ll have to redo it. If you realize you’ve complicated the system, you’ll have to start from scratch.

This system will ingest Excel and PDFs, focusing on adding context, keys, and neighbors. This will allow you to see what this looks like when doing retrieval later.

For this demo, I have stored data in Redis and Qdrant. We use Qdrant to do semantic, BM25, and hybrid search, and to expand content we fetch data from Redis.

Ingesting tabular files

First we’ll go through how you can chunk tabular data, add context, and keep information connected with keys.

When dealing with already structured tabular data, like in Excel files, it might seem like the obvious approach is to let the system search it directly. But semantic matching is actually quite effective for messy user queries.

SQL or direct queries only work if you already know the schema and exact fields. For instance, if you get a query like “Mazda 2023 specs” from a user, semantically matching rows will give us something to go on.

I’ve talked to companies that wanted their system to match documents across different Excel files. To do this, we can store keys along with the chunks (without going full KG).

So for instance, if we’re working with Excel files containing purchase data, we could ingest data for each row like so:

{
    “chunk_id”: “Sales_Q1_123::row::1”,
    “doc_id”: “Sales_Q1_123:1234”
    “location”: {”sheet_name”: “Sales Q1”, “row_n”: 1},
    “type”: “chunk”,
    “text”: “OrderID: 1001234f67 \n Customer: Alice Hemsworth \n Products: Blue sweater 4, Red pants 6”,
    “context”: “Quarterly sales snapshot”,
    “keys”: {”OrderID”: “1001234f67”},
}

If we decide later in the retrieval pipeline to connect information, we can do standard search using the keys to find connecting chunks. This allows us to make quick hops between documents without adding another router step to the pipeline.

Very simplified — connecting keys between tabular documents | Image by author

We can also set a summary for each document. This acts as a gatekeeper to chunks.

{
    “chunk_id”: “Sales_Q1::summary”,
    “doc_id”: “Sales_Q1_123:1234”
    “location”: {”sheet_name”: “Sales Q1”},
    “type”: “summary”,
    “text”: “Sheet tracks Q1 orders for 2025, type of product, and customer names for reconciliation.”,
    “context”: “”
}

The gatekeeper summary idea might be a bit complicated to understand at first, but it also helps to have the summary stored at the document level if you need it when building the context later.

When the LLM sets up this summary (and a brief context string), it can suggest the key columns (i.e. order IDs and so on).

As a note, always set the key columns manually if you can, if that’s not possible, set up some validation logic to make sure the keys aren’t just random (it can happen that an LLM will choose weird columns to store while ignoring the most vital ones).

For this system with the ArXiv papers, I’ve ingested two Excel files that contain information on title and author level.

The chunks will look something like this:

{
    “chunk_id”: “titles::row::8817::250930134607”,
    “doc_id”: “titles::250930134607”,
    “location”: {
      “sheet_name”: “titles”,
      “row_n”: 8817
    },
    “type”: “chunk”,
    “text”: “id: 2507 2114\ntitle: Gender Similarities Dominate Mathematical Cognition at the Neural Level: A Japanese fMRI Study Using Advanced Wavelet Analysis and Generative AI\nkeywords: FMRI; Functional Magnetic Resonance Imaging; Gender Differences; Machine Learning; Mathematical Performance; Time Frequency Analysis; Wavelet\nabstract_url: https://arxiv.org/abs/2507.21140\ncreated: 2025-07-23 00:00:00 UTC\nauthor_1: Tatsuru Kikuchi”,
    “context”: “Analyzing trends in AI and computational research articles.”,
    “keys”: {
      “id”: “2507 2114”,
      “author_1”: “Tatsuru Kikuchi”
    }
  }

These Excel files were strictly not necessary (the PDF files would have been enough), but they’re a way to demo how the system can look up keys to find connecting information.

I created summaries for these files too.

{
    “chunk_id”: “titles::summary::250930134607”,
    “doc_id”: “titles::250930134607”,
    “location”: {
      “sheet_name”: “titles”
    },
    “type”: “summary”,
    “text”: “The dataset consists of articles with various attributes including ID, title, keywords, authors, and publication date. It contains a total of 2508 rows with a rich variety of topics predominantly around AI, machine learning, and advanced computational methods. Authors often contribute in teams, indicated by multiple author columns. The dataset serves academic and research purposes, enabling catego”,
  }

We also store information in Redis at document level, which tells us what it’s about, where to find it, who is allowed to see it, and when it was last updated. This will allow us to update stale information later.

Now let’s turn to PDF files, which are the worst monster you’ll deal with.

Ingesting PDF docs

To process PDF files, we do similar things as with tabular data, but chunking them is much harder, and we store neighbors instead of keys.

To start processing PDFs, we have several frameworks to work with, such as LlamaParse and Docling, but none of them are perfect, so we have to build out the system further.

PDF documents are very hard to process, as most don’t follow the same structure. They also often contain figures and tables that most systems can’t handle correctly.

Nevertheless, a tool like Docling can help us at least parse normal tables properly and map out each element to the correct page and element number.

From here, we can create our own programmatic logic by mapping sections and subsections for each element, adding titles to each chunk, and smart-merging snippets so chunks read naturally (i.e. don’t split mid-sentence).

We also make sure to group chunks by section, keeping them together by linking their IDs in a field called neighbors.

This allows us to keep the chunks small but still expand them after retrieval.

The end result will be something like below:

{
    “chunk_id”: “S3::C02::251009105423”,
    “doc_id”: “2507.18910v1”,
    “location”: {
      “page_start”: 2,
      “page_end”: 2
    },
    “type”: “chunk”,
    “text”: “1 Introduction\n\n1.1 Background and Motivation\n\nLarge-scale pre-trained language models have demonstrated an ability to store vast amounts of factual knowledge in their parameters, but they struggle with accessing up-to-date information and providing verifiable sources. This limitation has motivated techniques that augment generative models with information retrieval. Retrieval-Augmented Generation (RAG) emerged as a solution to this problem, combining a neural retriever with a sequence-to-sequence generator to ground outputs in external documents [52]. The seminal work of [52] introduced RAG for knowledge-intensive tasks, showing that a generative model (built on a BART encoder-decoder) could retrieve relevant Wikipedia passages and incorporate them into its responses, thereby achieving state-of-the-art performance on open-domain question answering. RAG is built upon prior efforts in which retrieval was used to enhance question answering and language modeling [48, 26, 45]. Unlike earlier extractive approaches, RAG produces free-form answers while still leveraging non-parametric memory, offering the best of both worlds: improved factual accuracy and the ability to cite sources. This capability is especially important to mitigate hallucinations (i.e., believable but incorrect outputs) and to allow knowledge updates without retraining the model [52, 33].”,
    “context”: “Systematic review of RAG’s development and applications in NLP, addressing challenges and advancements.”,
    “section_neighbours”: {
      “before”: [
        “S3::C01::251009105423”
      ],
      “after”: [
        “S3::C03::251009105423”,
        “S3::C04::251009105423”,
        “S3::C05::251009105423”,
        “S3::C06::251009105423”,
        “S3::C07::251009105423”
      ]
    },
    “keys”: {}
  }

When we set up data like this, we can consider these chunks as seeds. We are searching for where there may be relevant information based on the user query, and expanding from there.

The difference from simpler RAG systems is that we try to take advantage of the LLM’s growing context window to send in more information (but there are obviously trade offs to this).

You’ll be able to see a messy solution of what this looks like when building the context in the retrieval pipeline later.

Building the retrieval pipeline

Since I’ve built this pipeline piece by piece, it allows us to test each part and go through why we make certain choices in how we retrieve and transform information before handing it over to the LLM.

We’ll go through semantic, hybrid, and BM25 search, building a multi-query optimizer, re-ranking results, expanding content to build the context, and then handing the results to an LLM to answer.

We’ll end the section with some discussion on latency, unnecessary complexity, and what to cut to make the system faster.

If you want to look at the output of several runs of this pipeline, go to this repository.

Semantic and hybrid search

The first part of this pipeline is to make sure we are getting back relevant documents for a user query. To do this, we work with semantic, BM25, and hybrid search.

For simple retrieval systems, people will usually just use semantic search. To perform semantic search, we embed dense vectors for each chunk of text using an embedding model.

If this is new to you, note that embeddings represent each piece of text as a point in a high-dimensional space. The position of each point reflects how the model understands its meaning, based on patterns it learned during training.

Texts with similar meanings will then end up close together.

This means that if the model has seen many examples of similar language, it becomes better at placing related texts near each other, and therefore better at matching a query with the most relevant content.

I have written about this before, using clustering on various embeddings models to see how they performed for a use case, if you’re keen to learn more.

To create dense vectors, I used OpenAI’s Large embedding model, since I’m working with scientific papers. This model is more expensive than their small one and perhaps not ideal for this use case.

I would look into specialized models for specific domains or consider fine-tuning your own. Because remember if the embedding model hasn’t seen many examples similar to the texts you’re embedding, it will be harder to match them to relevant documents.

To support hybrid and BM25 search, we also build a lexical index (sparse vectors). BM25 works on exact tokens (for example, “ID 826384”) instead of returning “similar-meaning” text the way semantic search does.

Do remember that BM25 does not match partial tokens, if users ask for an ID that starts with 8263 it may match semantically but it’s not a guarantee.

To test semantic search, we’ll set up a query that I think the papers we’ve ingested can answer, such as: “Why do LLMs get worse with longer context windows and what to do about it?”

[1] score=0.5071 doc=docs_ingestor/docs/arxiv/2508.15253.pdf chunk=S3::C02::251009131027
  text: 1 Introduction This challenge is exacerbated when incorrect yet highly ranked contexts serve as hard negatives. (...)
[2] score=0.5022 doc=docs_ingestor/docs/arxiv/2508.19614.pdf chunk=S3::C03::251009132038
  text: 1 Introductions Despite these advances, LLMs might underutilize accurate external contexts, disproportionately favoring internal parametric knowledge during generation [50, 40]. This overreliance risks propagating outdated information or hallucinations, undermining the trustworthiness of RAG systems. Surprisingly, recent studies reveal a paradoxical phenomenon: injecting noise-random documents or tokens-to retrieved contexts that already contain answer-relevant snippets can improve the generation accuracy (...)
[3] score=0.4982 doc=docs_ingestor/docs/arxiv/2508.19614.pdf chunk=S6::C18::251009132038
  text: 4 Experiments 4.3 Analysis Experiments Qualitative Study In Table 4, we analyze a case study from the NQ dataset using the Llama2-7B model, evaluating four decoding strategies: GD(0), CS, DoLA, and LFD. Despite access to groundtruth documents, both GD(0) and DoLA generate incorrect answers (e.g., ‘18 minutes’), suggesting limited capacity to integrate contextual evidence (...)
[4] score=0.4857 doc=docs_ingestor/docs/arxiv/2507.23588.pdf chunk=S6::C03::251009122456
  text: 4 Results Figure 4: Change in attention pattern distribution in different models. For DiffLoRA variants we plot attention mass for main component (green) and denoiser component (yellow). Note that attention mass is normalized by the number of tokens in each part of the sequence. The negative attention is shown after it is scaled by λ (...)
[5] score=0.4838 doc=docs_ingestor/docs/arxiv/2508.15253.pdf chunk=S3::C03::251009131027
  text: 1 Introduction To mitigate context-memory conflict, existing studies such as adaptive retrieval (Ren et al., 2025; Baek et al., 2025) and the decoding strategies (Zhao et al., 2024; Han et al., 2025) adjust the influence of external context either before or during answer generation. However, due to the LLM’s limited capacity in detecting conflicts, (...)
[6] score=0.4827 doc=docs_ingestor/docs/arxiv/2508.05266.pdf chunk=S27::C03::251009123532
  text: B. Subclassification Criteria for Misinterpretation of Design Specifications Initially, regarding long-context scenarios, we observed that directly prompting LLMs to generate RTL code based on lengthy contexts often resulted in certain code segments failing to accurately reflect high-level requirements. (...)
[7] score=0.4798 doc=docs_ingestor/docs/arxiv/2508.19614.pdf chunk=S3::C02::251009132038
  text: 1 Introductions Figure 1: Illustration for layer-wise behavior in LLMs for RAG. Given a query and retrieved documents with the correct answer (’Real Madrid’), shallow layers capture local context, middle layers focus on answer-relevant content, while deep layers may over-rely on internal knowledge and hallucinate (e.g., ‘Barcelona’). Our proposal, LFD fuses middle-layer signals into the final output to preserve external knowledge and improve accuracy.(...)

From the results above, we can see that it’s able to match some interesting passages where they discuss topics that can answer the query but not all are the most relevant.

If we try BM25 (which matches exact tokens) with the same query, we get back these results:

[1] score=22.0764 doc=docs_ingestor/docs/arxiv/2507.20888.pdf chunk=S4::C27::251009115003
  text: 3 APPROACH 3.2.2 Project Knowledge Retrieval Similar Code Retrieval. Similar snippets within the same project are valuable for code completion, even if they are not entirely replicable. In this step, we also retrieve similar code snippets. (...)
[2] score=17.4931 doc=docs_ingestor/docs/arxiv/2508.09105.pdf chunk=S20::C08::251009124222
  text: C. Ablation Studies Ablation result across White-Box attribution: Table V shows the comparison result in methods of WhiteBox Attribution with Noise, White-Box Attrition with Alternative Model and our current method Black-Box zero-gradient (...)
[3] score=17.1458 doc=docs_ingestor/docs/arxiv/2508.05100.pdf chunk=S4::C03::251009123245
  text: Preliminaries Based on this, inspired by existing analyses (Zhang et al. 2024c), we measure the amount of information a position receives using discrete entropy, as shown in the following equation: which quantifies how much information t i receives from the attention perspective. This insight suggests that LLMs struggle with longer sequences when not trained on them (...)

Here, the results are lackluster for this query except for the third one but sometimes queries include specific keywords we need to match, where BM25 is the better choice.

We can test this by changing the query to “papers from Anirban Saha Anik” using BM25.

[1] score=62.3398 doc=authors.csv chunk=authors::row::1::251009110024
  text: author_name: Anirban Saha Anik n_papers: 2 article_1: 2509.01058 article_2: 2507.07307
[2] score=56.4007 doc=titles.csv chunk=titles::row::24::251009110138
  text: id: 2509.01058 title: Speaking at the Right Level: Literacy-Controlled Counterspeech Generation with RAG-RL keywords. UTC author_1: Xiaoying Song author_2: Anirban Saha Anik author_3: Dibakar Barua author_4: Pengcheng Luo author_5: Junhua Ding author_6: Lingzi Hong
[3] score=56.2614 doc=titles.csv chunk=titles::row::106::251009110138
  text: id: 2507.07307 title: Multi-Agent Retrieval-Augmented Framework for Evidence-Based Counterspeech Against Health Misinformation keywords ... author_1: Anirban Saha Anik author_2: Xiaoying Song author_3: Elliott Wang author_4: Bryan Wang author_5: Bengisu Yarimbas author_6: Lingzi Hong

All the results above mention “Anirban Saha Anik,” which is exactly what we’re looking for.

If we ran this with semantic search, it would return not just the name “Anirban Saha Anik” but similar names as well.

[1] score=0.5810 doc=authors.csv chunk=authors::row::1::251009110024
  text: author_name: Anirban Saha Anik n_papers: 2 article_1: 2509.01058 article_2: 2507.07307
[2] score=0.4499 doc=authors.csv chunk=authors::row::55::251009110024
  text: author_name: Anand A. Rajasekar n_papers: 1 article_1: 2508.0199
[3] score=0.4320 doc=authors.csv chunk=authors::row::59::251009110024
  text: author_name: Anoop Mayampurath n_papers: 1 article_1: 2508.14817
[4] score=0.4306 doc=authors.csv chunk=authors::row::69::251009110024
  text: author_name: Avishek Anand n_papers: 1 article_1: 2508.15437
[5] score=0.4215 doc=authors.csv chunk=authors::row::182::251009110024
  text: author_name: Ganesh Ananthanarayanan n_papers: 1 article_1: 2509.14608

This is a good example of how semantic search isn’t always the ideal method as similar names don’t necessarily mean they’re relevant to the query.

So, there are cases where semantic search is ideal, and others where BM25 (token matching) is the better choice.

We can also use hybrid search, which combines semantic and BM25.

You’ll see the results below from running hybrid search on the original query: “why do LLMs get worse with longer context windows and what to do about it?”

[1] score=0.5000 doc=docs_ingestor/docs/arxiv/2508.15253.pdf chunk=S3::C02::251009131027
  text: 1 Introduction This challenge is exacerbated when incorrect yet highly ranked contexts serve as hard negatives... This misalignment leads to overriding correct internal representations, resulting in substantial performance degradation on questions that the model initially answered correctly...
[2] score=0.5000 doc=docs_ingestor/docs/arxiv/2507.20888.pdf chunk=S4::C27::251009115003
  text: 3 APPROACH 3.2.2 Project Knowledge Retrieval Similar Code Retrieval. (...)
[3] score=0.4133 doc=docs_ingestor/docs/arxiv/2508.19614.pdf chunk=S3::C03::251009132038
  text: 1 Introductions Despite these advances, LLMs might underutilize accurate external contexts, disproportionately favoring internal parametric knowledge during generation [50, 40]. This overreliance risks propagating outdated information or hallucinations, undermining the trustworthiness of RAG systems. (...)
[4] score=0.1813 doc=docs_ingestor/docs/arxiv/2508.19614.pdf chunk=S6::C18::251009132038
  text: 4 Experiments 4.3 Analysis Experiments Qualitative Study In Table 4, we analyze a case study from the NQ dataset using the Llama2-7B model, evaluating four decoding strategies.. suggesting limited capacity to integrate contextual evidence. Similarly, while CS produces a partially relevant response (’Texas Revolution’), it exhibits reduced factual consistency with the source material. In contrast, LFD demonstrates superior utilization of retrieved context...

I found semantic search worked best for this query, which is why it can be useful to run multi-queries with different search methods to fetch the first chunks (though this also adds complexity).

So, let’s turn to building something that can transform the original query into several optimized versions, and fuse the results.

Multi-query optimizer

For this part we look at how we can optimize messy user queries by generating multiple targeted variations and selecting the right search method for each. It can improve recall but it introduces trade-offs.

All the agent abstraction systems you see usually transform the user query when performing search.

For example, when you use the QueryTool in LlamaIndex, it uses an LLM to optimize the incoming query.

We can rebuild this part ourselves, but instead we give it the ability to create multiple queries, while also setting the search method. When you’re working with more documents, you could also have it set filters at this stage.

As for creating a lot of queries, I would try to keep it simple, as issues here will cause low-quality outputs in retrieval. The more unrelated queries the system generates, the more noise it introduces into the pipeline.

The function I’ve created here will generate 1–3 academic-style queries, along with the search method to be used, based on a messy user query.

Original query:
why is everyone saying RAG doesn’t scale? how are people fixing that?

Generated queries:
- hybrid: RAG scalability issues
- hybrid: solutions to RAG scaling challenges

We will get back results like these:

Query 1 (hybrid) top 20 for query: RAG scalability issues

[1] score=0.5000 doc=docs_ingestor/docs/arxiv/2507.18910.pdf chunk=S22::C05::251104142800
  text: 7 Challenges of RAG 7.2.1 Scalability and Infrastructure Deploying RAG at scale requires substantial engineering to maintain large knowledge corpora and efficient retrieval indices...
[2] score=0.5000 doc=docs_ingestor/docs/arxiv/2507.07695.pdf chunk=SDOC::SUM::251104135247
  text: This paper proposes the KeyKnowledgeRAG (K2RAG) framework to enhance the efficiency and accuracy of Retrieval-Augment-Generate (RAG) systems. It addresses the high computational costs and scalability issues associated with naive RAG implementations (...)

[...]

Query 2 (hybrid) top 20 for query: solutions to RAG scaling challenges

[1] score=0.5000 doc=docs_ingestor/docs/arxiv/2507.18910.pdf chunk=S22::C05::251104142800
  text: 7 Challenges of RAG 7.2.1 Scalability and Infrastructure Deploying RAG at scale requires substantial engineering to maintain large knowledge corpora and efficient retrieval indices. Systems must handle millions or billions of documents, demanding significant computational resources, efficient indexing, distributed computing infrastructure, and cost management strategies [21]...
[2] score=0.5000 doc=docs_ingestor/docs/arxiv/2508.05100.pdf chunk=S3::C06::251104155301
  text: Introduction Empirical analyses across multiple real-world benchmarks reveal that BEE-RAG fundamentally alters the entropy scaling laws governing conventional RAG systems, which provides a robust and scalable solution for RAG systems dealing with long-context scenarios....
[...]

We can also test the system with specific keywords like names and IDs to make sure it chooses BM25 rather than semantic search.

Original query:
any papers from Chenxin Diao?

Generated queries:
- BM25: Chenxin Diao

This will pull up results where Chenxin Diao is clearly mentioned.

If you want to do this better, you can build a retrieval system that generates a few example queries based on the input, so when the original query comes in, you fetch examples to help guide the optimizer.

This helps because smaller models aren’t great at transforming messy human queries into ones with more precise academic phrasing.

To give you an example, when a user is asking why the LLM is lying, the optimizer may transform the query to something like “causes of inaccuracies in large language models” rather than directly look for “hallicunations.”

After we fetch results in parallel, we fuse them. The result will look something like this:

RRF Fusion top 38 for query: why is everyone saying RAG doesn’t scale? how are people fixing that?

[1] score=0.0328 doc=docs_ingestor/docs/arxiv/2507.18910.pdf chunk=S22::C05::251104142800
  text: 7 Challenges of RAG 7.2.1 Scalability and Infrastructure Deploying RAG at scale requires substantial engineering to maintain large knowledge corpora and efficient retrieval indices. Systems must handle millions or billions of documents, demanding significant computational resources, efficient indexing, distributed computing infrastructure, and cost management strategies [21]....
[2] score=0.0313 doc=docs_ingestor/docs/arxiv/2507.18910.pdf chunk=S22::C42::251104142800
  text: 7 Challenges of RAG 7.5.5 Scalability Scalability challenges arise as knowledge corpora expand. Advanced indexing, distributed retrieval, and approximate nearest neighbor techniques facilitate efficient handling of large-scale knowledge bases [57]. Selective indexing and corpus curation...
[3] score=0.0161 doc=docs_ingestor/docs/arxiv/2507.07695.pdf chunk=SDOC::SUM::251104135247
  text: This paper proposes the KeyKnowledgeRAG (K2RAG) framework to enhance the efficiency and accuracy of Retrieval-Augment-Generate (RAG) systems. It addresses the high computational costs and scalability issues associated with naive RAG implementations by...
[4] score=0.0161 doc=docs_ingestor/docs/arxiv/2508.05100.pdf chunk=S3::C06::251104155301
  text: Introduction Empirical analyses across multiple real-world benchmarks reveal that BEE-RAG fundamentally alters the entropy scaling laws governing conventional RAG systems, which provides a robust and scalable solution for RAG systems dealing with long-context scenarios...
[...]

We see that there are some good matches, but also a few irrelevant ones that we’ll need to filter out further.

As a note before we move on, this is probably the step you’ll cut or optimize once you’re trying to reduce latency.

I find LLMs aren’t great at creating key queries that actually pull up useful information all that well, so if it’s not done right, it just adds more noise. So, don’t over do it with too many similar queries.

Adding a re-ranker

We do get results back from the retrieval system, and some of these are good while others are irrelevant, so most retrieval systems will use a re-ranker of some sort.

A re-ranker takes in several chunks and gives each one a relevancy score based on the original user query. You have several choices here, including using something smaller.

I’ll use Cohere’s re-ranker for this part though.

We can test this re-ranker on the first question we used in the previous section: “Why is everyone saying RAG doesn’t scale? How are people fixing that?”

At this point, we’ve already transformed the user query, done semantic or hybrid search, and fused the results before passing the chunks to the re-ranker.

The re-ranker returns these results:

[... optimizer... retrieval... fuse...]

Rerank summary:
- strategy=cohere
- model=rerank-english-v3.0
- candidates=32
- eligible_above_threshold=4
- kept=4 (reranker_threshold=0.35)

Reranked Relevant (4/32 kept ≥ 0.35) top 4 for query: why is everyone saying RAG doesn’t scale? how are people fixing that?

[1] score=0.7920 doc=docs_ingestor/docs/arxiv/2507.07695.pdf chunk=S4::C08::251104135247
  text: 1 Introduction Scalability: Naive implementations of Retrieval-Augmented Generation (RAG) often rely on 16-bit floating-point large language models (LLMs) for the generation component....
[2] score=0.4749 doc=docs_ingestor/docs/arxiv/2507.18910.pdf chunk=S22::C42::251104142800
  text: 7 Challenges of RAG 7.5.5 Scalability Scalability challenges arise as knowledge corpora expand. Advanced indexing, distributed retrieval, and approximate nearest neighbor techniques facilitate efficient handling of large-scale knowledge bases [57]...
[3] score=0.4304 doc=docs_ingestor/docs/arxiv/2507.18910.pdf chunk=S22::C05::251104142800
  text: 7 Challenges of RAG 7.2.1 Scalability and Infrastructure Deploying RAG at scale requires substantial engineering to maintain large knowledge corpora and efficient retrieval indices...
[4] score=0.3556 doc=docs_ingestor/docs/arxiv/2509.13772.pdf chunk=S11::C02::251104182521
  text: 7. Discussion and Limitations Scalability of RAGOrigin: We extend our evaluation by scaling the NQ dataset’s knowledge database to 16.7 million texts, combining entries from the knowledge database of NQ, HotpotQA, and MS-MARCO. Using the same user questions from NQ, we assess RAGOrigin’s performance under larger data volumes...

We can clearly see that it’s able to identify a few relevant chunks that we can use as seeds. Remember it only has 150 docs to go on in the first place.

You can also see that it returns multiple chunks from the same document. We’ll set this up later in the context construction, but if you want unique documents fetched, you can add some custom logic here to set the limit for unique docs rather than chunks.

We can try the re-ranker with another question: “hallucinations in RAG vs normal LLMs and how to reduce them”

[... optimizer... retrieval... fuse...]

Rerank summary:
- strategy=cohere
- model=rerank-english-v3.0
- candidates=35
- eligible_above_threshold=12
- kept=5 (threshold=0.2)

Reranked Relevant (5/35 kept ≥ 0.2) top 5 for query: hallucinations in rag vs normal llms and how to reduce them

[1] score=0.9965 doc=docs_ingestor/docs/arxiv/2508.19614.pdf chunk=S7::C03::251104164901
  text: 5 Related Work Hallucinations in LLMs Hallucinations in LLMs refer to instances where the model generates false or unsupported information not grounded in its reference data [42]. Existing mitigation strategies include multi-agent debating, where multiple LLM instances collaborate to detect inconsistencies through iterative debates [8, 14]...
[2] score=0.9342 doc=docs_ingestor/docs/arxiv/2508.05509.pdf chunk=S3::C01::251104160034
  text: ...Despite the success, these models are often criticized for their tendency to produce hallucinations, generating incorrect statements on tasks beyond their knowledge and perception (...). Recently, retrieval-augmented generation (RAG) (...) has emerged as a promising solution to alleviate such hallucinations. By dynamically leveraging external knowledge from textual corpora, RAG enables LLMs to generate more accurate and reliable responses without costly retraining (Lewis et al. 2020; Figure 1: Comparison of three paradigms. LAG exhibits greater lightweight properties compared to GraphRAG while...
[3] score=0.9030 doc=docs_ingestor/docs/arxiv/2509.13702.pdf chunk=S3::C01::251104182000
  text: ABSTRACT Hallucination remains a critical barrier to the reliable deployment of Large Language Models (LLMs) in high-stakes applications. Existing mitigation strategies, such as Retrieval-Augmented Generation (RAG) and post-hoc verification, are often reactive, inefficient, or fail to address the root cause within the generative process...
[4] score=0.9007 doc=docs_ingestor/docs/arxiv/2509.09360.pdf chunk=S2::C05::251104174859
  text: 1 Introduction Figure 1. Standard Retrieval-Augmented Generation (RAG)...aims to mitigate hallucinations by grounding model outputs in retrieved, up-to-date documents, as illustrated in Figure 1. By injecting retrieved text from re- a...
[5] score=0.8986 doc=docs_ingestor/docs/arxiv/2508.04057.pdf chunk=S20::C02::251104155008
  text: Parametric knowledge can generate accurate answers. Effects of LLM hallucinations. To assess the impact of hallucinations when large language models (LLMs) generate answers without retrieval, we conduct a controlled experiment based on a simple heuristic: if a generated answer contains numeric values, it is more likely to be affected by hallucination....

This query also performs well enough (if you look at the full chunks returned).

We can also test messier user queries, like: “why is the llm lying and rag help with this?”

[... optimizer...]

Original query:
why is the llm lying and rag help with this?

Generated queries:
- semantic: explore reasons for LLM inaccuracies
- hybrid: RAG techniques for LLM truthfulness

[...retrieval... fuse...]

Rerank summary:
- strategy=cohere
- model=rerank-english-v3.0
- candidates=39
- eligible_above_threshold=39
- kept=6 (threshold=0)

Reranked Relevant (6/39 kept ≥ 0) top 6 for query: why is the llm lying and rag help with this?

[1] score=0.0293 doc=docs_ingestor/docs/arxiv/2507.05714.pdf chunk=S3::C01::251104134926
  text: 1 Introduction Retrieval Augmentation Generation (hereafter referred to as RAG) helps large language models (LLMs) (OpenAI et al., 2024) reduce hallucinations (Zhang et al., 2023) and access real-time data 1 *Equal contribution....
[2] score=0.0284 doc=docs_ingestor/docs/arxiv/2508.15437.pdf chunk=S3::C01::251104164223
  text: 1 Introduction Large language models (LLMs) augmented with retrieval have become a dominant paradigm for knowledge-intensive NLP tasks. In a typical retrieval-augmented generation (RAG) setup, an LLM retrieves documents from an external corpus and conditions generation on the retrieved evidence (Lewis et al., 2020b; Izacard and Grave, 2021). This setup mitigates a key weakness of LLMs-hallucination-by grounding generation in externally sourced knowledge...
[3] score=0.0277 doc=docs_ingestor/docs/arxiv/2509.09651.pdf chunk=S3::C01::251104180034
  text: ...despite their versatility, LLMs are prone to generating false or misleading content, a phenomenon commonly referred to as hallucination...
[4] score=0.0087 doc=docs_ingestor/docs/arxiv/2507.07695.pdf chunk=S4::C08::251104135247
  text: 1 Introduction Scalability: Naive implementations of Retrieval-Augmented Generation (RAG) often rely on 16-bit floating-point large language models (LLMs) for the generation component. However, this approach introduces significant scalability challenges due to the increased memory demands required to host the LLM as well as longer inference times due to using a higher precision number type. To enable more efficient scaling, it is crucial to integrate methods or techniques that reduce the memory footprint and inference times of generator models...

Before we move on, I need to note that there are moments where this re-ranker doesn’t do that well, as you’ll see above from the scores.

At times it estimates that the chunks doesn’t answer the user’s question but it actually does, at least when we look at these chunks as seeds.

Usually for a re-ranker, the chunks should hint at the entire content, but we’re using these chunks as seeds, so in some cases it will rate results very low, but it’s enough for us to go on.

This is why I’ve kept the score threshold very low.

There may be better options here that you might want to explore, maybe building a custom re-ranker that understands what you’re looking for.

Nevertheless, now that we have a few relevant documents, we’ll use its metadata that we set before on ingestion to expand and fan out the chunks so the LLM will get enough context to understand how to answer the question.

Build the context

Now that we have a few chunks as seeds, we’ll pull up more information from Redis, expand, and build the context.

This step is obviously a lot more complicated, as you need to build logic for which chunks to fetch and how (keys if they exist, or neighbors if there are any), fetch information in parallel, and then clean out the chunks further.

Once you have all the chunks (plus information on the documents themselves), you need to put them together, i.e. de-duping chunks, perhaps setting a limit on how far the system can expand, and highlighting which chunks were fetched and which were expanded.

The end result will look like something below:

Expanded context windows (Markdown ready):

## Document #1 — Fusing Knowledge and Language: A Comparative Study of Knowledge Graph-Based Question Answering with LLMs
- `doc_id`: `doc::6371023da29b4bbe8242ffc5caf4a8cd`
- **Last Updated:** 2025-11-04T17:44:07.300967+00:00
- **Context:** Comparative study on methodologies for integrating knowledge graphs in QA systems using LLMs.
- **Content fetched inside document:**
```text
[start on page 4]
    LLMs in QA
    The advent of LLMs has steered in a transformative era in NLP, particularly within the domain of QA. These models, pre-trained on massive corpora of diverse text, exhibit sophisticated capabilities in both natural language understanding and generation. Their proficiency in producing coherent, contextually relevant, and human-like responses to a broad spectrum of prompts makes them exceptionally well-suited for QA tasks, where delivering precise and informative answers is paramount. Recent advancements by models such as BERT [57] and ChatGPT [58], have significantly propelled the field forward. LLMs have demonstrated strong performance in open-domain QA scenarios-such as commonsense reasoning[20]-owing to their extensive embedded knowledge of the world. Moreover, their ability to comprehend and articulate responses to abstract or contextually nuanced queries and reasoning tasks [22] underscores their utility in addressing complex QA challenges that require deep semantic understanding. Despite their strengths, LLMs also pose challenges: they can exhibit contextual ambiguity or overconfidence in their outputs (’hallucinations’)[21], and their substantial computational and memory requirements complicate deployment in resource-constrained environments.
    RAG, fine tuning in QA
    ---------------------- this was the passage that we matched to the query -------------
    LLMs also face problems when it comes to domain specific QA or tasks where they are needed to recall factual information accurately instead of just probabilistically generating whatever comes next. Research has also explored different prompting techniques, like chain-of-thought prompting[24], and sampling based methods[23] to reduce hallucinations. Contemporary research increasingly explores strategies such as fine-tuning and retrieval augmentation to enhance LLM-based QA systems. Fine-tuning on domain-specific corpora (e.g., BioBERT for biomedical text [17], SciBERT for scientific text [18]) has been shown to sharpen model focus, reducing irrelevant or generic responses in specialized settings such as medical or legal QA. Retrieval-augmented architectures such as RAG [19] combine LLMs with external knowledge bases, to try to further mitigate issues of factual inaccuracy and enable real-time incorporation of new information. Building on RAG’s ability to bridge parametric and non-parametric knowledge, many modern QA pipelines introduce a lightweight re-ranking step [25] to sift through the retrieved contexts and promote passages that are most relevant to the query. However, RAG still faces several challenges. One key issue lies in the retrieval step itself-if the retriever fails to fetch relevant documents, the generator is left to hallucinate or provide incomplete answers. Moreover, integrating noisy or loosely relevant contexts can degrade response quality rather than enhance it, especially in high-stakes domains where precision is critical. RAG pipelines are also sensitive to the quality and domain alignment of the underlying knowledge base, and they often require extensive tuning to balance recall and precision effectively.
    --------------------------------------------------------------------------------------
[end on page 5]
```

## Document #2 — Each to Their Own: Exploring the Optimal Embedding in RAG
- `doc_id`: `doc::3b9c43d010984d4cb11233b5de905555`
- **Last Updated:** 2025-11-04T14:00:38.215399+00:00
- **Context:** Enhancing Large Language Models using Retrieval-Augmented Generation techniques.
- **Content fetched inside document:**
```text
[start on page 1]
    1 Introduction
    Large language models (LLMs) have recently accelerated the pace of transformation across multiple fields, including transportation (Lyu et al., 2025), arts (Zhao et al., 2025), and education (Gao et al., 2024), through various paradigms such as direct answer generation, training from scratch on different types of data, and fine-tuning on target domains. However, the hallucination problem (Henkel et al., 2024) associated with LLMs has confused people for a long time, stemming from multiple factors such as a lack of knowledge on the given prompt (Huang et al., 2025b) and a biased training process (Zhao, 2025).
    Serving as a highly efficient solution, RetrievalAugmented Generation (RAG) has been widely employed in constructing foundation models (Chen et al., 2024) and practical agents (Arslan et al., 2024). Compared to training methods like fine-tuning and prompt-tuning, its plug-and-play feature makes RAG an efficient, simple, and costeffective approach. The main paradigm of RAG involves first calculating the similarities between a question and chunks in an external knowledge corpus, followed by incorporating the top K relevant chunks into the prompt to guide the LLMs (Lewis et al., 2020).
    Despite the advantages of RAG, selecting the appropriate embedding models remains a crucial concern, as the quality of retrieved references directly influences the generation results of the LLM (Tu et al., 2025). Variations in training data and model architecture lead to different embedding models providing benefits across various domains. The differing similarity calculations across embedding models often leave researchers uncertain about how to choose the optimal one. Consequently, improving the accuracy of RAG from the perspective of embedding models continues to be an ongoing area of research.
    ---------------------- this was the passage that we matched to the query -------------
    To address this research gap, we propose two methods for improving RAG by combining the benefits of multiple embedding models. The first method is named Mixture-Embedding RAG, which sorts the retrieved materials from multiple embedding models based on normalized similarity and selects the top K materials as final references. The second method is named Confident RAG, where we first utilize vanilla RAG to generate answers multiple times, each time employing a different embedding model and recording the associated confidence metrics, and then select the answer with the highest confidence level as the final response. By validating our approach using multiple LLMs and embedding models, we illustrate the superior performance and generalization of Confident RAG, even though MixtureEmbedding RAG may lose to vanilla RAG. The main contributions of this paper can be summarized as follows:
    We first point out that in RAG, different embedding models operate within their own prior domains. To leverage the strengths of various embedding models, we propose and test two novel RAG methods: MixtureEmbedding RAG and Confident RAG. These methods effectively utilize the retrieved results from different embedding models to their fullest extent.
    --------------------------------------------------------------------------------------
    While Mixture-Embedding RAG performs similarly to vanilla RAG, the Confident RAG method exhibits superior performance compared to both the vanilla LLM and vanilla RAG, with average improvements of 9.9% and 4.9%, respectively, when using the best confidence metric. Additionally, we discuss the optimal number of embedding models for the Confident RAG method based on the results.
[...]

The total context will contain a few documents and lands around 2–3k tokens. There is some waste here, but instead of deciding for the LLM, we send in more information so it can scan entire documents rather than isolated chunks.

Remember you can take a look at the pipeline for five different queries here to see the results.

For the system you build, you can use this context as state so the LLM can answer follow-up questions from the same information.

Add in the LLM to answer (the G in RAG)

Let’s finish up with the last step, just to see how the LLM answers with the context it has been handed.

I did not plan on adding this part, as this article is primarily about the retrieval pipeline, but it’s good to test with an LLM to see if it can interpret the information into something coherent.

If you use a larger LLM here, you can expect the latency to go up quite a bit.

I decided to test with GPT-5 (low reasoning), as it’s very cheap to use but also quite slow. You’ll see the result below with the messy user query: “why is the llm lying and can rag help with this?”

[optimize query... retrieve... fuse... re-rank... expand... build context...]

LLM Answers:

LLMs sometimes “lie” because they generate the next likely words based on patterns, not on verified facts. Without grounding in sources, they can produce confident but false statements—known as hallucinations—and their built‑in knowledge can be outdated or incomplete [6][3]. This is risky in domains that demand strict accuracy (e.g., law or telecom regulations) where small errors have big consequences [3].
RAG helps by having the model first retrieve relevant documents from an external corpus and then generate answers grounded in that evidence. This reduces hallucinations and brings in up‑to‑date information [2][6][1]. In practice, domain‑tailored RAG has raised accuracy substantially; for example, a radio‑regulations RAG pipeline boosted GPT‑4o accuracy by +11.9% over vanilla prompting and achieved up to 97% retrieval accuracy, supporting more reliable answers [3].
However, RAG is only as good as its retrieval and how well the model uses the retrieved text. Static, off‑the‑shelf retrievers can be brittle for complex tasks (e.g., multihop questions or procedural queries) that need iterative lookups or query reformulation [2]. Naive RAG can also drown the model in long, noisy contexts or pull semantically similar but irrelevant chunks—the “needle‑in‑a‑haystack” problem—leading to wrong answers [4].
Better RAG practices address this: instruction‑tuning for RAG that teaches filtering, combining multiple sources, and RAG‑specific reasoning with a “think before answering” approach (HIRAG) improves use of evidence [1]; adaptive, feedback‑driven retrieval decides when and how to retrieve and re‑rank evidence [2]; and pipeline designs that optimize chunking and retrieval raise answer accuracy [4].
If hallucination still persists, techniques that steer decoding directly (beyond RAG) can further suppress it [5].

cited documents:
  [1] doc::b0610cc6134b401db0ea68a77096e883 — HIRAG: Hierarchical-Thought Instruction-Tuning Retrieval-Augmented Generation
  [2] doc::53b521e646b84289b46e648c66dde56a — Test-time Corpus Feedback: From Retrieval to RAG
  [3] doc::9694bd0124d0453c81ecb32dd75ab489 — Retrieval-Augmented Generation for Reliable Interpretation of Radio Regulations
  [4] doc::6d7a7d88cfc04636b20931fdf22f1e61 — KeyKnowledgeRAG (K^2RAG): An Enhanced RAG method for improved LLM question-answering capabilities
  [5] doc::3c9a1937ecbc454b8faff4f66bdf427f — DSCC-HS: A Dynamic Self-Reinforcing Framework for Hallucination Suppression in Large Language Models
  [6] doc::688cfbc0abdc4520a73e219ac26aff41 — A Systematic Review of Key Retrieval-Augmented Generation (RAG) Systems: Progress, Gaps, and Future Directions

You’ll see that it cites sources correctly and uses the information it has been handed, but as we’re using GPT-5, the latency is quite high with this large context.

It takes about 9 seconds to first token with GPT-5 (but it will depend on your environment) if you use GPT-5-mini this is reduced to about 4–5 seconds.

If the entire retrieval pipeline takes about 3–5 seconds, this means the last part will take about 2–3 times longer.

Some people will argue that you need to send in less information in the context window to decrease latency for this part but that also defeats the purpose of what we’re trying to do.

Others will argue for using chain prompting, having one smaller LLM extract useful information and then letting another bigger LLM answer with an optimized context window but I’m not sure how much you save in terms of time.

Others will go as small as possible, sacrificing “intelligence” for speed and cost. In cases where we serve the LLM the information it needs up front, like we do here, it may do well to use smaller.

Nevertheless, it’s up to you how you optimize the system. That is the hard part.

If you want to examine the entire pipeline for a few queries remember to see this folder.

Let’s talk latency & cost

People talking about sending in entire docs into an LLM are probably not ruthlessly optimizing for latency in their systems. This is the part you’ll spend the most time with, users don’t want to wait.

Yes you can apply some UX tricks while the user is waiting, such as “searching documents”, “finding a few relevant passages” but you still need to work on speed.

This is also why it’s interesting that we see this shift into agentic search in the wild, it’s so much slower to add large context windows, LLM-based query transforms, auto “router” chains, sub-question decomposition and multi-step “agentic” query engines.

For this system here (mostly built with Codex and my instructions) we land at around 4 seconds for retrieval in a Serverless environment.

This is kind of slow (but pretty cheap).

You can optimize each step here to bring that number down, keeping most things warm. However, using the APIs you can’t always control how fast they return a response.

Some people will argue to host your own smaller models for the optimizer and routers, but then you need to add in costs to host which can easily add a few hundred dollars per month.

Adding the final LLM at the end will be the worst culprit (as you saw in the previous section).

But with this pipeline here, each run (without caching) costs us 1.2 cents ($0.0121) so if you had your org ask 200 questions every day you would pay around $2.42 with GPT-5.

If you use GPT-5-mini for the main LLM, one pipeline run would drop to 0.0041 cents, and amount to about $0.82 per day for 200 runs.

You can cache repeat questions to lower this too.

As for embedding the documents, I paid around $0.5 for 200 PDF files using OpenAI’s large model. This will increase as you scale which is something to consider, then it can make sense with small or specialized model (or setting up a two-part system to use keyword search first to narrow candidates).

Notes on this system

As we’re only working with recent RAG papers, once you scale it, you can add some stuff to make it more robust.

I should first note though that you may not see most of the real issues until your docs start growing. Whatever feels solid with a few hundred docs will start to feel messy once you ingest tens of thousands.

You can have the optimizer set filters, perhaps using semantic matching for topics. You can also have it set the dates to keep the information fresh while introducing an authority signal in re-ranking that boosts certain sources.

Some teams take this a bit further and design their own scoring functions to decide what should surface and how to prioritize documents, but this depends entirely on what your corpus looks like.

If you need to ingest several thousand docs, it might make sense to skip the LLM during ingestion and instead use it in the retrieval pipeline, where it analyzes documents only when a query asks for it. You can then cache that result for next time.

If you’re still with me this far, a question you can ask yourself is whether it’s worth it to build a system like this and how much work it takes to do it well?

❤

Your first agent might ship with a 500-token system prompt and two tools, but those numbers usually balloon fast.

Just to illustrate, the leaked Claude system prompt ran around 24,000 tokens, and OpenClaw users have reported more than 150,000 input tokens sent to Gemini 3.1 Pro for 29 tokens of output on the first turn.

An unoptimized agent that runs at 100 messages a day at 166K input tokens can cost around $996 a month on Gemini 3.1 Pro and roughly $2,490 on Claude Opus 4.6.

There are tricks to keep these costs down, closer to $50 and $100 a month.

So I wanted to go through a few design principles people usually consider when building.

We’ll go through how prompt caching works and why it’s a quick win, semantic caching, lazy-loading tools and MCPs, routing and cascading, delegating to subagents, and a bit on what it saves to keep the context clean.

I am including interactive graphs throughout this article that help you visualize the cost savings each principle can get you based on the amount of tokens you are using.

Yes, I am obviously staying real throughout, every saving comes with trade-offs.

Four design principles to keep in mind

In this article we’ll go through four different parts with four different interactive calculators.

First, we’ll be looking at how to reuse tokens when possible, looking at prompt caching and semantic caching. Then we’ll look at how to minimize the stable and always added tokens like memory and tool definitions.

It will also go through how to route to smaller models, or escalate to a larger model, looking at the quality risks and the savings thereof.

The last section will talk about keeping the context clean for performance and economic reasons, while briefly mentioning compaction.

Reuse tokens when possible

LLM cost doesn’t just come from calling the model too often. It also comes from repeatedly paying to process the same tokens again and again.

So for this section we’ll cover K/V caching, the under-the-hood mechanism behind prompt caching, and semantic caching, which are two very different things. We’ll go through what they are, what they do, what you can save.

Prompt caching is a quick win for long system prompts, while semantic caching is a bit more work and comes with a bit more risk.

K/V caching & prefix caching

Before a model can generate anything, it first has to process the prompt. This part costs compute, which means latency and money. So, to be efficient, we shouldn’t keep re-processing the same content.

When you use a large language model, the prompt first gets tokenized, then those tokens turn into vectors, and then inside each attention layer those vectors get projected into K/V tensors.

Caching means holding on to these do we don’t recompute them next time | Image by author

The inference engine caches the K/V tensors during generation, otherwise the math doesn’t work at any reasonable speed as I understand it.

But instead of throwing the cache away when the response ends, it’s possible to store it.

Next time a request comes in, we’d check whether that same part of the prompt matches something we already have tensors for. If yes, we load those tensors and skip re-processing it.

To get a sense of why this matters economically: let’s say it takes one second to process 2,000 tokens, and you have a system prompt of 10,000 tokens.

That’s 5 seconds saved on every single LLM call, just by not recomputing that same start of the prompt through the model over and over again (though prefill throughput varies a lot based on the setup).

It’s important to note that we have to match the input exactly to the stored K/V cache.

If the tokens change, we don’t have precomputed K/V tensors for that exact part of the prompt anymore, so it has to be processed again. This is where people keep stumbling: a new space added, a reordered tool definition, a timestamp in the wrong place.

So, storing the cache has real value in terms of speeding up the request, and in turn making the request cheaper.

Note that storing these tensors is not free. Cached K/V takes up memory on the serving side which is why a lot of providers have a TTL window of around 5–10 minutes.

Now, we don’t have to build this ourselves, this was just to think through the mechanics. There are frameworks that help with this, and the API providers have their own prompt-caching rules, and we’ll go through both.

Prefix caching for self-hosted inference

If you are hosting an open source model, you’d ideally use an LLM serving framework, like vLLM. Though there are other frameworks that will help with the caching layer, vLLM has an add-on feature we can run through.

The caching layer in vLLM works by chopping up the prompt into blocks, hashing each block based on its tokens (plus the tokens before it), and storing the K/V tensors against those hashes.

Like most setups the static part that should be cached should go in the first part of the prompt.

To enable caching in vLLM use the flag --enable-prefix-caching

To adjust the block size you can use the flag --block-size. Block sizes means tokens per block. If block size is 16 then you have 16 tokens per block before it cuts off and starts another.

You can also use the flag --kv-cache-memory-bytes which explicitly sets KV cache size per GPU. The more memory you give it, the longer it can hold on to cached blocks. But if you have lots of different long requests happening at the same time, that memory fills up faster, so old blocks get removed faster.

There are other solutions out there, but you get the idea. It’s the same mechanics we spoke about for the previous section.

You can also check out SGLang and RadixAttention for prefix caching, as well as LMCache that should plug into serving engines.

Most people though use the API providers and they have their own policies on how to use prompt caching so let’s walk through those.

Prompt caching via API providers

Using the API providers you need to make sure to structure your prompts so they hit the cache. There are things you need to follow for this to be done correctly.

I will use OpenAI first here as an example.

For OpenAI, they are explicit, to cache part of the prompt they require an exact prefix match. I.e. the same static input at the start of the prompt.

This means you always put stable instructions, examples, and tools first, and variable content later.

You can also send in prompt-cache-key which can help route similar requests together and improve cache hit rates.

There are more specifics around this too. Caching is enabled automatically for prompts that are 1,024 tokens or longer, but they use the first 256 tokens to route requests back to the same cache. So, that static part of the prompt needs to be more than 256 tokens.

For Anthropic you have to enable caching with the cache-control parameter.

Worth mentioning too that typically the evictions (TTL) occur around 5–10 minutes of inactivity but can be extended. It’s the same for Anthropic but you can push it to one hour (but it would cost you more at 2x).

Earlier I talked about the time you save, and if you’re self-hosting, this also saves money. With API providers, the savings show up as cheaper cached input tokens.

With OpenAI cached input is up to 90% off the base input.

Anthropic gives you the same discount on cached inputs, but you also pay to store that cache. So, if you’re not using it correctly, Anthropic will be more expensive.

In general though, if you have 90% of your prompt being static, you can look below at your possible pricing if you use the cache right.

I decided to create an interactive graph for this with Claude here, so you can play around with it.

So, prompt caching is a pretty good win for everyone if you’re using longer system prompts that stay the same and something to consider to save on tokens.

Let’s move onto semantic caching, which is something else entirely.

Semantic caching

Semantic caching matches on meaning, i.e. if it is a similar enough request, return the cached result. Although it sounds easy enough, there are clear pitfalls to watch out for.

To semantically match texts, we use embeddings. You can do some research here if the word is new to you. I wrote about it a few years ago.

In essence, embeddings are vectors that we can compare against each other using cosine similarity. If similarity is high, the meaning should be similar, though it depends on the model.

What semantic caching is proposing is then to match similar requests to answers that already exist. Asking for “What’s the capital of France?” and “Quick, give me the capital of France” should then route to the same answer.

No need to use an LLM to answer the same thing over and over again.

This works fine if many people ask near-identical generic questions and the data isn’t going stale too fast.

So why not do it for every case? There are a ton of pitfalls here.

Just at the top of my head, you need to consider what threshold to use for similarity, how long the answer should stay valid, and what happens on multi-turn questions.

Then you need to think about what actually gets stored, whether there should be a router implemented too, how to separate users, and what happens if the wrong answer is cached.

You also need to consider (Time to Live — TTL), as in when information turns stale and for which questions.

So even though the mechanics are quite simple, you still need metadata filters and tags, such as user, workspace, corpus version, persona, session/user scoping, smart TTL, and some rule for “is the return enough?”

This then turns into a bit of a project.

So, if you want to do it, perhaps use the semantic index to find a previous question. Different questions can point to the same stored answer, which means less storage blowup. Be smart about TTL by usage, if something is reused often, retain it longer, otherwise, remove it.

I would also suggest you do it after you see repetition in the logs rather than at the start. It may be that the use case is just not good for it.

As for how to do it, many databases can do this for you but there are also libraries like semanticcache, prompt-cache, GPTCache, vCache, Upstash semantic-cache, Redis + LangCache, that help with plumbing.

There are obviously savings here to be made though. Redis claims up to 68.8% fewer API calls and 40–50% latency improvement, though be aware this is a bit of marketing as they are using a clear Q&A use cases here.

So it completely depends on your setup. If you have a Q/A bot with a lot of redundant calls, then you can save more. If you have a coding bot with unique calls, then you’ll save less.

Prompt caching does well when the changing question sits inside a large static prompt. Semantic caching does well when people keep asking the same thing in different words.

You can play around with this interactive tool to see the cost savings with both.

If you’re looking at 15% semantic match, the additional savings aren’t as steep, but if that number increased above 45–50% then it could make more sense.

Before we move on from this section, I would just point out that there are a lot of savings to be made from standard caching too.

Remember to cache the expensive deterministic stuff like SQL query results, tool outputs, and retrieval results. Never run this stuff more times than you need to.

I do this for one of my tools. It gathers keyword data to summarize, then caches it until that data is stale. If it is stale, it reruns it when that route is hit.

So, semantic caching is an interesting concept and could save you tokens for certain use cases but it takes engineering to do it well.

Don’t preload dormant tokens

This part is about what happens when your system prompt starts growing because of things like bulky tools or growing memory.

For smaller agents, this isn’t really an issue, but if you are working with agent specs that keep growing, there are ways to slim it down and fetch information on-demand (or at least try to).

Keep context slim and fetch details on-demand

Once your agent prompt grows beyond a certain point, it can be good to keep the always-loaded layer as small and stable as possible, and keep growing details separate.

This matters because once these layers start to grow, such as when you load a few hundred tools or send full MCP server descriptions that keep changing, it gets noisy.

The problem is obviously not just cost, but also performance. And if one of these layers keeps changing, prompt caching becomes much harder to hit properly.

So, the idea is to keep the top layer as compact and stable as you can. The top layer should help the model understand where it is and where to go next, but it does not need to carry the whole world up front.

If you’ve looked through the source code for Claude Code, you’ve seen that they use something like this for their memory system.

They have an always-loaded index file that shouldn’t grow beyond 200 lines, with detailed topic files elsewhere. Though what the agent does in practice versus what the system wants is a topic for another time.

You can see the same idea pop up elsewhere too, such as in Claude’s advanced tool setup, Claude Skills’ layered setup, and attempts to lazy-load MCP tools instead of dumping every server definition into the prompt up front.

Where this is done and if it works

The idea is sound. When context grows, it gets harder for an LLM to pick the right action. But this space is still early, so we’ll go through a tool as an example to see how this can work.

A few months ago, Anthropic released something called advanced Tool Search. This goes into the space of how to keep context slim while still giving the model access to hundreds of tools.

Anthropic says they have seen 55K to 134K tokens of tool definitions before optimization, and that wrong tool selection is a common failure mode when the context grows this large.

So, a search tool would then optimize the context by having the LLM use it to find tools, rather than define them all up front.

tools=[
        {
            “type”: “tool_search_tool_bm25_20251119”, 
            “name”: “tool_search”
        },
        {
            “name”: “search_contacts”,
            “description”: “Find a contact by name or email.”,
            “input_schema”: {
                “type”: “object”,
                “properties”: {
                    “query”: {”type”: “string”}
                },
                “required”: [”query”]
            }
        },
        {
            “name”: “send_email”,
            “description”: “Send an email to one or more recipients.”,
            “input_schema”: {
                “type”: “object”,
                “properties”: {
                    “to”: {”type”: “string”},
                    “subject”: {”type”: “string”},
                    “body”: {”type”: “string”}
                },
                “required”: [”to”, “subject”, “body”]
            },
            “defer_loading”: True
        }
    ]

What you see above is that we define one tool called tool_search. You can pick one of the out-of-the-box options, BM25 or Regex, or build your own custom one. Then we set one tool as deferred as an example. You would only do this if you had 10+ tools though.

Anthropic does the searching for you, so you don’t see how it adds this tool schema into the system prompt, nor do you see how the search happens under the hood.

They do say that once there is a tool match, its definition is appended inline as a tool_reference block in the conversation for the LLM.

The idea is neat: smaller initial context, but you’re still adding one extra search step. People have also tested this tool with somewhat lackluster results, but that was with 4,000 tools, so there is more room for testing.

It’s also on us to define the tools well enough that they can be searched. But it becomes harder to debug what is happening when you can’t see the intermediate step.

This idea pops up elsewhere too, but people generally just call it good AI engineering. Do not expose the agent to huge messy context. Instead, give it a way to narrow things down, and only then let it inspect or load the tool when needed.

For this part, there are serious savings to be made as well, though it depends on how many tokens you are sending in the first place. We created this additional calculator that compares tool search and prompt caching.

Vibe calculated example of savings for prompt caching + lazy-loading tools

What we see is that both prompt caching and lazy-loading context give you savings, but together it’s not a huge change. Tool search like this isn’t just about savings though as it helps keep the context clean performance wise.

But if you’re just looking for savings, the biggest win is to pick at least one.

Use cheap models for cheap work

This section is about routing prompts to different models, along with using subagents with cheaper models for certain tasks, and how this can decrease token costs but also risk quality.

This space is interesting because most people argue that 60% or more of incoming questions are easy tasks, and thus do not need the strongest model, especially not a thinking one.

ChatGPT does this using signals like conversation type, complexity, tool needs, and explicit intent (“think hard”). Claude uses description-based delegation and built-in subagents like Explore.

The idea is simple to understand, but being able to do it right without risking too much quality is the hard part.

So, let’s go through both predictive routing and output-checked approaches like cascades and subagents, so you can get a feel for what you can test on your own.

The savings that can be made here are very real. I created another interactive graph for this part that you can find here.

Route to models based on task difficulty

Request-level routing means trying to estimate difficulty and intent before seeing any output. The upside is high, but a bad choice can poison the whole session, so there are some quality drawbacks to keep in mind.

To do this, you need some kind of router model that decides where to route the request.

We don’t know exactly what OpenAI uses as signals to route to different models, but I don’t know about you, I frequently feel like I’m being delegated to a less competent model at times and it can be infuriating.

There are ways for us to still gather intel though, looking at the open source community. We can look at RouteLLM from LMSYS, the Berkeley group behind Chatbot Arena. This solution learns from real preference data from Chatbot Arena.

RouteLLM uses standard embeddings and then a tiny router head, so hosting this shouldn’t be that expensive.

I have not tested this solution myself, but they report large cost reductions while keeping most of GPT-4’s performance.

I did, though, dig into the LLMRouterBench paper, which pretty much said that many learned routers barely beat simple baselines, such as keyword/heuristic routing, embedding nearest-neighbor, or kNN-style routing.

This means that the fancy routers around may not give you that much of a boost compared to just using something simple.

People haven’t abandoned routing because of this, but people are still tinkering with it, so it’s not a slam dunk in terms of savings if the quality of the answers can’t keep up.

Now, there are also out-of-the-box solutions in this space, such as OpenRouter Auto and Switchpoint. However, there is nothing public on their routing internals or public accuracy numbers in the way I wanted.

But for this section, also check out the savings calculator we did for LLMRouter, heuristics, self-hosted classifier, LLM-as-router, RouteLLM, OpenRouter Auto.

As for quality and how this works for real-world projects, I can’t say before doing better testing on my own first with clear numbers to show, so this space really deserves its own article in the future.

We should also briefly cover cascading and then subagents before moving on.

Start with cheap and cascade on low confidence

Instead of guessing from the prompt whether a request is “easy” or “hard,” we can also let the cheap model try first, then decide whether to keep that answer or escalate.

Google’s “Speculative Cascades” write-up frames this tradeoff: use smaller models first for cost and speed, and defer to larger models only when needed.

To do this, you have the cheap model generate first, then use a lightweight checker that looks at things like logprobs/token probabilities, entropy or margin-style uncertainty, and/or semantic alignment.

This idea is pretty attractive, as prompt difficulty is often hard to predict and most routers don’t do perfectly. Furthermore, quality is easier to judge after you have an answer.

It also only makes sense if you think most questions can be answered by a simpler model, as you need to pay for two calls for the ones being escalated.

But from the people implementing this, I’ve heard it’s an attractive choice as validation latency between calls can stay under 20ms.

I did look into some open-source implementations like CascadeFlow, which claims 69% savings and 96% quality retention vs GPT-5. But it’s good to note that the prompts they tested had verifiable ground truth, such as math answers and multiple choice.

A main issue to consider is that small models are often “confidently wrong,” so it might make sense to start with conservative thresholds and escalate more often. That will inevitably bring costs up.

I also added in Cascade (cheap-first) into the interactive graph, so you can compare the savings with the other choices. If true, it may slash costs by 50% using this technique, if you need larger models for certain requests at all, that is.

Delegate work to subagents

Subagents are about delegating work to isolated agents. Sometimes these use smaller models, so we can say it’s a form of routing as well. The savings aren’t as steep here, but it’s worth mentioning.

Delegating to subagents is not just about cost. It’s also about keeping the context clean so each agent can fully focus on the task it should complete.

Anthropic ships Claude Code with built-in subagents, as many have seen. The Explore subagent is explicitly a Haiku worker for codebase search and exploration. So, the design principle is there: use smaller models for cheaper tasks.

The main Claude session also delegates via description matching, but we don’t see it. We just get cheaper aggregate cost.

But because the orchestrator often still stays in the loop for planning, synthesis, and retries, you don’t save as much as we saw with routing.

You can look at the graphs we created above and see that subagents may shave off around 11% from the “no routing” option by our calculations, so it is not the main thing to go for if you’re just looking to cut costs.

My next article will dig into subagents, but more as a way to delegate work and isolate tasks when working with deepagents.

Let’s go through the last section before rounding off.

Keep your context clean

Good context engineering is usually about performance, but it can also be about cost efficiency. So, let’s go through context compaction and talk about how keeping the context clean can save tokens.

The issue is that agents keep accumulating junk: tool outputs, logs, repeated observations, old plans, stale attempts, and duplicated state.

This is especially true for people building agents for the first time, where they keep dumping results into the working state for the main agent.

I’ve naturally done this myself too, especially with a draft agent, to see “how it does” first.

But I’ve also seen people complain about OpenClaw context build-up, so it happens everywhere. People complain about it in Claude Code too, because in general, it’s easier to just have it add stuff to it than to work on cleaning it up.

Let’s briefly talk about this without going too much into the performance side, which is also why you should do it.

The hard part is building a state pipeline

This is a two-tier problem. Not only are you “compressing the chat,” but you also need to keep things clean as you add them to the working state, and this becomes tedious engineering work.

First, to keep the context clean, we don’t want this kind of result to start eating up the context.

bad state:
agent does work
→ dumps tool output into context
→ reads files
→ dumps files into context
→ runs tests
→ dumps logs into context
→ retries
→ keeps everything

So, the real job is first to preserve the right state while deleting exhaust as you go along.

Raw output like this can go into an archive, and only what is needed goes into active context. In general, the enemy is probably tool-output bloat here, so the work is to make tools less noisy by default.

Good active context

[system rules]
[project rules]
[user task]
[current working state]

Keep:
+ auth flow lives in auth.ts + session.ts
+ bug only happens on refresh path
+ failing test: session_refresh_keeps_user
+ likely overwrite during refresh
+ files in scope: auth.ts, session.ts, auth.test.ts

Drop:
- raw grep results
- full test logs
- duplicate file dumps
- dead-end retries

I’m also thinking certain pieces of context can have a lifecycle or a set expiry.

Then, once you reach the point where you need to compress it, it will be easier to know what is useful for the LLM.

If we do a little Anthropic reading on long-horizon tasks, they note that you need to figure out a way to preserve architectural decisions, unresolved bugs, and implementation details for compression once it gets to that point as well.

For LangChain’s autonomous compression, they have the agent decide when to compact, instead of only doing it after the context is already bloated, as I think is the case with Anthropic.

It’s interesting that teams are starting to evaluate compression as a systems problem too, with benchmarks and agent-specific policies, not as a generic summarization trick.

We can look at a recent paper here too to get an idea of the results we can get. This one by Jia et al. argued that at 6x compression, it gave a 51.8–71.3% token-budget reduction, while achieving a 5.0–9.2% improvement in issue resolution rates on SWE-bench Verified.

So, this is not just about cost, but also about performance in general.

As for costs, there is obviously a lot of work in building good context itself, but removing junk can probably clear up 30–70% of your context, which then saves just as much in dollars.

To illustrate, for a 10k context window, if you clean up 30% to 50% at 100k runs, you might save up to $1,500. At a 40k context window, that number goes up to $6,000.

We did a calculation for this here as well so you can visualize it. It’s good to note that compressing agents that are using very small cheap models may turn out to be more expensive.

Nevertheless, what’s good about trying to keep the context clean is that you’re not sacrificing quality, as can happen with semantic caching or routing, so if done well, it’s a clear gain.

The issue is obviously the work to do so.

Rounding up the conversation

This is a very long article that serves up four different ways you can cut token costs when building agents.

It very much depends on your use case, use prompt caching when dealing with large system prompts that stay unchanged as you loop LLM calls, use semantic caching if you are dealing with a generic Q/A bot that needs to stay cheap.

Test routing if you need to be able to answer both easy and hard questions, and if you want to make sure you don’t send unnecessary tokens keep the context as clean as possible.

It may be worth it in the future to make a shorter, more economics focused article to focus on certain setups.

Nevertheless I hope it was informational, follow me here, at LinkedIn, or via my website, if just want to read more of the same stuff.

❤

I would have figured that most companies would have built or implemented their own RAG agents by now.

An AI knowledge agent can dig through internal documentation — websites, PDFs, random docs — and answer employees in Slack (or Teams/Discord) within a few seconds. So, these bots should significantly reduce time sifting through information for employees.

I’ve seen a few of these in bigger tech companies, like AskHR from IBM, but they aren’t all that mainstream yet.

If you’re keen to understand how they are built and how much resources it takes to build a simple one, this is an article for you.

I’ll go through the tools, techniques, and architecture involved, while also looking at the economics of building something like this. I’ll also include a section on what you’ll end up focusing the most on.

There is also a demo at the end for what this will look like in Slack.

If you’re already familiar with RAG, feel free to skip the next section — it’s just a bit of repetitive stuff around agents and RAG.

What is RAG and Agentic RAG?

Retrieval-Augmented Generation (RAG) is a way to fetch information that gets fed into the large language model (LLM) before it answers the user’s question.

This allows us to provide relevant information from various documents to the bot in real time so it can answer the user correctly.

This retrieval system is doing more than simple keyword search, as it finds similar matches rather than just exact ones. For example, if someone asks about fonts, a similarity search might return documents on typography.

Many would say that RAG is a fairly simple concept to understand, but how you store information, how you fetch it, and what kind of embedding models you use still matter a lot.

If you’re keen to learn more about embeddings and retrieval, I’ve written about this here.

Today, people have gone further and primarily work with agent systems.

In agent systems, the LLM can decide where and how it should fetch information, rather than just having content dumped into its context before generating a response.

It’s important to remember that just because more advanced tools exist doesn’t mean you should always use them. You want to keep the system intuitive and also keep API calls to a minimum.

With agent systems the API calls will increase, as it needs to at least call one tool and then make another call to generate a response.

That said, I really like the user experience of the bot “going somewhere” — to a tool — to look something up. Seeing that flow in Slack helps the user understand what’s happening.

But going with an agent or using a full framework isn’t necessarily the better choice. I’ll elaborate on this as we continue.

Technical Stack

There is a ton of options for agent frameworks, vector databases, and deployment options, so I’ll go through some.

For the deployment option, since we’re working with Slack webhooks, we’re dealing with event-driven architecture where the code only runs when there’s a question in Slack.

To keep costs to a minimum, we can use serverless functions. The choice is either going with AWS Lambda or picking a new vendor.

Platforms like Modal are technically built to serve LLM models, but they work well for long-running ETL processes, and for LLM apps in general.

Modal hasn’t been battle-tested as much, and you’ll notice that in terms of latency, but it’s very smooth and offers super cheap CPU pricing.

I should note though that when setting this up with Modal on the free tier, I’ve had a few 500 errors, but that might be expected.

As for how to pick the agent framework, this is completely optional. I did a comparison piece a few weeks ago on open-source agentic frameworks that you can find here, and the one I left out was LlamaIndex.

So I decided to give it a try here.

The last thing you need to pick is a vector database, or a database that supports vector search. This is where we store the embeddings and other metadata, so we can perform similarity search when a user’s query comes in.

There are a lot of options out there, but I think the ones with the highest potential are Weaviate, Milvus, pgvector, Redis, and Qdrant.

Both Qdrant and Milvus have pretty generous free tiers for their cloud options. Qdrant, I know, allows us to store both dense and sparse vectors. Llamaindex, along with most agent frameworks, support many different vector databases so any can work.

I’ll try Milvus more in the future to compare performance and latency, but for now, Qdrant works well.

Redis is a solid pick too, or really any vector extension of your existing database.

Cost & time to build

In terms of time and cost, you have to account for engineering hours, cloud, embedding, and large language model (LLM) costs.

It doesn’t take that much time to boot up a framework to run something minimal. What takes time is connecting the content properly, prompting the system, parsing the outputs, and making sure it runs fast enough.

But if we turn to overhead costs, cloud costs to run the agent system is minimal for just one bot for one company using serverless functions as you saw in the table in the last section.

However, for the vector databases, it will get more expensive the more data you store.

Both Zilliz and Qdrant Cloud has a good amount of free tier for your first 1 to 5GBs of data, so unless you go beyond a few thousand chunks you may not pay for anything.

You will start paying though once you go beyond the thousands mark, with Weaviate being the most expensive of the vendors above.

As for the embeddings, these are generally very cheap.

You can see a table below on using OpenAI’s text-embedding-3-small with chunks of different sizes once you embed 1 to 10 million texts.

When people start optimizing for embeddings and storage, they’ve usually moved beyond embedding millions of texts.

The one thing that matters the most though is what large language model (LLM) you use. You need to think about API prices, since an agent system will typically call an LLM two to four times per run.

For this system, I’m using GPT-4o-mini or Gemini Flash 2.0, which are the cheapest options.

So let’s say a company is using the bot a few hundred times per day and each run costs us 2–4 API calls, we might end up at around less of a dollar per day and around $10–50 dollars per month.

You can see that switching to a more expensive model would increase the monthly bill by 10x to 100x. Using ChatGPT is mostly subsidized for free users, but when you build your own applications you’ll be financing it.

There will be smarter and cheaper models in the future, so whatever you build now will likely improve over time. But start small, because costs add up and for simple systems like this you don’t need them to be exceptional.

The next section will get into how to build this system.

The architecture (processing documents)

The system has two parts. The first is how we split up documents — what we call chunking — and embed them. This first part is very important, as it will dictate how the agent answers later.

So, to make sure you’re preparing all the sources properly, you need to think carefully about how to chunk them.

If you look at the document above, you can see that we can miss context if we split the document based on headings but also on the number of characters where the paragraphs attached to the first heading is split up for being too long.

You need to be smart about ensuring each chunk has enough context (but not too much). You also need to make sure the chunk is attached to metadata so it’s easy to trace back to where it was found.

This is where you’ll spend the most time, and honestly, I think there should be better tools out there to do this intelligently.

I ended up using Docling for PDFs, building it out to attach elements based on headings and paragraph sizes. For web pages, I built a crawler that looked over page elements to decide whether to chunk based on anchor tags, headings, or general content.

Remember, if the bot is supposed to cite sources, each chunk needs to be attached to URLs, anchor tags, page numbers, block IDs, permalinks so the system can locate the information correctly being used.

There is also the option to keep the chunks small, but fetching surrounding chunks for context expansion after retrieval.

You then make sure that the retrieval stage has a greater chance of finding the correct information while making sure the LLM has enough information to provide a coherent answer.

Since most of the content you’re working with is scattered and often low quality, I also decided to just push in summarizations using an LLM.

These summaries were given different labels with higher authority, which meant they were prioritized during retrieval. I suppose you can use them to gatekeep other chunks, working with it as a filter.

Here another option is to add semantic topics to the chunks that you later use on retrieval.

There is also the option to push in the summaries in their own tools, and keep deep dive information separate. Letting the agent decide which one to use but it will look strange to users as it’s not intuitive behavior.

Still, I have to stress that if the quality of the source information is poor, it’s hard to make the system work well.

For example, if a user asks how an API request should be made and there are four different web pages giving different answers, the bot won’t know which one is most relevant. You can naturally timestamp the chunks but if the docs are ingested at the same time then it will be harder.

To show companies how it worked, I had to do some manual review. I also had AI do deeper research around the company to help fill in gaps, and then I embedded that too.

In the future, I think I’ll build something better for document ingestion.

The architecture (the agent)

For the second part, where we connect to this data, we need to build a system where an agent can connect to different tools that contain different amounts of data from our vector database.

We keep to one agent only to make it easy enough to control. This one agent can decide what information it needs based on the user’s question

It’s good not to complicate things and build it out to use too many agents, or you’ll run into issues, especially with these smaller models.

Although this may go against my own recommendations, I did set up a first LLM function that decides if we need to run the agent at all.

This was primarily for the user experience, as it takes a few extra seconds to boot up the agent (even when starting it as a background task when the container starts).

As for how to build the agent itself, this is easy, as LlamaIndex does most of the work for us. For this, you can use the FunctionAgent, passing in different tools when setting it up.

# Only runs if the first LLM thinks it is necessary

access_links_tool = get_access_links_tool()
public_docs_tool = get_public_docs_tool()
onboarding_tool = get_onboarding_information_tool()
general_info_tool = get_general_info_tool()
    
formatted_system_prompt = get_system_prompt(team_name)
    
agent = FunctionAgent(
  tools=[onboarding_tool, public_docs_tool, access_links_tool, general_info_tool],
  llm=global_llm,
  system_prompt=formatted_system_prompt
)

The tools have access to different data from the vector database, and they are wrappers around the CitationQueryEngine. This engine helps to cite the source nodes in the text. We can access the source nodes at the end of the agent run, which you can attach to the message and in the footer.

To make sure the user experience is good, you can tap into the event stream to send updates back to Slack.

handler = agent.run(user_msg=full_msg, ctx=ctx, memory=memory)

async for event in handler.stream_events():
  if isinstance(event, ToolCall):
     display_tool_name = format_tool_name(event.tool_name)
     message = f”✅ Checking {display_tool_name}”
     post_thinking(message)
  if isinstance(event, ToolCallResult):
     post_thinking(f”✅ Done checking...”)

final_output = await handler  
final_text = final_output
blocks = build_slack_blocks(final_text, mention)

post_to_slack(
  channel_id=channel_id, 
  blocks=blocks,
  timestamp=initial_message_ts,
  client=client 
)

Make sure to format the messages and Slack blocks well, and refine the system prompt for the agent so it formats the messages correctly based on the information that the tools will return.

The architecture should be easy enough to understand, but there are still some retrieval techniques we should dig into.

Techniques you can try

A lot of people will emphasize certain techniques when building RAG systems, and they’re partially right. You should use hybrid search along with some kind of re-ranking.

The first I will mention is hybrid search when we perform retrieval.

I mentioned that we use semantic similarity to fetch chunks of data in the various tools, but you also need to account for cases where exact keyword search is required.

Just imagine a user asking for a specific certificate name, like CAT-00568. In that case, the system needs to find exact matches just as much as fuzzy ones.

With hybrid search, supported by both Qdrant and LlamaIndex, we use both dense and sparse vectors.

# when setting up the vector store (both for embedding and fetching)
vector_store = QdrantVectorStore(
   client=client,
   aclient=async_client,
   collection_name=”knowledge_bases”,
   enable_hybrid=True,
   fastembed_sparse_model=”Qdrant/bm25”
 )

Sparse is perfect for exact keywords but blind to synonyms, whereas dense is great for “fuzzy” matches (“benefits policy” matches “employee perks”) but they can miss literal strings like CAT-00568.

Once the results are fetched, it’s useful to apply deduplication and re-ranking to filter out irrelevant chunks before sending them to the LLM for citation and synthesis.

reranker = LLMRerank(llm=OpenAI(model=”gpt-3.5-turbo”), top_n=5)
dedup = SimilarityPostprocessor(similarity_cutoff=0.9)

engine = CitationQueryEngine(
    retriever=retriever,
    node_postprocessors=[dedup, reranker],
    metadata_mode=MetadataMode.ALL,
)

This part wouldn’t be necessary if your data were exceptionally clean, which is why it shouldn’t be your main focus. It adds overhead and another API call.

It’s also not necessary to use a large model for re-ranking, but you’ll need to do some research on your own to figure out your options.

These techniques are easy to understand and quick to set up, so they aren’t where you’ll spend most of your time.

What you’ll actually spend time on

Most of the things you’ll spend time on aren’t so sexy. It’s prompting, reducing latency, and chunking documents correctly.

Before you start, you should look into different prompt templates from various frameworks to see how they prompt the models. You’ll spend quite a bit of time making sure the system prompt is well-crafted for the LLM you choose.

The second thing you’ll spend most of your time on is making it fast. I’ve looked into internal tools from tech companies building AI knowledge agents and found they usually respond in about 8 to 13 seconds.

So, you need something in that range.

Using a serverless provider can be a problem here because of cold starts. LLM providers also introduce their own latency, which is hard to control.

That said, you can look into spinning up resources before they’re used, switching to lower-latency models, skipping frameworks to reduce overhead, and generally decreasing the number of API calls per run.

The last thing, which takes a huge amount of work and which I’ve mentioned before, is chunking documents.

If you had exceptionally clean data with clear headers and separations, this part would be easy. But more often, you’ll be dealing with poorly structured HTML, PDFs, raw text files, Notion boards, and Confluence notes — often scattered and formatted inconsistently.

The challenge is figuring out how to programmatically ingest these documents so the system gets the full information needed to answer a question.

Just working with PDFs, for example, you’ll need to extract tables and images properly, separate sections by page numbers or layout elements, and trace each source back to the correct page.

You want enough context, but not chunks that are too large, or it will be harder to retrieve the right info later.

This kind of stuff isn’t well generalized. You can’t just push it in and expect the system to understand it — you have to think it through before you build it.

How to build it out further

At this point, it works well for what it’s supposed to do, but there are a few pieces I should cover (or people will think I’m simplifying too much). You’ll want to implement caching, a way to update the data, and long-term memory.

Caching isn’t essential, but you can at least cache the query’s embedding in larger systems to speed up retrieval, and store recent source results for follow-up questions. I don’t think LlamaIndex helps much here, but you should be able to intercept the QueryTool on your own.

You’ll also want a way to continuously update information in the vector databases. This is the biggest headache — it’s hard to know when something has changed, so you need some kind of change-detection method along with an ID for each chunk.

You could just use periodic re-embedding strategies where you update a chunk with different meta tags altogether (this is my preferred approach because I’m lazy).

The last thing I want to mention is long-term memory for the agent, so it can understand conversations you’ve had in the past. For that, I’ve implemented some state by fetching history from the Slack API. This lets the agent see around 3–6 previous messages when responding.

We don’t want to push in too much history, since the context window grows — which not only increases cost but also tends to confuse the agent.

That said, there are better ways to handle long-term memory using external tools. I’m keen to write more on that in the future.

Learnings and so on

After doing this now for a bit I have a few notes to share about working with frameworks and keeping it simple (that I personally don’t always follow).

You learn a lot from using a framework, especially how to prompt well and how to structure the code. But at some point, working around the framework adds overhead.

For instance, in this system, I’m bypassing the framework a bit by adding an initial API call that decides whether to move on to the agent and responds to the user quickly.

If I had built this without a framework, I think I could have handled that kind of logic better where the first model decides what tool to call right away.

I haven’t tried this but I’m assuming this would be cleaner.

Also, LlamaIndex optimizes the user query, which it should, before retrieval.

But sometimes it reduces the query too much, and I need to go in and fix it. The citation synthesizer doesn’t have access to the conversation history, so with that overly simplified query, it doesn’t always answer well.

With a framework, it’s also hard to trace where latency is coming from in the workflow since you can’t always see everything, even with observation tools.

Most developers recommend using frameworks for quick prototyping or bootstrapping, then rewriting the core logic with direct calls in production.

It’s not because the frameworks aren’t useful, but because at some point it’s better to write something you fully understand that only does what you need.

The general recommendation is to keep things as simple as possible and minimize LLM calls (which I am not even fully doing myself here).

But if all you need is RAG and not an agent, stick with that.

You can create a simple LLM call that sets the right parameters in the vector DB. From the user’s point of view, it’ll still look like the system is “looking into the database” and returning relevant info.

Once finished though, the result will look like this:

If you’re going down the same path, I hope this was useful.

There is bit more to it though. You’ll want to implement some kind of evaluation, guardrails, and monitoring (I’ve used Phoenix here).

We’ve all heard of CrewAI and AutoGen, but did you know there are dozens of open-source agentic frameworks out there — and many of them have been released in the last year.

I’ve briefly tested some of the more popular ones to get a feel for how they work and how easy they are to get started with. So follow along as I go through what each one brings to the table.

The focus will be on LangGraph, Agno, SmolAgents, Mastra, Pydantic AI, and Atomic Agents. We’ll also compare them to CrewAI and AutoGen.

We’ll look at what a framework actually does, the different design choices, how they differ from one another, and a bit about the schools of thought behind them.

Agentic AI

Agentic AI is basically about building systems around LLMs so they can have accurate knowledge, access to data, and the ability to act. You can think of it as using natural language to automate processes and tasks.

Using natural language processing in automation isn’t new — we’ve used NLP for years to extract and process data. What’s new is the amount of freedom we can now give language models, allowing them to handle ambiguity and make decisions dynamically.

But just because LLMs can understand language doesn’t mean they have agency — or even understand the task you’re trying to automate. That’s why there’s a lot of engineering involved in building reliable systems.

I go through Agentic AI in more detail here and here, if you’re keen to get a beginner friendly overview.

What does a framework do?

At their core, agentic frameworks help you with prompt engineering and routing data to and from the LLMs— but they also offer additional abstractions that make it easier to get started.

If you were to build a system from scratch where an LLM should use different APIs — tools — you’d define that in the system prompt. Then you’d request that the LLM returns its response along with the tool it wants to call, so the system can parse and execute the API call.

So basically, we’re talking about prompt engineering — which forms the foundation of any framework.

The framework usually helps in two ways: it structures the prompt properly to make sure the LLM responds in the right format, and then parses the response to route it to the correct tool — or API, document or what have you.

When we set up knowledge, a framework might help with chunking documents, embedding them and storing them. This gets added to the prompt as context, similar to how we build standard RAG systems.

A framework can also help with things like error handling, structured outputs, validation, observability, deployment — and generally help you organize your code so you can build more complex systems, like multi-agent setups.

Still, a lot of people feel that using a full framework is overkill.

The issue is: if the LLM doesn’t use the tool correctly or something breaks, the abstraction becomes a pain because you can’t debug it easily. This can also be a problem if you switch models — the system prompt might have been tailored for one and not transfer well to others.

That’s why some developers end up rewriting parts of a framework —such as create_react_agent in LangGraph — to get better control.

Some frameworks are lighter, some heavier and offer additional features, but there’s community around them to help you get started. And once you learn one (including how it works under the hood), it becomes easier to pick up others.

Different open source frameworks

We do look to the community to understand how well a framework performs in real cases. However, the most popular frameworks may not always be the ideal choice.

The ones we’ve all heard about are CrewAI and AutoGen.

CrewAI is a very high-abstraction framework that lets you build agent systems quickly by hiding low-level details. AutoGen focuses on autonomous, asynchronous agent collaboration, where agents have the freedom to collaborate as they see fit — which may make it more suited for testing and research.

LangGraph is still a fairly well-known system but deserves to be highlighted as one of the main frameworks for developers. It uses a graph-based approach where you build nodes and connect them via agents. Compared to the other two, it gives you stricter engineering control over workflows and doesn’t assume agents should have much agency.

It should be noted that many feel that LangGraph is overly complicated in its abstractions and difficult to debug. The idea is that it has a steep learning curve but once you learn the fundamentals it should get easier.

Now, there are a few newer frameworks I also want to include here.

The next one is Agno (previously Phi-Data) which focuses on providing a very good developer experience. It also has one of the cleanest documentations I’ve seen. It’s very plug-and-play, helping you get started quickly with a lot of built-in features, organized into logical, clean abstractions that make sense.

SmolAgents is a very bare-bones framework that introduces an agent — CodingAgent — which routes data via code rather than JSON. It also gives you direct access to the entire Hugging Face model library out of the box.

As for the open-source frameworks that aren’t as often mentioned:

PydanticAI builds on Pydantic with minimal abstraction, offering a bare-bones framework that’s highly transparent. It’s great when you need strict type safety and predictable, validated outputs, for fine-grained control, making it easier to debug.

Atomic Agents is developed by an individual agent builder and uses schema-driven building blocks you connect like Lego, with a strong focus on structure and control. It was built in response to the lack of alternatives that worked well in practice.

Both of PydanticAI’s and Atomic Agent’s goal is to move away from black-box AI that acts independently.

Mastra, created by the team behind Gatsby, is a JavaScript framework built for frontend developers to easily build agents within their own ecosystem.

We’ll go through what each of these have, and what makes them different.

What they all have

Most frameworks come with the same core building blocks: support for different models, tools, memory, and RAG.

Most open-source frameworks are more or less model agnostic. This means they’re built to support various providers. However, as mentioned earlier, each framework has its own structure for system prompts — and that structure may work better with some models than others.

That’s also why you ideally want access to the system prompt and the ability to tweak it if needed.

All agentic frameworks support tooling, since tools are essential for building systems that can act. They also make it easy to define your own custom tools through simple abstractions. Today, most frameworks support MCP, either officially or through community solutions.

It’s important to understand that not all models are built for function calling, which is necessary for using tools. To figure out which models are best suited as the base LLM, you can check Hugging Face’s agent leaderboard.

To enable agents to retain short-term memory between LLM calls, all frameworks make use of state. State helps the LLM remember what was said in earlier steps or parts of the conversation.

Most frameworks also offer simple options to set up RAG with different databases to provide the agent with knowledge.

Finally, nearly all frameworks support asynchronous calls, structured outputs, streaming, and the ability to add observability.

What some don’t have

Frameworks will differ in some areas, such as supporting multimodal input, memory, and multi-agent systems. Some handle it for you, while others leave the wiring to you.

First, some frameworks have built-in solutions for handling multimodality — i.e. text, image, and voice. It’s all possible to implement this yourself, as long as the model supports it.

As said previously, short-term memory (state) is always included — without it, you can’t build a system that uses tools. However, long-term memory is trickier to implement, and this is where frameworks differ. Some offer built-in solutions, while others you’ll have to connect other solutions on your own.

Frameworks also vary in how they handle multi-agent capabilities. Multi-agent systems allow you to build collaborative or hierarchical setups with teams of agents connected via supervisors.

Most frameworks recommend keeping agents focused — a narrow scope with a limited set of tools. That means you’ll likely need to build out teams of agents to handle complex workflows. All frameworks let you build one team, but some get complicated when scaling into multi-hierarchical systems with multiple layers.

This is where LangGraph stands out — you can build out nodes, connect them to various supervisors, and visualize how different teams interact. It’s clearly the most flexible when building multi-agent systems at scale.

Agno recently added support for teams, both collaborative and hierarchical, but there aren’t many examples yet for more complex, multi-hierarchical setups.

SmolAgents lets you connect agents to a supervisor but can get complex as the system grows. It reminds me of CrewAI in how it structures agent teams. Mastra is similar in that sense.

With PydanticAI and Atomic Agents, you’ll need to manually chain your agent teams, so orchestration falls on you.

I have gathered a lot of the research I have done in this google sheet, you’ll also find a table in this repository if you need a better overview.

How they’re all different

Frameworks differ in how much they abstract away, how much control they give agents, and how much coding you’ll need to do to get something working.

First, some frameworks make it a point to include a lot of built‑in features, making it easy to get started quickly.

I’d say Mastra, CrewAI, and to some extent Agno are built to be plug and play.

LangGraph also has a good amount of abstraction, but it uses a graph-based system where you manually connect nodes. That gives you more control but also means you have to set up and manage every connection yourself, which comes with a steeper learning curve.

Then we have the low-level abstraction frameworks like PydanticAI, SmolAgents, and Atomic Agents.

These make it a point to be transparent, but you often have to build out the orchestration yourself. This gives you full control and helps with debugging — but it also increases time to build.

Another point of difference is how much agency the framework assumes the agent should have. Some are built on the idea that LLMs should be smart enough to figure out how to complete the task on their own. Others lean toward tight control — giving agents one job and guiding them step by step.

AutoGen and SmolAgents fall into the first camp. The rest lean more towards control.

There’s something to consider here: when developers build frameworks that focus on tight control, it’s often because they haven’t found a way to let agents work on their own yet — at least not reliably.

This space is also starting to look more and more like engineering.

If you’re going to build these systems, you do need to understand how to code. The real question is how much the frameworks differ in terms of how technical you need to be.

If you’re less experienced, going with CrewAI, Agno, or Mastra might be a good idea.

SmolAgents is also pretty straightforward for simple use cases.

As for PydanticAI, Atomic Agents, and LangGraph — you’ll be writing a lot more of the logic yourself. Though to be fair, it’s always possible to build an agent to help you structure your code correctly as well.

If you’re completely new to programming then you can check out Flowise or Dify.

Lastly, it’s worth mentioning the developer experience across these frameworks.

From what I’ve seen, most developers find CrewAI and AutoGen tricky to debug. SmolAgents’ CodeAgent introduces a novel approach where agents output code to route data — a cool idea, but it doesn’t always work as intended.

LangGraph, especially when paired with LangChain, comes with a steep learning curve and some confusing abstractions that you may end up having to break apart and rebuild.

PydanticAI and Atomic Agents are generally liked by developers, but they do require you to build the orchestration yourself.

Agno and Mastra are solid choices, but you might run into issues like looping calls that can be hard to debug.

Some notes

The best way to get started is just to jump in and try something out. But I hope this gave you a decent overview of the open-source frameworks out there — and what might work for you.

This was a fairly shallow look at each one though, as I haven’t gone into things like enterprise-grade scalability or operational robustness. You’ll need to research those parts separately if that’s what you’re building for.

Some devs say AI agent frameworks are some of the worst forms of abstraction — that they often make things more complicated than just using the official LLM provider’s SDK directly.

I’ll leave that one up to you to decide.

Remember the full list of features for each of these frameworks you can find here, and the repository with the full list of agentic frameworks you can find here.

I hope you liked it, and if so be sure to like it, comment or share it.

Note: this article is from May 2024 but the content still holds up well.

I haven’t ventured out of natural language processing much, but I have wanted to write about using computer vision for some time, specifically with a practical use case.

The task here is to classify traffic levels by public traffic cameras planted across the country in Norway. The cameras are almost updated in real-time, while the traffic API that is also available is sometimes updated hours after the traffic levels have been calculated.

The idea is to automate a process that checks the pictures of the cameras every minute, so we can estimate the traffic levels well before the API catches on.

The model I will build in this article was originally created for a media outlet in Norway but has been open-sourced, and you can find it here. It has been built using images from fifty out of the hundreds of public traffic cameras in Norway.

You’ll see me demonstrate how to use it below.

Do grab a picture from this or this camera and try it out to see how it does via the model page*

For due diligence though, I did check other options such as Google Traffic, which does provide us with good data for urban areas. However, we weren’t allowed to tap into it and it had very little data on areas outside of the city limits so it wasn’t a viable solution.

For Google Traffic, they estimate levels based on data that is shared by their user’s phones whereas we would do this manually, checking the public cameras for traffic levels with computer vision by estimating how many cars there are on the road or are waiting in line.

To understand the data we’re working with, do take a look at the dataset we’ll be using here.

As you can see, the images require us to interpret not just the amount of cars but whether they are waiting in line in various directions, in various different sceneries.

This article will go into the different architectures for the task — image classification — trying an older CNN model like ResNet and comparing it to newer models — ViT (Vision Transformer), Swin Transformer and ConvNEXT. I will also give a CLIP model a try in the introduction section.

I have not tried edge cases for this model, which you should do before you push it into production.

I’ve shared most cook books I’ve used in a Github repository that you can find here. These will help you prepare your image dataset to fine-tune for image classification, as well as how to train it with a ViT or a CNN model like ResNet.

I will go through the case and the results but feel free to scroll past the introduction if you want to go directly to the training section.

Introduction

The Economics of Custom Models

If your mind is spinning right now with the question on why I would bother to build my own vision model versus just using a CLIP model or GPT-4 Vision, I’ll go through it briefly.

A CLIP model allows you to do zero-shot inference, which means you will provide it with the labels and it will do the best that it can with those labels to correctly estimate what is pictured in the image. For simpler cases, you should definitely go for a CLIP model. They are small enough that it won’t cost that much to host them.

Test out a ViT based CLIP model here by providing a few labels and an image it should classify.

However, with a CLIP model I wouldn’t be able to train the model to understand the nuances between medium and high traffic, and in most cases a high traffic image would get classified as medium traffic. It very much depends on how the model interprets the images.

The CLIP model I linked to is also 5 times larger than the custom trained model that does better at the same task.

The same would happen if I used GPT-4 Vision; it wouldn’t always classify an image correctly that I would deem as high traffic. This is not accounting for cost, considering GPT-4 is vastly larger than a CLIP or a custom model.

If your first idea is to go for GPT-4 Vision, let’s first estimate the cost day to day if we were to monitor 50 to 150 cameras every minute, so you get an idea of the economics of using such a large model.

It seems dumb to visualize it, as most would never use these bigger models at such a high frequency but sometimes the economics of it isn’t taken into account. You need to consider the resources you are using and how to optimize those resources for the task at hand.

Let’s also look at the cost difference for hosting a CLIP at 428M parameters, or this custom built model at 85M parameters. The cost difference here is less, so in most cases it makes sense to use a CLIP.

Computer Vision

Computer vision is a subset of AI that trains computers to interpret and understand what we can “see.” Just as with natural language processing, we sort different areas into tasks, where we can choose different models and architectures to work with the task.

The most popular tasks within vision is image classification, object detection and segmentation.

The task I’m working with here is image classification, where I want a model to assign a label or a class to an image. If you’ve worked with text classification, it follows the same principles.

So for image classification, as with other tasks, you have several pre-trained models to choose from. So which one should you pick? Vision Transformers are fairly new and if you Google a bit you’ll find that most have been working with Convolutional Neural Networks (CNNs) such as ResNet.

The idea is that ViTs should be good at capturing global context and dependencies between distant parts of an image because of their self-attention mechanisms. However, the catch is that ViTs need a lot of data to perform. CNNs on the other hand, should be more efficient on smaller models and datasets. CNNs also have a proven track record.

A ViT will need a lot of data to perform, but it is a bit unclear if it is enough that the pre-trained model has been trained on enough data or if it needs a hefty amount for fine-tuning as well. If you listen to some people, it’s enough if it has been pre-trained on enough data.

The ideal thing is to test the different models, using a CNN — like ResNet — and a ViT model along with the newer models such as ConvNEXT and Swin Transformer on your dataset to see which does better.

I have done exactly this below, and you’ll see the metrics I achieved for each model.

If you are muddling through the same process, you’ll find cook books here to fine-tune for all different models.

The Use Case: Estimating Traffic Levels

Like I mentioned before, the use case we’re working with here is to classify the level of traffic from images from public road cameras in Norway.

These cameras are accessible to the public. You can freely download and use any photos and illustrations from the Norwegian Public Roads Administration.

I set up a script that would fetch images for certain times during the day and then I collected them into a folder that I later speed sorted whenever I had a few hours available.

I did not have an unlimited amount of time, but in most cases you need more varied data to train with.

The finished dataset I used you’ll find here.

The problem with these images is that some roads consistently experience high traffic whereas other roads don’t. This then means that we’ll get a very skewed dataset as it’s rare that the roads will be packed with cars during all hours of the day. As you’ll notice, we have 4,200 images with low traffic and only 800 images with high traffic.

For this specific case, I did not need it to perform perfectly but I couldn’t have an image being classified as high traffic when it is clearly low traffic.

If it started generalizing it would be useless.

Nevertheless, if it sometimes classifies a high traffic image as medium traffic or a medium traffic image as a low traffic this would be less of a concern.

I’ll go through the results from the training directly, and if you’re keen you can check how I trained the model at the next section.

To train the model, I tested several models using 5e-5 as a learning rate with 5 epochs. I didn’t find that much of an improvement by increasing the number of epochs.

Surprisingly, a pre-trained standard ViT model did well on only 6,800 images.

The ViT model is three times as large as the ResNet I used, but still quite small at 85M parameters.

The metrics show us that a ViT performed better on all metrics. Accuracy is the one we’re looking at the most, whereas the F1 will help us estimate how much our skewed dataset is a problem. A high F1 score means the model is not just guessing well for the common categories, but also doing a good job at correctly estimating the rare categories, such as medium and high traffic.

Furthermore, testing the model manually on new images, I found that the ResNet was more likely to classify a low traffic image as high traffic than a ViT, so this is why I primarily went with ViT along with the higher performance metrics.

The result could have primarily been because the ViT model was naturally larger but I needed a fairly large model for this case. I also wonder if a ViT is a better choice for these images where I need it to analyze the entirety of it to estimate traffic levels.

Continuing my experiment, I did not find that using a ConvNEXT or a Swin Transformer model gave me any positive change. They did surprisingly well though.

The metrics for ConvNEXT and Swin Transformer were good. However, testing the model on new images gave me inflated results where medium traffic images on high traffic roads were classified as high traffic, which was less of an issue with a standard ViT model.

This could have been a result of how my dataset had been filtered confusing the model with too many medium-traffic images in the high-traffic folder so I wouldn’t disregard these other models.

The metrics did not improve with a custom trainer to account for weight imbalances in the dataset when training a ViT. It performed about the same in practice with worse metrics. But there is probably room for improvement there as well.

Lastly, trying to balance the dataset manually by further introducing more high traffic images unfortunately gave me more generalized results on some high traffic roads — i.e. for high traffic roads it inflated the results regardless if the road was mostly empty — so for this case having an unbalanced dataset proved fruitful.

I did not try data augmentation to increase the size of the imbalanced dataset as I didn’t believe it would do any good.

Nevertheless, continuing to work on the dataset by setting up a script to pick up images over a longer period of time is a good idea. I would still keep it skewed to reflect the real thing though as I found it did better if it reflected reality.

Traffic images are notoriously difficult, as each camera will show a different scenario and a different scenery. Because of this I would also set up an algorithm later that would check how many images have been classified with high traffic between a few minutes to estimate how bad the traffic congestion is.

I would stress that this model is not battle-tested, and it would need to get further tested and trained.

Training the Model

I have provided several cook books so you can test both a CNN and ViT model using the HuggingFace trainer. You can tweak them as you go along, if needed.

The process to train this model, is the following.

1. Prepare the dataset

2. Preprocess the dataset

3. Decide on your metrics

4. Train the model

5. Evaluate the model

Preparing a Dataset

When you work with image classification, you want to sort your images into folders. The folders will act as your labels, or categories.

From here it is easy enough to prepare and load your dataset so it can be pushed to the HuggingFace hub.

I usually mount my Google Drive in Colab and then simply load the dataset.

from google.colab import drive
drive.mount(’/content/drive’)

from datasets import load_dataset  dataset = load_dataset(’imagefolder’, data_dir=dataset_path)   

You may want to check that the folders don’t have any corrupt files that will later be a problem when you train the model.

from PIL import Image
import os

dataset_path = ‘/content/drive/MyDrive/your-image-folder’

def verify_images(folder_path):
    for subdir, dirs, files in os.walk(folder_path):
        for file in files:
            filepath = os.path.join(subdir, file)
            try:
                with Image.open(filepath) as img:
                    img.verify()
            except (IOError, SyntaxError) as e:
                print(f’Corrupt image: {filepath} | Error: {e}’)
                os.remove(filepath)

verify_images(dataset_path)

If you first do the check above, and delete any corrupt files, load the dataset after this.

Once you’ve loaded it, you can continue to split it into a training and validation set.

from datasets import load_dataset, DatasetDict

train_dataset = dataset[”train”]

split_datasets = train_dataset.train_test_split(test_size=0.1, seed=42, stratify_by_column=’label’)

train_dataset = split_datasets[’train’]
val_dataset = split_datasets[’test’]

dataset_dict = DatasetDict({
    ‘train’: train_dataset,
    ‘validation’: val_dataset
})

We always use a training set to fit the model and a validation set to evaluate its performance.

When you’re done you can train it directly or you can push it to the HuggingFace hub to store for later training.

To push it to the hub you login like so.

!huggingface-cli login

You’ll need an access token that you can find in your HuggingFace account under Settings.

Then you simply push it.

repo_name = “username/traffic-camera-norway-images” 
dataset_dict.push_to_hub(repo_name)

See the full script here to prepare your image data.

Find a Pre-Trained Model

Now as mentioned you’ll have to decide which pre-trained model you want to train. I will use a ViT here because it did best, but if you want to go for a ConvNEXT or a CNN model see other cook books here.

I’ll be going with the model vit-base-patch16–224 but you can try another one but just make sure you match the input specifications of each model, particularly the image size, when you preprocess your data later. I’ll explain it once we get to this part.

The vit-base-patch16–224 model has been pre-trained on the ImageNet-21k which should be enough data.

Open the Colab Notebook

If you’re good to go you can open this Colab notebook that I have already prepared.

At the start you’ll be able to set a few variables, your dataset url in HuggingFace, the pre-trained model you’ll fine-tune, the new model name as well as the learning rate, epochs and batch size.

dataset_url = “ilsilfverskiold/traffic-camera-norway-images” 
model_checkpoint = “google/vit-base-patch16-224” 
new_model_name = ‘traffic-image-classification’
learning_rate = 5e-5
epochs = 5
batch_size= 32

The standard learning rate is 5e-5, and with 6800 images something like 5–10 epochs should be ideal. I didn’t find that it better after 5 epochs but it is up to you if you’d like to test it on more or less.

There should be a ton of information on this out there if you’d like to dig deeper.

I will skip a few sections, but do make sure you follow along in the Colab notebook.

Preprocess Dataset

This pre-trained model we’re using has expectations on what the data should look like when we train it. Different models may have been trained with different image normalization standards.

So we load something called an image processor.

from transformers import AutoImageProcessor

image_processor  = AutoImageProcessor.from_pretrained(model_checkpoint)

We’ll use this one to normalize the images we’ll be training with so it will adjust the color channels to the same range and scale that the model was originally trained on.

We also resize the images to 256 pixels and then crop to the center to achieve a final size of 224x224 pixels, to follow the model’s input requirements.

Remember we picked a model with a resolution of 224x224 with vit-base-patch16–224.

from torchvision.transforms import (
    Compose,
    Resize,
    Normalize,
    CenterCrop,
    RandomHorizontalFlip,
    RandomResizedCrop,
    ToTensor,
)

normalize = Normalize(mean=image_processor.image_mean, std=image_processor.image_std)

train_transform = Compose([
    Resize(256),
    CenterCrop(224),
    RandomHorizontalFlip(),
    ToTensor(),
    normalize,
])

val_transform = Compose([
    Resize(256),
    CenterCrop(224),
    ToTensor(),
    normalize,
])

def apply_transform(examples, transform):
    examples[’pixel_values’] = [transform(image.convert(’RGB’)) for image in examples[’image’]]
    return examples

def set_dataset_transform(dataset, transform):
    dataset.set_transform(lambda examples: apply_transform(examples, transform))

set_dataset_transform(dataset[’train’], train_transform)
set_dataset_transform(dataset[’validation’], val_transform)

The other random stuff we do, like random flips and other similar augmentation techniques are used to improve the robustness and generalization ability of machine learning models.

Lastly, the ToTensor() transformation converts PIL images to PyTorch tensors, formatting the data that is required by PyTorch models.

After this it is good to check that you have another field called pixel_values for each item with tensors.

dataset[’train’][0]

Remember to follow along in the Colab notebook for the entire script.

Evaluation Metrics

You’ll also want to set up some evaluation metrics. You saw earlier that I was evaluating the model based on a few metrics although you always have to test it manually as well to see how it does with new data.

The most important metric you’re interested in is Accuracy, which measures the amount of predictions the model got right across all categories. This one you’ll see everywhere but there are other metrics you may want to use as well.

Precision measures how often predictions for a specific category are correct. So for my case, with traffic levels, high precision in a traffic category like ‘high traffic’ means that when the model predicts high traffic, it is usually right.

Recall tells us how well the model can identify all instances within a specific category, such as ‘low traffic’. High recall means the model is good at recognizing most low traffic situations.

The F1 Score is the weighted average of Precision and Recall.

When you have a skewed dataset like I have where some categories don’t show up as much as others, a high F1 Score is really good. It means the model is not just guessing well for the common categories, but also doing a great job at picking up the rare ones correctly.

To tweak the metrics I’ve set up for this use case, you can navigate to this part of the Colab notebook.

import numpy as np
from datasets import load_metric

accuracy_metric = load_metric(”accuracy”)
precision_metric = load_metric(”precision”)
recall_metric = load_metric(”recall”)
f1_metric = load_metric(”f1”)

def compute_metrics(eval_pred):
    logits, labels = eval_pred
    predictions = np.argmax(logits, axis=1)

    accuracy = accuracy_metric.compute(predictions=predictions, references=labels)
    precision = precision_metric.compute(predictions=predictions, references=labels, average=’macro’)
    recall = recall_metric.compute(predictions=predictions, references=labels, average=’macro’)
    f1 = f1_metric.compute(predictions=predictions, references=labels, average=’macro’)

    metrics = {
        “accuracy”: accuracy[’accuracy’],
        “precision”: precision[’precision’],
        “recall”: recall[’recall’],
        “f1”: f1[’f1’]
    }
    return metrics

Model Training

From here we can prepare to train the model. Remember not to skip any parts in the notebook as I’m not going through all the steps here.

I’m using an L4 in Colab to train as I have a pro membership but this should work with a T4 as well, only the training may be slightly slower. Just remember to use a GPU.

Ideally you can keep the training arguments as is, and tweak epochs and learning_rate at the start.

args = TrainingArguments(
    f”{new_model_name}”,
    remove_unused_columns=False,
    evaluation_strategy = “epoch”,
    save_strategy = “epoch”,
    learning_rate=learning_rate,
    per_device_train_batch_size=batch_size,
    gradient_accumulation_steps=4,
    per_device_eval_batch_size=batch_size,
    num_train_epochs=epochs,
    warmup_ratio=0.1,
    logging_steps=10,
    weight_decay=weight_decay,
    load_best_model_at_end=True,
    metric_for_best_model=”accuracy”,
    push_to_hub=False,
)

Do not remove remove_unused_columns as you’ll get an error. I have specified to not push it to the hub as I want to evaluate the model first.

We also set up the trainer with the prepared train and validation datasets.

trainer = Trainer(
    model,
    args,
    train_dataset=dataset[’train’],
    eval_dataset=dataset[’validation’],
    tokenizer=image_processor,
    compute_metrics=compute_metrics,
    data_collator=collate_fn,
)

If you’re satisfied you can go ahead and train the model.

train_results = trainer.train()

trainer.save_model()
trainer.log_metrics(”train”, train_results.metrics)
trainer.save_metrics(”train”, train_results.metrics)
trainer.save_state()

You’ll see the metrics for each epoch once it is training. What you’re looking for here is the training loss, which should consistently going down, while validation loss should do the same. I sometimes see it fluctuating, though it shouldn’t so do keep an eye on it.

Accuracy should obviously increase, as I mentioned when I talked about the evaluation metrics earlier.

Evaluate Model

Once it has finished training you can evaluate the final metrics.

metrics = trainer.evaluate()
trainer.log_metrics(”eval”, metrics)
trainer.save_metrics(”eval”, metrics)

My metrics weren’t stellar for this first run, but good enough for this case.

***** eval metrics *****
  epoch                   =     4.9215
  eval_accuracy           =     0.8292
  eval_f1                 =     0.7721 # good enough
  eval_loss               =     0.4394
  eval_precision          =     0.8232
  eval_recall             =     0.7366

The loss was quite high while the training loss a bit lower, which could indicate overfitting. I had better metrics for a few other models.

This is where you’ll want to also test the model on new data to see how it does. This model did better at new images than the other models that inflated the labels from medium to high traffic.

This could have been an issue with my validation set just not being large enough, and not representing real use cases.

So, I would also test it on various images to see how it does. For me this was easy enough as I ran it through a few new traffic images that I had stored in my Google Drive.

To do this save the model and then set up the pipeline with the new model.

trainer.save_model(’new_model’)

from transformers import pipeline

pipe = pipeline(’image-classification’, model=’new_model’)

Connect your Google Drive.

from google.colab import drive
drive.mount(’/content/drive’)

Then do some inference on the images you want.

from PIL import Image

image_path = ‘/content/drive/MyDrive/image_to_test.jpg’ # path to your image

image = Image.open(image_path)

results = pipe(image)
results

Push to Hub (Optional)

If you’re decently satisfied, you can push the model to the Hub to use from there or to deploy it as an inference endpoint so you can use it in production.

You’ll see me doing this in the Colab notebook at the end. The notebook should help you from start to finish.

Now we can test the model in the hub using the Inference API and see how it does.

I’m always terrified that I’ll find something I don’t want to see so I haven’t yet tried any edge cases yet, like a picture with a bunch of trees in the middle of the screen.

Nevertheless, we know it might miss a few high traffic images but at least it’s not classifying low traffic as high traffic.

I open sourced the model I created here which is probably far from perfect but you are very welcome to use it.

I also open sourced the first dataset that you can find here.

Hopefully this proved resourceful and you get some inspiration for your next project.

To follow my writing, building and whatnot look into my website, or my LinkedIn as well.

LangGraph is one of the more popular low-level orchestration frameworks for building agentic workflows among the developer community. It was built by the LangChain team, who hoped it would give builders better control by making the flow explicit.

It might feel overwhelming at first, but it gets easier once you understand the fundamentals—which is exactly what we’ll go through here by building a single-agent workflow that we can visualize and test in LangGraph Studio.

Essentially, with LangGraph, you’re writing code that can be visualized as a graph with nodes. You won’t be working in a visual programming system; instead, you code it manually and then visualize and debug it via LangGraph Studio (though that part is optional).

Understanding the fundamentals

To build something simple, we still have to go through a few core pieces: graphs, nodes, edges, and state.

The graph is the overall framework for setting up your agent’s workflow. See it as the environment we need to execute our workflow. We always set up the graph first.

After defining the graph, we set up our nodes—these are the core functionalities or operations. Nodes are where everything happens. In an agent workflow, a node could do anything from calling an LLM to invoking a tool (like a search function or an API), or performing some computation.

The edges are the connections between nodes. They tell the graph which node to go to next. Edges can be static (always go from A to B) or conditional (branch based on something in the state). So essentially, nodes do the work, edges decide what happens next.

State is the current memory or context of the workflow, where we hold all the data that flows between nodes.

You can think of state as short-term memory—it sticks around during the current run or conversation. It often holds things like conversation history between LLM calls or variables we need in multiple places.

Technically, it’s just an object where we store relevant info and pass it along from node to node. This is especially important for LLMs, since we need to pass in the conversation history for each call (as LLMs are stateless by nature).

So in short: the graph is the canvas, nodes are the actions, edges define the flow, and state carries the data or information throughout.

Setting up the environment

To make this work, you’ll need to have Python installed on your computer—version 3.9 or above.

You’ll also need the latest version of Docker. If you run into issues with an older version not updating, uninstall Docker completely and download the latest version from scratch.

Then, download LangGraph Studio (desktop), since we’ll use it to visualize and debug the workflow. If you’re not on a Mac, there should be other options you can explore.

Setting up the files

To start, you can create a new folder to set up this workflow in:

mkdir langgraph_example
cd langgraph_example

Inside this folder, create another folder called agent:

mkdir agent

This is where we’ll set up the code for the agent.

Since I’m using LangGraph Studio, I’m organizing the code for that—but you don’t have to follow this exact structure if you’re not using the Studio.

Now create a langgraph.json file:

touch langgraph.json

This file tells LangGraph where to find the agent code and the .env file for credentials:

{
  "dependencies": ["./agent"],
  "graphs": {
    "agent": "./agent/agent.py:graph"
  },
  "env": ".env"
}

Next, go into the agent folder and create a requirements.txt file:

langgraph
python-dotenv

Also create a file called agent.py in the same folder—this is where we define our workflow.

So far, the structure should look like this:

langgraph_example/
│
├── langgraph.json
│
└── agent/
    ├── agent.py
    └── requirements.txt

Defining the graph

Now we can start building our agent by defining the graph. As mentioned earlier, the graph is the foundational structure for the workflow. It acts as the execution environment where nodes, edges, and state work together in a coordinated way.

In agent.py, start with this:

from langgraph.graph import StateGraph

# define a new graph
workflow = StateGraph('')

This will allow us to later attach nodes as operations/functions, and edges to define transitions between them.

Define the state

Next, we define the custom state and pass it to StateGraph:

from langgraph.graph import StateGraph
from langgraph.graph import add_messages
from langchain_core.messages import BaseMessage
from typing import TypedDict, Annotated, Sequence

class AgentState(TypedDict):
    messages: Annotated[Sequence[BaseMessage], add_messages]

workflow = StateGraph(AgentState) # Add the AgentState here

Think of state as a schema that defines what fields can exist and flow between nodes. When a node returns data, it’ll be checked against this schema. If the key exists, it gets updated—if not, you’ll likely get an error.

Here we’re only using messages, but you can expand this to include any data you want passed between nodes.

If you're unfamiliar with how message history works in LLMs—remember they’re stateless. We need to store messages ourselves so the model “remembers” context between calls.

Is it necessary to have this state object in LangGraph? Yes. Without it, the model wouldn’t know what tools were called or what information it had gathered. And building this without tools defeats the purpose entirely.

Set Up the LLM

Next up we’ll set up the LLM model we’ll be using as the base for our agent. We’ll use LangChain to connect to Gemini 2.0 Flash:

from langchain_google_genai import ChatGoogleGenerativeAI

def _get_model():
    model = ChatGoogleGenerativeAI(model="gemini-2.0-flash-001")
    
    return model

To be able to import ChatGoogleGenerativeAI we need to add this dependency to requirements.txt:

langchain_google_genai

Also make sure to create a .env file in the root with:

GOOGLE_API_KEY=your_key_here

LangChain will automatically pick this up based on the path set in langgraph.json.

If you want to use another model you can look into LangChain documentation on models here.

Call the Model

Now let’s define the function that we’ll attach to a node that calls the LLM:

system_prompt = "You are such a nice helpful bot"

def call_model(state):
    messages = state["messages"]
    
    # add the system message
    system_message = SystemMessage(content=system_prompt)
    full_messages = [system_message] + messages
    
    model = _get_model()
    response = model.invoke(full_messages)
    
    return {"messages": [response]}

As you see here we can directly access state via the params where we store the messages of the function and then simply invoke the model. The messages are then returned to state.

Here you will ideally add some error handling in case the LLM call fails, but to keep things very simple I’ve just added the bare minimum.

Define and Attach Nodes

Now that we have the LLM setup we can start to define our nodes.

Remember the nodes are the functions or operations in our workflow. The first node for the agent we’ll point to the call_model function we defined earlier.

# define nodes
workflow.add_node("agent", call_model)
workflow.add_node("tools", tool_node)

We can then setup one for the tools it should have access to. As of right now we still haven’t defined what is in the ‘tools’ node so we need to code this as well.

Adding tools

To add tools, we’ll create a few mock tools (functions) just for demonstration. These should ideally do something meaningful—like calling an API and returning real information:

from langgraph.prebuilt import ToolNode
from langchain_core.tools import tool

@tool
def weather_tool() -> str:
    """Get the current weather."""
    return "Weather is 19°C and Partly Cloudy"

@tool
def calendar_tool() -> str:
    """Check your calendar for meetings on a specific date."""
    return "No meetings scheduled"

all_tools = [weather_tool, calendar_tool]
tool_node = ToolNode(all_tools)

We then bundle the tools together into a ToolNode that we import from LangGraph. This one should handls the execution of tools in our workflow.

If you define parameters for the tools, the model will understand that it can use them after we bind them. It will receive the tool name, description, parameters, and return type.

We can now visualise what we’ve set up so far.

As noted earlier, when we bind the tools to the model, it will become aware of their existence. So we need to go back to the _get_model() function and update it:

from langchain_google_genai import ChatGoogleGenerativeAI

def _get_model():
    model = ChatGoogleGenerativeAI(model="gemini-2.0-flash-001")

    model = model.bind_tools(all_tools) # add this line
    
    return model

This part is crucial—it tells the model which tools are available and how to use them. There’s a lot of abstraction happening here where LangChain structures the interaction between your Python code and the LLM API using bind_tools.

I would say there are abstractions in this entire framework that you may want to do more proper research around later.

Adding edges

Now we can start connecting these nodes, but first we set the entry point for the workflow:

# entry point
workflow.set_entry_point("agent")

Next, we’ll define some conditional logic, where we set up a function that decide if we should keep calling the LLM based on what it did last.

# define the function that determines whether to continue or not
def should_continue(state):
    messages = state["messages"]
    last_message = messages[-1]
    # if there are no tool calls, then we finish
    if not last_message.tool_calls:
        return "end"
    # if there is, we continue
    else:
        return "continue"

This logic tells the workflow to continue if the model just used a tool, and end if it didn’t.

To support this, we set up a conditional edge (a dashed connection between the agent and tool nodes):

from langgraph.graph import END

workflow.add_conditional_edges(
    "agent",
    should_continue,
    {
        "continue": "tools",
        "end": END,
    },
)

If the last LLM call didn’t use any tools, we assume the workflow is complete and stop it.

Here we are also importing END from LangGraph which is a termination signal to stop node traversal and return the final result.

We also need to connect the tools back to the agent:

# connect tools back to agent
workflow.add_edge("tools", "agent")

This is needed so that after a tool executes, its result is sent back to the LLM.

A question that might come up here is: why do we need this graph structure if the model already knows about the tools through bind_tools?

The answer is: there's a difference between the model knowing about the tools and actually executing them. The model can't execute tools on its own—we need the graph to define and control that execution.

Compiling the workflow

The last thing we need to do is compile the workflow:

graph = workflow.compile()

Without this step, nothing will run.

The finished code will be very simple for our agent.py file.

Now, this is a very simple workflow we’ve gone through—just for demonstration. To build a multi-agent workflow, you’ll need to add to the state and then add more nodes and conditional logic to the graph.

Testing the workflow

To test this workflow, open up LangGraph Studio, connect to LangSmith (free to create an account), turn on Docker Desktop (make sure it's the latest version), and then open the langgraph_example project.

Once it’s loaded, you’ll be able to add a human message and submit it to see what happens.

Remember, it only has those two mock tools available, so it won’t be terribly useful.

You can see in my example that I first asked it to check the weather and my calendar, and then got the final response at the end.

I would keep playing around with it until these concepts stick or do some research around how everything works to get more comfortable with it. As a suggestion, you can start setting up real tools that check the news (or actual weather) in the workflow above.

Hopefully, you learned something—if LangGraph is new to you. Feel free to check out more of my work on TDS and Medium as well.

😘