It’s been a while since the last article I published on The Tech Buffet.

New here? The Tech Buffet is a newsletter about practical ML engineering and software, one hands-on issue at a time — subscribe below if you want the rest of this series landing in your inbox.

Subscribe now

For this comeback, I wanted to start with a tool I’ve been spending quite a bit of time looking into lately: Claude Code.

You’ve probably already seen the demos.

Someone opens a terminal, types:

Build me a SaaS for managing my personal finances.

Claude creates 30 files, installs a bunch of packages, and, a few minutes later, there’s an application running.

Impressive stuff.

But it’s not the use case I’m most interested in.

In real software development, we rarely start from an empty directory.

We join existing projects. We navigate code written by other people. We investigate bugs. We try to understand why something was implemented a certain way before touching it.

That’s how we’re going to learn Claude Code today.

Instead of asking it to build another Todo app, we’re going to give Claude an existing FastAPI application that we didn’t write and see how useful it can be.

By the end of this article, we’ll use Claude Code to:

• Explore an unfamiliar repository

• Understand its architecture

• Trace how an existing feature works

• Run commands and tests

• Make a small code change

• Review what Claude actually modified

Nothing particularly advanced yet.

No MCP servers.

No subagents.

No Skills.

We’ll first get comfortable using one Claude on one real codebase.

The application we’ll use

For this tutorial, we’ll use the open-source fastapi-realworld-example-app repository.

It’s an implementation of the RealWorld API specification built with FastAPI.

The application contains CRUD operations, authentication, routing, pagination, PostgreSQL, SQLAlchemy, database migrations and a pytest test suite. Its source code is split across API routes, CRUD functions, database code, models and schemas.

In other words, there is enough code for Claude to actually have something to understand.

Let’s clone it:

git clone https://github.com/adr1enbe4udou1n/fastapi-realworld-example-app.git

cd fastapi-realworld-example-app

Before doing anything with Claude, let’s have a quick look at the repository. By typing this command:

ls -la

You’ll see something roughly like:

Inside app/, things become a little more interesting:

And inside the API routes:

The repository also includes a PostgreSQL setup, Alembic migrations, seed data and a full pytest suite.

You could now start opening those files one by one.

But that’s precisely what we’re going to ask Claude to do for us.

Installing Claude Code

In this section, I assume you have a working Claude Pro or Max subscription.

If you haven’t installed Claude Code yet, the recommended native installation on macOS, Linux and WSL is currently:

curl -fsSL https://claude.ai/install.sh | bash

Once installed, you can type this command to check Claude’s version.

claude --version

As I’m writing this post, my Claude version is 2.1.233.

Now, from inside the repository, run:

claude

That’s basically all we need to start.

Claude Code is now running with our repository as its working directory and can inspect files, search the codebase and execute development commands from the terminal.

Claude starts by asking me if it should trust this folder.

To which I’ll answer yes. And we’re good to go.

Don’t ask Claude to write code yet

The first thing I’m interested in here isn’t code generation.

It’s this:

Can Claude understand an application it has never seen before?

Let’s start with a very simple prompt.

Explore this repository and explain what the application does.

Don't modify any files.

Claude starts exploring the repository.

It starts by listing the files:

It might inspect things such as:

README.md
pyproject.toml
app/main.py
app/api/
app/crud/
app/models/
tests/

Claude Code starts by running multiple shell commands.

Then produces its answer (while identifying a tiny bug along the way).

All this in 47s.

The interesting part here isn’t really the final explanation.

We could have read the README.

What’s more interesting is that we didn’t tell Claude which files to read.

Claude decided that itself.

You can also ask Claude to describe the core functionalities of the app in simple terms. This helps me validate my understanding. This is also useful for non-technical users.

OK, now let’s push Claude a little further and have it dig into the codebase.

Ask Claude to explain the architecture

When we join an existing project, we may first not be interested in the implementation details of every function and class.

We’d rather have a high-level picture of the architecture, how files are organized and how data flows between the layers.

Let’s ask Claude that:

Explain the architecture of this application.

Identify the main layers, their responsibilities and how they
interact with each other.

Mention the most important files and directories.

Don't modify anything.

Now Claude has to go beyond the README.

It needs to understand how things like:

app/api
app/crud
app/db
app/models
app/schemas

fit together.

When it’s done analyzing, it first gives a mental model organized in a diagram.

 HTTP request
     │
     ▼
  app/main.py            – ASGI app assembly, mounting, CORS
     │
     ▼
  app/api/api.py         – top-level router, prefixes/tags per resource
     │
     ▼
  app/api/routes/*.py      – endpoint functions (path ops, request validation via Pydantic)
     
     ▼
  app/api/deps.py          – auth (JWT→User), DB session injection, repository factories

     ▼
  app/crud/crud_*.py       – repository classes: SQLAlchemy queries, writes, transactions
     │                       reads/writes
     ▼
  app/models/*.py          – SQLAlchemy ORM entities (tables, relationships)
     │
     ▼
  app/schemas/*.py         – Pydantic DTOs returned to the client

Then, it details each component separately.

There are multiple questions I often find myself asking when working on a new codebase:

Where is the FastAPI application initialized?

or:

How is the database connection configured?

or:

Where are request and response schemas defined?

or:

Where are the API routes registered?

Claude can drastically shorten that exploration phase.

These are simple questions. But on a large repository, they can save a surprising amount of time.

Let’s trace a real feature

Understanding folders is useful.

Understanding execution paths is much more useful.

Our application lets users create articles.

Instead of manually searching for the endpoint, let’s ask Claude:

Trace what happens when an authenticated user creates a new article.

Start from the incoming HTTP request and follow the execution
until the article is persisted in the database.

Mention the relevant files and functions along the way.

Don't modify anything.

Now Claude has to connect several pieces of the codebase together.

It should discover the article routes, schemas, authentication dependencies, CRUD logic and database models involved in the operation.

When it’s done, you see that it detailed what exactly happened at each step.

If you’re a bit reluctant to read a lot of text, you can also ask Claude to create a simple diagram. Again, it’s a better mental model.

This is where Claude Code starts becoming much more interesting to me.

Without it, investigating an unfamiliar flow often looks something like:

find route
   ↓
open file
   ↓
follow import
   ↓
open dependency
   ↓
follow function
   ↓
find model
   ↓
inspect tests

Claude can perform much of that navigation before we even open an editor.

We’re still responsible for understanding the code.

But Claude can help us get to the right code much faster.

What about authentication?

Let’s say you’re new to authentication. Having Claude Code as a developer companion can help you learn about this topic.

Let’s try something like this:

Explain how authentication works in this application.

I specifically want to understand:

1. how a user logs in
2. where the access token is created
3. how the token is validated on subsequent requests
4. how an authenticated request is associated with a user

Give me the relevant files and functions.

Notice something important about this prompt.

I didn’t ask:

Explain auth.py.

I’m describing the thing I want to understand, not the files I think contain the answer.

That’s an important difference.

In the following screenshot, Claude explains, step by step how authentication is performed when a user queries the /articles endpoint to create a post.

If I already knew exactly where the implementation lived, I probably wouldn’t need Claude to explore it for me.

You can use the same pattern for almost anything:

Explain how article favorites work.

Trace how comments are created and deleted.

Explain how pagination is implemented for article listing.

Show me how database sessions are created and injected into routes.

This turns Claude Code into something more useful than a code generator.

It becomes a way to interrogate your codebase.

Let Claude run commands

So far Claude has only been reading code.

Let’s let it interact with the project.

The repository uses PostgreSQL and provides a Docker Compose setup. Its README also includes commands for installing dependencies with uv, running migrations, adding seed data and launching the application.

We could execute everything manually.

Or we could ask:

Set up this application locally following the repository instructions.

Before running commands, explain what you intend to do.

Claude can inspect the README and determine the steps.

You’ll still see the commands it wants to execute.

For example, the repository setup includes commands along these lines:

docker compose up -d

uv sync

cp .env.example .env

uv run alembic upgrade head

uv run app/seed.py

uv run uvicorn app.main:app --reload

Depending on your permissions, Claude may ask you before executing certain commands.

This is something I like about the terminal workflow.

Claude proposes an action.

You can inspect it.

Then decide whether it should actually happen.

Once the application is running, its OpenAPI interface is available at /api.

Let’s see whether the tests pass

Before modifying an unfamiliar repository, it’s usually useful to know whether you’re starting from a healthy state.

Ask:

Run the existing test suite.

Don't change anything.

If anything fails, explain why before attempting to fix it.

The project includes a pytest suite, and the repository explicitly documents pytest as its test runner.

Claude can execute the tests and inspect their results.

This gives us something very important before we touch the code:

a baseline.

If the tests pass now and fail after our modification, we know we’ve probably introduced a regression.

That feedback loop is going to become increasingly important throughout this series.

Our first change

We’ve spent most of this tutorial asking Claude to read code.

Let’s finally make it write some.

But we’ll keep the task deliberately small.

Let’s add a health-check endpoint to the API:

GET /health

which returns:

{
  "status": "ok"
}

Instead of telling Claude which file to modify, let’s give it the requirement:

Add a health-check endpoint to this application.

Requirements:

- GET /health
- return HTTP 200
- return {"status": "ok"}
- follow the existing project structure and conventions
- add an automated test

Before modifying anything, inspect the repository and tell me
where you intend to implement it.

There are two things I like about this prompt.

First, we’re giving Claude a requirement, rather than implementation instructions.

We don’t say:

Create health.py inside this folder and import it here.

We let Claude inspect the project and determine how the change should fit into the existing architecture.

Second, we’re giving it a way to know when it has succeeded:

add an automated test

Claude should first explain its approach.

You can review it.

Then let it make the changes.

Review the code, not just Claude’s message

Once Claude finishes, don’t stop at:

Done! The health endpoint has been implemented.

Ask:

Show me the git diff and explain every change you made.

Now inspect the result.

You can also ask:

Did you make any changes that were not required by my request?

And then:

Run the tests relevant to this change.

Finally:

Run the full test suite and confirm that the existing behavior
still works.

Our workflow now looks something like:

Requirement
    ↓
Claude explores repository
    ↓
Claude proposes implementation
    ↓
Developer reviews
    ↓
Claude modifies code
    ↓
Tests
    ↓
Developer reviews diff

This is very different from:

prompt
  ↓
code
  ↓
copy/paste

And I think that’s the important thing to understand about Claude Code.

The prompts matter less than the workflow

You may have noticed that none of the prompts in this tutorial were particularly sophisticated.

There was no:

You are a world-class senior software engineer with 25 years of experience...

We mostly asked simple questions.

Explore

Explain the architecture of this application.

Trace

Trace what happens when an authenticated user creates an article.

Understand

Explain how authentication works.

Execute

Run the existing test suite.

Modify

Add this feature while following the existing project conventions.

Review

Show me the diff and explain every change.

That’s already enough to start getting value from Claude Code.

The important part isn’t finding the perfect magic prompt.

It’s getting used to a workflow where Claude can:

inspect → reason → act → verify

inside the same development environment.

One thing I would avoid

After your first successful task, there is a temptation to immediately ask:

Now refactor the entire architecture.

I wouldn’t.

At least not yet.

Claude Code becomes much easier to use once you understand how it behaves on small, verifiable tasks.

Start with changes where you can easily answer:

Did Claude actually do what I asked?

A health endpoint is trivial.

That’s intentional.

Once you’re comfortable with the mechanics, you can give Claude bigger tasks.

And that’s when planning becomes much more important.

What’s next?

So far, Claude has been able to understand the repository, trace existing features, run commands and make a small change.

But every time you open a new Claude Code session, it starts with the repository in front of it — not with everything you already explained in the previous one.

That means you can quickly find yourself repeating things like:

Use uv.

Run tests before finishing.

Keep database access out of route handlers.

Follow the existing repository patterns.

Don't add a dependency unless it's really needed.

Some of these instructions are worth repeating. Most aren’t.

Claude Code has a built-in way to make project-specific instructions persistent across sessions: CLAUDE.md.

You can even generate a first version automatically with:

/init

The interesting question isn’t how to create the file, though.

It’s:

What information actually belongs in CLAUDE.md, and what is just noise that Claude could rediscover by reading the repository?

That’s what we’ll look at in the next issue. We’ll generate a real CLAUDE.md, criticize its output, cut the parts that don’t change its behavior, and keep only the instructions that are actually useful.

The Tech Buffet provides in-depth tutorials and technical reviews of topics in ML and programming. Consider supporting it by subsribing.

Subscribe now

Building a RAG to «chat with your data» is easy: install a popular LLM orchestrator like LangChain or LlamaIndex, turn your data into vectors, index those in a vector database, and quickly set up a pipeline with a default prompt.

A few lines of code and you call it a day. 

Or so you’d think.

The reality is more complex than that. Vanilla RAG implementations, purposely made for 5-minute demos, don’t work well for real business scenarios.

Don’t get me wrong, those quick-and-dirty demos are great for understanding the basics. But in practice, getting a RAG system production-ready is about more than just stringing together some code. It’s about navigating the realities of messy data, unforeseen user queries, and the ever-present pressure to deliver tangible business value.

In this issue, we’ll first explore the business imperatives that make or break a RAG-based project. Then, we’ll dive into the common technical hurdles — from data handling to performance optimization — and discuss strategies to overcome them.

Disclaimer: The insights shared here don’t represent my employer and come from my personal experiences, research, and more than a few late-night coding sessions.

1 — Clarify the business value from the start: the context, the users, and the data

“Why should we bother building a RAG in the first place?”

Before writing a single line of code, it’s critical to understand why a RAG (Retrieval-Augmented Generation) is even needed from a business perspective. This might seem obvious, but many developers neglect this part, rushing into implementation headfirst without clearly defining long-term objectives. 

Let’s keep you from losing time in the future. Here are some business requirements to consider before starting a RAG/LLM-based project. These principles also apply to general ML projects:

→ Clarify the context: Know your users and what they do. Formulate their main business issue in simple terms, put yourself in their shoes, dissect what they do, and then discuss how RAGs and LLMs can help. Take sample questions and simulate how the system would handle them.

→ Educate non-technical users on the capabilities and limitations of generative AI. Since this is a new field, this is very helpful. This can be achieved through workshops, training sessions, and practical demonstrations that use RAG systems. For example, I’ve seen that internally crafted prompting guides are often helpful for getting started with interacting with LLMs.

→ Understand the user journey: What types of questions will they answer with the RAG? What are the outputs to expect from such a system? How will the generated answers be used down the line? 
Most importantly, you must know how your RAG will integrate into an existing workflow. This foresight will anticipate further developments, such as whether the RAG will be embedded into a browser extension, an API endpoint, or a Word plugin.
Generating an answer using a RAG system is simple. What’s complex, however, is driving practical value from it.

Keep this in mind: the business won’t come to you and play with your fancy models unless those change and improve their ways of working.

→ Anticipate the data to be indexed: Qualify it, map it to the users, and understand why and how it will be used to form a good answer when the RAG pipeline is triggered. This will determine what processing and metadata should be implemented. After qualification, you’ll eventually need to set up a data acquisition strategy: purchasing, leveraging internal data, discarding unnecessary data, or sourcing from open resources (open data, GitHub repos, etc.).

If a process heavily relies on business requirements, it must be identified early on at this step.

→ Define success criteria: What is a correct answer? What are the key success factors? How can we establish a clear way to compute the ROI? These metrics should not be evaluated only when the project is done. They should be iterated continuously throughout the project to catch early mistakes, easy bugs, and inconsistencies.

Personal note: If the business asks you to build a RAG, one possible reason could be they’ve seen the competition doing it, (FOMO is a real thing, my friend). My advice to you in this situation is to challenge this request first. Very often, what the business needs can be less complex than we think.

2 — Understand what you’re indexing

After mapping and qualifying the data sources needed to build a RAG, you’ll quickly identify different kinds of modalities that will be indexed upon further discussion with the business.

• text

• images and diagrams

• tables

• code snippets

Each of these modalities will be processed differently, turned into vectors, and used in retrieval.

There are different approaches to combining multimodal data: here’s a common one that deals with text, tables, and images:

• text data is chunked and embedded with an embedding model

• tables are summarized with an LLM and their descriptions are embedded and used for indexing. After it’s retrieved, a table is used as-is, in its raw tabular format.

• code snippets are chunked in a special way (to avoid inconsistencies) and embedded using a text embedding model

• images are converted into embeddings using a multi-modal vision and language model — When retrieval is performed, the same multimodal LLM will take in your text query convert it to a vector and search related images in the vector db.

Additional notes:

• The vectors you index are not necessarily the vectors you retrieve: you can do things differently: For example, you can index sentences by the paragraphs they occur into, by the questions they answer, or by their summary. Check this post to learn more.

• There are a lot of great resources on multimodal RAGs. Check this blog from LlamaIndex for a deep dive.

3 — Improve chunk quality — garbage in, garbage out

Before indexing text data, you need to break it into smaller pieces called chunks. Chunks are the text snippets the vector database retrieves and passes to the LLM as a context to generate an answer.

So it’s obvious that the chunks' quality impacts the answer.

If the chunks:

• are pulled out of their surrounding context (if cut in the middle for example)

• include irrelevant data that doesn’t answer the question

• contradict each other

they’ll provide a poor answer.

There are different chunking techniques that you can explore here and here.

Other techniques depend on the data type (raw text, markdown, code). 

However, what works best in general is to analyze the data precisely and come up with a chunking rule that makes sense from a business perspective:

Here are some tips:

• Leverage document metadata like table of contents, titles, or headers to provide contextually relevant chunks. For example, a chunk that overlaps between an introduction and the first chapters is not consistent and rarely makes sense

• There’s no rule of thumb regarding chunk size. If your documents are wordy and express a single idea in relatively long paragraphs, the chunk size would be longer than documents that are written in bullet points

• Some data don’t need chunking because it’s relatively short and self-contained. Imagine building a RAG-based system on JIRA tickets: you won’t need chunking for that.

Note on semantic chunking: this method generates chunks that are semantically relevant. It’s however time consuming since it relies on embedding models underneath.

Resources about chunking:

• https://towardsdatascience.com/the-art-of-chunking-boosting-ai-performance-in-rag-architectures-acdbdb8bdc2b

4 — Improve pre-retrieval

Before your query reaches the RAG system, you can refine it with additional processing steps.

These steps are called pre-retrieval: they optimize the input query, ensuring the system retrieves the most relevant and high-quality documents.

Here are some key pre-retrieval steps to consider ⤵️

Read more

Hello everyone, Ahmed here. I write The Tech Buffet to deliver practical insights in machine learning and system design. This is based on my experience as an ML engineer, blogger and open source contributor.

In today’s issue, I explain my current motivations of selecting Weaviate as a production-ready vector database to power RAG systems.

Subscribe now

A vector database is a critical component in Retrieval Augmented Generation (RAG) applications: besides storing the vectors and the metadata, it efficiently computes similarity measures and powers semantic search.

But picking the right vector database when building production-ready RAG applications is no easy task: this requires a close evaluation of the cost, the query performance, and the scalability. This is even more important when you process millions of vectors or expect real-time use.

I'm currently working at a pharma company and I've gone through this decision process. So, in this issue, I'll share with you why I think Weaviate is a good option for managing your vector data and building RAGs, especially if you're a developer with no particular experience in LLMs.

To make this tutorial easy to follow, I'll first load some data from a public API, index it in Weaviate, and experiment with some of its cool features.

Here's the agenda

1. Load research papers from the Papers With Code public API

2. Setup Weaviate locally

3. Index the data in batches

4. Overview of Weaviate search functionalities:

   • vector search

   • keyword search

   • hybrid search

   • re-ranking

   • advanced filtering

5. Generative search: turning your database into a RAG system

1—Load research papers from the Papers With Code public API 🔌

To experiment with Weaviate features, we first need to ingest some data. Many datasets are available out there.

The one we'll experiment today with consists of research papers about Large Language Models.

We'll extract this dataset from the public API of paperswithcode.

To collect this dataset, you need to run the following function:

This dataset contains ~11K paper abstracts with the following attributes and metadata:

2—Setup Weaviate locally ⚙️

Weaviate is an open-source, AI-native vector database that helps developers create intuitive and reliable AI-powered applications.

You can use Weaviate in different ways:

• Via a docker-compose locally

• Via Kubernetes by deploying it on cluster

• Via Weaviate Cloud Services (WCS), the managed offering

At my company, we deployed Weaviate on Kubernetes but in this tutorial, we’ll use docker-compose for demo purposes.

The setup is straightforward.

1. Create an empty weaviate_data folder to store Weaviate data on the host

2. Create a docker-compose file with this configuration:

   • Enabled modules: text2vec-palm,reranker-cohere,generative-palm

   • AUTHENTICATION_APIKEY_ALLOWED_KEYS: 'admin'

   • AUTHENTICATION_APIKEY_USERS: 'ahmed'

3. Run this command:

   docker-compose up

To check that Weaviate started successfully, you can hit this URL

After launching Weaviate, you must create a client to interact with it.

This client can have additional credentials to connect to other third-party cloud services like OpenAI, VertexAI, or Cohere.

Why would a vector database need these connections?

Weaviate can take care of other things than storing your vectors and computing similarities.

• It can embed the data while indexing it

• It can perform document reranking after retrieval

• It can perform a RAG operation by answering questions based on search results

• and more…

In this tutorial, I'll connect Weaviate to VertexAI to power text embedding and generation and connect it to the Cohere API for the reranking task.

3—Index the data in batches 🗂️

Before indexing the data into Weaviate, we must create a collection and define the schema.

This is done via the create method that defines:

• The collection name: MLPapers

• The properties (or fields)

• The vectorizer that'll be used to embed the data while indexing it

• The generator that'll be used to generate answers based on retrieved documents (To perform RAG directly in Weaviate)

Here’s the full code:

After the collection is created, you can index the data in batches. This is another useful functionality of Weaviate that speeds up the indexing process.

Once the indexing is complete, we can check the number of vectors using the aggregate method:

collection.aggregate.over_all(total_count=True)
# AggregateReturn(properties={}, total_count=10592)

4—Weaviate search functionalities 🔍

Weaviate has multiple search options. We’ll review some of them in this section.

1. Vector Search

The obvious search method is based on vector similarities. Here's how to do it:

Behind the scenes, Weaviate first embeds the text query using the vectorizer, then fetches the similar results from the collection.

The response contains a list of objects and each one has the following structure.

By inspecting the three retrieved documents, we notice that they match the query accurately.

2. Keyword Search

Weaviate handles the keyword search by using the BM25 algorithm.

Keyword search is a simple technique that relies on string matching only.

You can run a keyword search in Weaviate by calling the bm25 method.

This gives the following results:

After reading these documents, they appear less relevant than the previous ones. They mention the words “fine-tuning” or “LLMs” but don’t necessarily discuss the process of model fine-tuning.

Keyword search is a useufl functionality when you need to match specific terms that an embedding model doesn’t necessarily know or represent well (e.g. a drug nomenclature or an e-commerce product name)

3. Hybrid Search

One feature that I find particularly useful in Weaviate is hybrid search.

Read more

Hi 👋 Ahmed here, your host at The Tech Buffet. This issue covers app deployment, an important skill to make your work accessible to the world.

Subscribe to access exclusive content.

Subscribe now

In this issue, you’ll learn how to deploy a Streamlit application quickly and cost-effectively on the Google Cloud Platform.

To do that, you’ll need:

• A GCP account with a valid credit card (although following this tutorial won’t cost you a cent)

• Pulumi: an Infrastructure as Code (IaC) tool to provision the cloud resources needed to deploy the app

• Some knowledge of Google Cloud services (Cloud Run), Docker and Poetry

• A Github account to trigger automatic deployment with the Cloud Build CI/CD platform

If you’re unfamiliar with some of these tools, everything will be covered below, and links will be provided to learn more.

Without further ado, let’s have a look 👇 .

What is Cloud Run? ☁️

Cloud Run is a Google Cloud service that provides a fully managed platform to run your code in a container.

Whether you’re building a web server, an API, or running processing jobs, Cloud Run automatically scales your containers up and down from zero.

This means you only pay when your code is running.

With Cloud Run, the workflow is simple. You write your code in your favorite programming language (Python, Go, NodeJS, or Java), package it into a container, and push it for deployment.

Then, Google Cloud provides everything the app needs to run in production.

What does an app need in production? Here are some benefits:

• Auto-scaling: Cloud Run scales up and down your containers depending on the load. When the CPU usage exceeds 60% in a container, Cloud Run starts another one and splits the traffic

• HTTPS 🔒: when your app is deployed on Cloud Run, it gets a secure endpoint with a TLS termination (even on a custom domain)

• Native integration with other GCP services: Cloud Run conveniently integrates with other services from the GCP suite. One of these services is Cloud Build, a CI/CD platform that streamlines the development process (we’ll see that in the last section)

• Traffic management: Cloud Run allows you to split your app’s traffic across multiple versions already deployed. This is particularly helpful for AB testing.

If you’re interested in learning more about Cloud Run, check out this link ⬇️.

Learn more about Cloud Run

Configure Pulumi and start an initial Deployment 🛠️

In the previous issue, I covered the process of deploying a cloud function on GCP using an infrastructure-as-code (IaC) tool called Pulumi.

With IaC, you don’t need to manipulate the UI. Instead, you define the resources in imperative code (e.g. in Python or TypeScript) and hit a special command to launch the deployment. This automates manual operations, versions the state of the infrastructure in a repo and increases collaboration.

We’ll use Pulumi again today to provision everything the Cloud Run instance needs.

Let’s go over some preliminary steps to Pulumi up and running.

⚙️ Initial setup

▶️ Install Docker and start it

▶️ Install Pulumi with Homebrew if you’re on Mac. (Check this link if you’re on another OS.)

brew install pulumi/tap/pulumi

▶️ Assuming you have a GCP account with the Cloud Run API activated, install the gcloud CLI if you haven’t already

▶️ Authenticate gcloud with your GCP account

gcloud auth application-default login

▶️ Create a new folder and run pulumi new inside. Then pick the container-gcp-python template. This will bootstrap a project architecture suited to Cloud Run.

In this step, you’ll be asked to provide some settings about the project. Then, Pulumi will create a virtual environment.

When the installation is completed, you’ll get the following structure:

▶️ Change the Pulumi.yaml file by specifying a backend on GCS. This will tell Pulumi to store the remote state of the architecture in your bucket*. (Mine is pulumi-gcp-state, so you need to create your own if you’re following the tutorial.)

PS: Note that other backends are possible to use.*

▶️ Pin the pulumi-gcp dependency version to 7.11.2

and reinstall the dependencies.

source venv/bin/activate
pip install -r requirements.txt

▶️ Now perform an initial deployment to check everything is working:

pulumi up

Pulumi first previews the updates:

Then it creates the resources:

If we click on the output URL, we’ll get this:

Our endpoint is deployed! For now, it’s a simple REST API that returns a cute “Hello, World!”.

Ok, let’s now modify the code a bit:

And run pulumi up again:

This time, Pulumi only previews the updates because everything else is already created.

If we hit the URL again, we will see our changes!

Pretty cool, right?

Deploy a Streamlit application 🎈

To deploy a Streamlit application on Cloud Run by using Pulumi, we’ll need to modify some files:

▶️ Change the requirements.txt in the app folder to:

streamlit

▶️ Change the app.py file to:

import streamlit as st


st.header("Hello from Cloud Run !")
st.balloons()

▶️ Update the Dockerfile by changing the base image and the entry point

▶️ Then deploy!

Ta-da! 🥁

What exactly happens here?

We looked at the deployment output without understanding how Pulumi does it under the hood.

To understand how Pulumi defines the resources, let’s have a look at the __main__.py file.

Let’s go over it step by step:

▶️ We start by loading the configuration settings:

▶️ We create a unique repo on Artifact Registry to store the Docker image

▶️ We create a container image for the service

▶️ We define the service by supplying the needed configurations: image URL, resource limits, ports, environment variables, etc.

▶️ Finally, we add the run.invoker role to the Cloud Run service and apply it to all users so that the app can be invoked by anyone

Pulumi integrates well with many Cloud providers and services, check out the documentation and the examples to learn more.

Don’t hesitate to check the documentations of the functions and classes used in the code: Pulumi provides a deep level of customization.

Automate the deployment with a CICD pipeline ⚙️

We’ve seen that to deploy the app, you needed to run pulumi up.

But this would be tedious to repeat each time we modify the source code, wouldn’t it?

Instead, it’s more convenient to trigger the deployment automatically as we push modifications to a remote repository.

That’s where Cloud Build comes into play.

Cloud Build is a serverless CI/CD platform designed to build, test, and deploy code.

The good news is that it also integrates perfectly well with Cloud Run!

Let’s see how.

▶️ Go to the GCP console, under Cloud Run, select the service you’ve just deployed, and hit the EDIT CONTINUOUS DEPLOYMENT button.

▶️ Set up the source repository (You may probably need to connect your Github or GitLab account to GCP before), and select the repository where you pushed your code

▶️ Update the build configuration: the branch that triggers the build, the build type (Dockerfile), and the path to the Dockerfile in the repository.

Before triggering the CI/CD, you may also need to activate the IAM API.

Let’s now see how modifications trigger automatic deployment:

▶️ Go to your project locally and make some modifications:

import streamlit as st

st.header("Hello from Cloud Run !")

button = st.button("Show balloons!")

if button:
    st.balloons()

▶️ Now push your code on the main branch. (This would be a feature branch in real life)

This will immediately trigger the build and the deployment.

And we can monitor it on the Cloud Build interface👇

After ~2 minutes the modifications are deployed.

How great is that?

Imaging doing this on a complex codebase: this would be a great time-saving.

Thanks for reading! 🙏

I hope you’ve learned a few things about Cloud Run today. To me, this is the go-to solution to deploy Streamlit applications quickly and cost-effectively.

I’ve been using Cloud Run to deploy generative AI applications, and it’s been working great for my client.

Maybe you should give it a try too: you could use Cloud Run to deploy RAG applications or Chatbots.

That’d be all for today. Until next time! 👋

Hello everyone 👋 Ahmed here. I’m the editor of The Tech Buffet, a newsletter that delivers practical insights on shipping production-ready ML applications.

Subscribe now

In this issue, you will learn how to automate the deployment of a serverless Cloud Function on Google Cloud.

This application will do something that I always wanted to prototype: summarizing YouTube videos 📹.

With the powerful LLMs that are currently available, the scalable cloud architectures, and the efficient developer tools, bringing this idea to life is now possible.

So here’s what we’ll cover to build this project:

• A Brief Introduction to Cloud Functions

• How to develop and test a Cloud Function locally

• How to design the infrastructure on GCP and deploy the Cloud Function on it

In the last step, we will use Python code only to provision the infrastructure and deploy the Cloud Function. We won’t be manipulating the GCP UI or using an Infrastructure as Code (IaC) tool that requires another syntax.

If you’re interested in DevOps or MLOps and how to ship ML applications at scale, it might be useful to stick to the end 👇

Let’s have a look.

What are Cloud Functions?

If you come from the AWS world, Cloud Functions are the equivalent of Lambda..

Cloud Functions is a Google Cloud service that allows you to run your code in the cloud with no servers or containers to manage and in a scalable and cost-efficient way.

That’s why Cloud Functions are called serverless.

With Cloud Functions, the developer experience is simple: you write and test your code locally then send it to Google Cloud to deploy and handle the operational infrastructure.

The second great feature of Cloud Functions is the cost: you only pay for what you use. Specifically, you’re billed for your function’s execution time, metered to 100 milliseconds. And when your function is idle, you pay nothing!

Cloud Functions are primarily designed to process short-lived, event-based actions triggered from other systems such as Cloud Storage or PubSub.

For example, you can build a Cloud Function that gets triggered when a file is saved under a bucket or when a row is inserted in BigQuery.

I also see Cloud Functions being used for sending notifications or triggering other systems.

Learn more about Cloud Functions

Develop and Test the Cloud Function locally

In this small project, the Cloud Function we’ll build will act as a REST API. It’ll receive user inputs consisting of YouTube URLs and send back the summary.

Before going through the deployment on GCP, let’s first build and test this function locally.

To summarize YouTube videos, we will need these Python libraries:

• youtube-transcript-api: to extract transcripts from a YouTube video

• langchain and langchain-openai: to interact with OpenAI LLMs and generate the summary

• python-dotenv: to load OpenAI credentials as environment variables

• functions-framework: a FaaS (Function as a Service) framework for writing portable Python functions and testing them locally

The simplest thing to get started is to create a virtual environment:

python -m venv ./venv
source venv/bin/activate
pip install -r function/requirements.txt

Then, the function’s code will be defined in function/main.py:

Here’s the main function that’ll summarize the video (for more details check the remaining code in the repo):

→ It first extracts the video transcript and the title, initializes an LLM, builds a prompt, and sends it to the LLM for summarization.

def summarize_youtube_video(url, additional_instructions):
    transcript = get_youtube_video_transcript(url)
    title = get_youtube_video_title(url)
    llm = OpenAI(api_key=os.environ.get("OPENAI_API_KEY"))
    prompt = get_prompt(title, transcript, additional_instructions)
    summary = llm.predict(prompt)
    data = {
        "url": url,
        "title": title,
        "summary": summary,
        "transcript": transcript,
    }
    return data

To wrap this logic in a REST API, we’ll use the functions_framework package. This allows us to define the Cloud Function handler (or entry point) that will be called once an HTTP request is sent:

@functions_framework.http
def main(request: flask.Request):
    if request.method == "POST":
        url = request.form.get("url")
        additional_instructions = request.form.get("instructions")
        data = summarize_youtube_video(url, additional_instructions)
        return flask.jsonify(data)
    else:
        return "Method Not Allowed", 400

The full Cloud Function code is available here.

To test the Cloud Function locally, run the following command:

functions-framework --target=main --source=function/main.py --debug

Running the command will start a local web server. So let’s open Postman to send it some requests:

Let’s send this video that discusses the differences between Terraform and Pulumi:

By filling in the form data with the URL and hitting the Send button, we get the following result: a JSON response with the following keys:

• summary

• title

• transcript (the full text)

• url

Here’s a closer look at the summary:

Terraform and Pulumi are two tools that are often compared in the world of DevOps. Terraform is known for its consistency in defining state, while Pulumi offers a more fluid approach using imperative programming languages such as Python and C#. However, this can also lead to potential issues when working in a team or switching organizations.\n\nTerraform, on the other hand, uses HCL (Hashicorp Configuration Language) to define state in a more limited and consistent manner. This allows for easier collaboration and a more secure way of managing state files. Additionally, Terraform is not a SaaS product, meaning your state is not automatically pushed to a remote network. This can be a concern when handling sensitive information.\n\nOverall, the speaker strongly prefers Terraform over Pulumi due to its simplicity and consistency. They also mention that Terraform has a community of nearly 2,000 members who can offer support and answer any questions. In the end, the speaker urges viewers to use Terraform in its simplest form and avoid overcomplicating things.

Pretty cool, right? Imagine doing this for 30-minute world news reports every day. This would be a huge time saving!

If you reproduce this tutorial, you can play with the Postman client, try different videos, and even add a custom instruction to the LLM.

In the following example, you can ask the LLM to answer in French.

The Cloud Function is tested locally.

We’re now ready to deploy.

Provision an infrastructure for the Cloud Function

If you asked me to deploy a Cloud Function a few years ago, I would happily log into the GCP console, put the source code into a bucket, create the Cloud Function from the UI, add a secret in Google Secret Manager, connect it to the app, and hit the deploy button.

If we were to perform these steps every time we make a change to the source code, this would be:

• tedious

• prone to errors

• untraceable in a codebase

• difficult to collaborate on

Enter Pulumi 🚀

Read more

Hey there 👋 Happy to have you here. Today’s issue is a practical end-to-end tutorial that’ll teach you how to build and deploy a personal assistant to digest the recent ML research.

Do you find it difficult to keep up with the latest ML research? Are you overwhelmed with the massive amount of papers about LLMs, vector databases, or RAGs?

In this issue, I will show how to build an AI assistant that mines this large amount of information easily. You’ll ask it your questions in natural language and it’ll answer according to relevant papers it finds on Papers With Code.

⚙️ On the backend side, this assistant will be powered with a Retrieval Augmented Generation (RAG) framework that relies on a scalable serverless vector database, an embedding model from VertexAI, and an LLM from OpenAI.

💻 On the front-end side, this assistant will be integrated into an interactive web application built with Streamlit and deployed on Google Cloud Run.

Every step of this process will be detailed below with an accompanying source code that you can reuse and adapt👇.

Before we start, here’s the global app workflow:

And here’s the agenda:

1. Collect data from Papers With Code

2. Create an index on Upstash

3. Embed the chunks and index them

4. Ask questions about the indexed papers

5. Build a Streamlit application

6. Deploy the Streamlit application on Google Cloud Run

PS: You can reuse every piece of code shared in this issue and adapt it to your personal or professional projects.

Ready? Let’s dive in 🔍.

1. Collect data from Papers With Code

Papers With Code (a.k.a PWC) is a free website for researchers and practitioners to find and follow the latest state-of-the-art ML papers, source code, and datasets.

It’s also possible to interact with PWC through an API to programmatically retrieve research papers. If you look at this Swagger UI, you can find all the available endpoints and try them out.

Let’s, for example, search papers on a specific keyword.

Here’s how to do it from the interface: you locate the papers/ endpoint, fill in the query ( q ) argument.

and hit the execute button.

Equivalently, you can perform this same search by hitting this URL.

The output response shows the first page of results only. The following pages are available by accessing the next key.

By exploiting this structure, we can retrieve 7200 papers matching “Large Language Models”. This can be done with a function that requests the URL and loops over all the pages.

Once the results are extracted, we convert them from their raw JSON format into LangChain Documents to simplify chunking and indexing.

Document objects have two parameters:

• page_content (str): to store the text of the paper’s abstract

• metadata (dict): to store additional information. In our use case, we’ll keep: id, arxiv_id, url_pdf, title, authors, published

Before embedding the documents, we need to chunk them into smaller pieces. This helps overcome LLMs’ limitations in terms of input tokens and provides fine-grained information per chunk.

After chunking the documents with a chunk_size of 1200 characters and a chunk_overlap of 200, we end up with over 11K splits.

2 — Create an index on Upstash

To store document embeddings (and metadata) somewhere, we first have to create an index.

In this tutorial, we’ll use Upstash, a serverless database that is very cost-efficient.

Read more

Hello everyone 👋 Ahmed here. I’m the author of The Tech Buffet, a newsletter that demystifies complex topics in programming and ML from an engineering perspective.

Consider subscribing to access exclusive content.

Subscribe now

In this issue, I list the previous articles I wrote on Retrieval Augmented Generation.

If you’re new to The Tech Buffet, this might be useful to catch up on the content you missed: explaining RAG mechanisms and unraveling their pitfalls, applying RAGs to SQL generation, getting RAGs to work well in production, different indexing and retrieval methods, etc.

These posts provide my perspective. After spending several months deploying RAGs in production for my clients, I came to know their ins and outs.

Expect this list to be regularly updated, serving as an evolving glossary for easy reference.

I hope you’ll find something that interests you in this list. If you have any questions regarding one of the issues, don’t hesitate to send me a message or drop a comment here. If you also have another topic in mind about RAGs, let me know.

In future issues, The Tech Buffet will cover practical tutorials on programming and DevOps. The goal is to help you build ML solutions (like RAGs) that scale in a production environment.

There’s already a backlog of posts, but if you’re interested in a particular topic, I’d love to hear your suggestions.

Until next time 👋

Ahmed

Subscribe now

Share

The Tech Buffet demystifies complex topics in programming and ML and delivers practical tips from the industry.

100% bulsh** free, 0% ChatGPT content.

Subscribe for the good stuff 👇

Subscribe now

Today’s issue will cover 3 powerful techniques that efficiently improve document retrieval in RAG-based applications.

1. Query expansion

2. Cross-encoder re-ranking

3. Embedding adaptors

By implementing them, you will retrieve more relevant documents that better align with the user query, thus making the generated answer more impactful.

1— Query expansion

Query expansion refers to a set of techniques that rephrase the original query.

We’ll look at two methods:

👉 Query expansion with a generated answer

Given an input query, this method first instructs an LLM to provide a hypothetical answer, whatever its correctness.

Then, the query and the generated answer are combined in a prompt and sent to the retrieval system.

This technique surprisingly works well. Check the findings of this paper to learn more about it.

The rationale behind this method is that we want to retrieve documents that look more like an answer. The correctness of the hypothetical answer doesn’t matter much because what we’re really interested in is its structure and formulation.

At best, you could consider the hypothetical answer as a template that helps identify a relevant neighborhood in the embedding space.

Here’s an example of a prompt I used to augment the query for a RAG that answers questions about financial reports.

You are a helpful expert financial research assistant. 

Provide an example answer to the given question, that might be found in a document like an annual report.

👉 Query expansion with multiple related questions

This second method instructs an LLM to generate N questions related to the original query and then sends them all (+ the original query) to the retrieval system.

By doing this, more documents will be retrieved from the vectorstore. However, some of them will be duplicates which is why you need to perform post-processing.

The idea behind this method is that you extend the initial query that may be incomplete and incorporate related aspects that can be eventually relevant and complementary.

Here’s a prompt I used to generate related questions:

You are a helpful expert financial research assistant. 
Your users are asking questions about an annual report.
Suggest up to five additional related questions to help them find the information they need, for the provided question.
Suggest only short questions without compound sentences. Suggest a variety of questions that cover different aspects of the topic.
Make sure they are complete questions, and that they are related to the original question.
Output one question per line. Do not number the questions."

The downside of this method is that we end up with a lot more documents that may distract the LLM from generating a useful answer.

That’s where re-ranking comes into play.

To learn more about different query expansion techniques, check this paper for Google.

2—Cross encoder re-ranking

This method re-ranks the retrieved documents according to a score that quantifies their relevancy with the input query.

To compute this score, we will use a cross-encoder.

A cross-encoder is a deep neural network that processes two input sequences together as a single input. This allows the model to directly compare and contrast the inputs, understanding their relationship in a more integrated and nuanced way.

Cross-encoders can be used for Information Retrieval: given a query, encode the query with all retrieved documents. Then, sort them in a decreasing order. The high-scored documents are the most relevant ones.

See SBERT.net Retrieve & Re-rank for more details.

Here’s how to quickly get started with re-ranking using cross-encoders:

• Install sentence-transformers:

  pip install -U sentence-transformers

• Import the cross-encoder and load it:

  from sentence_transformers import CrossEncoder
  cross_encoder = CrossEncoder('cross-encoder/ms-marco-MiniLM-L-6-v2')

• Score each pair of (query, document):

  pairs = [[query, doc] for doc in retrieved_documents]
  scores = cross_encoder.predict(pairs)
  print("Scores:")
  for score in scores:
      print(score)
  
  Scores:
  0.98693466
  2.644579
  -0.26802942
  -10.73159
  -7.7066045
  -5.6469955
  -4.297035
  -10.933233
  -7.0384283
  -7.3246956

• Reorder the documents:

  print("New Ordering:")
  for o in np.argsort(scores)[::-1]:
      print(o+1)

Cross-encoder re-ranking can be used with query expansion: after you generate multiple related questions and retrieve the corresponding documents (say you end up with M documents), you re-rank them and pick the top K (K < M).
That way, you reduce the context size while selecting the most important pieces.

In the next section, we’re going to dive into adaptors, a powerful yet simple-to-implement technique to scale embeddings to better align with the user’s task.

Read more

Hello everyone 👋 Ahmed here. I’m the author of The Tech Buffet, a newsletter that demystifies complex topics in programming and AI. Subscribe for exclusive content.

Subscribe now

2023 was, by far, the most prolific year in the history of NLP. This period saw the emergence of ChatGPT alongside numerous other Large Language Models, both open-source and proprietary.

At the same time, fine-tuning LLMs became way easier and the competition among cloud providers for the GenAI offering intensified significantly.

Interestingly, the demand for personalized and fully operational RAGs also skyrocketed across various industries, with each client eager to have their own tailored solution.

Speaking of this last point, creating fully functioning RAGs, in today’s issue we will discuss a paper that reviews the current state of the art of building those systems.

I started reading this piece during my vacation

and it’s a must.

It covers everything you need to know about the RAG framework and its limitations. It also lists modern techniques to boost its performance in retrieval, augmentation, and generation.

The ultimate goal behind these techniques is to make this framework ready for scalability and production use, especially for use cases and industries where answer quality matters *a lot*.

I won’t discuss everything in this paper, but here are the key ideas that, in my opinion, would make your RAG more efficient.

1—🗃️ Enhance the quality of indexed data

As the data we index determines the quality of the RAG’s answers, the first task is to curate it as much as possible before ingesting it. (Garbage in, garbage out still applies here)
You can do this by removing duplicate/redundant information, spotting irrelevant documents, and checking for fact accuracy (if possible).
If the maintainability of the RAG matters, you also need to add mechanisms to refresh outdated documents.

Cleaning the data is a step that is often neglected when building RAGs, as we usually tend to pour in all the documents we have without verifying their quality.

Here are some quick fixes that I suggest you go through:

• Remove text noise by cleaning special characters, weird encodings, unnecessary HTML tags… Remember that old NLP techniques using regex? You can reuse them.

• Spot document outliers that are irrelevant to the main topics and remove them. You can do this by implementing some topic extraction, dimensionality reduction techniques, and data visualization.

• Remove redundant documents by using similarity metrics

2—🛠️ Optimize index structure

When constructing your RAG, the chunk size is a key parameter. It determines the length of the documents we retrieve from the vector store.
A small chunk size might result in documents that miss some crucial information while a large chunk size can introduce irrelevant noise.

Coming up with the optimal chunk size is about finding the right balance.

How to do that efficiently? Trial and error.

However, this doesn’t mean that you have to make some random guesses and perform qualitative assessments for every experience.

You can find that optimal chunk size by running evaluations on a test set and computing metrics. LlamaIndex has interesting features to do this. You can read more about that in their blog.

3—🏷️ Add metadata

Incorporating metadata with the indexed vectors helps better structure them while improving search relevance.

Here are some scenarios where metadata is useful:

• If you search for items and recency is a criterion, you can sort over a date metadata

• If you search over scientific papers and you know in advance that the information you’re looking for is always located in a specific section, say the experiment section for example, you can add the article section as metadata for each chunk and filter on it to match experiments only

Metadata is useful because it brings an additional layer of structured search on top vector search.

4—↔️ Align the input query with the documents

LLMs and RAGs are powerful because they offer the flexibility of expressing your query in natural language, thus lowering the entry barrier for data exploration and more complex tasks. (Learn here how RAGs can help generate SQL code from plain English instructions).

Sometimes, however, a misalignment appears between the input query the user formulates in the form of a few words or a short sentence, and the indexed documents, which are often written in the form of longer sentences or even paragraphs.

Let’s go through an example to understand this.

Read more

Hello everyone 👋 Ahmed here. I’m the author of The Tech Buffet, a newsletter that demystifies complex topics in programming and AI. Subscribe for exclusive content.

Subscribe now

Today’s topic is evaluating your RAG without manually labeling test data.

Measuring the performance of your RAG is something you should care about especially if you’re building such systems and serving them in production.

Besides giving you a rough idea of how your application behaves, evaluating your RAG provides quantitative feedback that guides experimentations and the appropriate selection of parameters (LLMs, embedding models, chunk size, top K, etc.)

Evaluating your RAG is also important for your client or stakeholders because they are always expecting performance metrics to validate your project.

Here’s what this issue covers:

1. Automatically generating a synthetic test set from your RAG’s data

2. Overview of popular RAG metrics

3. Computing RAG metrics on the synthetic dataset using the Ragas package

PS*: The full code is accessible on a Colab notebook.

Get the notebook

PS**: To learn more about RAGs, you can check my previous articles here.

1—Generate a synthetic test set 🧪

Let’s say you’ve just built a RAG and now want to evaluate its performance.

To do that, you need an evaluation dataset that has the following columns:

• question (str): to evaluate the RAG on

• ground_truths (list): the reference (i.e. true) answers to the questions

• answer (str): the answers predicted by the RAG

• contexts (list): the list of relevant contexts the RAG used for each question to generate the answer

→ the first two columns represent ground-truth data and the last two columns represent the RAG predictions.

To build such a dataset, we first need to generate tuples of questions and corresponding answers.

Then, in the next step, we need to run the RAG over these questions to make the predictions.

👉 Generate questions and ground-truth answers (the theory)

To generate tuples of (question, answer), we first need to prepare the RAG data, split it into chunks, and embed it into a vector database.
Once the splits are created and embedded, we will instruct an LLM to generate N_q questions from N_c topics to finally obtain N_q x N_c tuples of questions and answers.

To generate questions and answers from a given context, we need to go through the following steps:

1. Sample a random split and use it as a root context

2. Fetch K similar contexts from the vector database

3. Concatenate the texts of the root context with its K neighbors to build a larger context

4. Use the large context and num_questions in the following prompt template to generate questions and answers

   """\
   Your task is to formulate exactly {num_questions} questions from given context and provide the answer to each one.
   
   End each question with a '?' character and then in a newline write the answer to that question using only 
   the context provided.
   Separate each question/answer pair by "XXX"
   Each question must start with "question:".
   Each answer must start with "answer:".
   
   The question must satisfy the rules given below:
   1.The question should make sense to humans even when read without the given context.
   2.The question should be fully answered from the given context.
   3.The question should be framed from a part of context that contains important information. It can also be from tables,code,etc.
   4.The answer to the question should not contain any links.
   5.The question should be of moderate difficulty.
   6.The question must be reasonable and must be understood and responded by humans.
   7.Do no use phrases like 'provided context',etc in the question
   8.Avoid framing question using word "and" that can be decomposed into more than one question.
   9.The question should not contain more than 10 words, make of use of abbreviation wherever possible.
       
   context: {context}
   """

5. Repeat steps 1 to 4 N_c times to vary the context and generate different questions

I’ve used this workflow to produce questions and answers on Python programming and here’s a sample of the results I got.

Pretty neat, right? no manual intervention.

👉 Python implementation 💻

Let’s first start by building a vectorstore that includes the data used by the RAG.

We can load that from Wikipedia (but if you have any interesting PDF on Python programming, you can load that too)

from langchain.document_loaders import WikipediaLoader

topic = "python programming"

wikipedia_loader = WikipediaLoader(
    query=topic,
    load_max_docs=1,
    doc_content_chars_max=100000,
)
docs = wikipedia_loader.load()
doc = docs[0]

After loading the data, we split it into chunks.

from langchain.text_splitter import RecursiveCharacterTextSplitter

CHUNK_SIZE = 512
CHUNK_OVERLAP = 128

splitter = RecursiveCharacterTextSplitter(
    chunk_size=CHUNK_SIZE,
    chunk_overlap=CHUNK_OVERLAP,
    separators=[". "],
)

splits = splitter.split_documents([doc])

Then, we create an index in Pinecone:

import pinecone

pinecone.init(
    api_key=os.environ.get("PINECONE_API_KEY"),
    environment=os.environ.get("PINECONE_ENV"),
)

index_name = topic.replace(" ", "-")
pinecone.init(
    api_key=os.environ.get("PINECONE_API_KEY"),
    environment=os.environ.get("PINECONE_ENV"),
)

if index_name in pinecone.list_indexes():
    pinecone.delete_index(index_name)

pinecone.create_index(index_name, dimension=768)

and use a LangChain wrapper to index the splits’ embeddings in it.

from langchain.vectorstores import Pinecone

docsearch = Pinecone.from_documents(
    splits,
    embedding_model,
    index_name=index_name,
)

Now comes the interesting part: generating the synthetic dataset.

To do that, we initialize an object from a TestsetGenerator class using an LLM, the document splits, an embedding model, and the name of the Pinecone index.

from langchain.embeddings import VertexAIEmbeddings
from langchain.llms import VertexAI
from testset_generator import TestsetGenerator

generator_llm = VertexAI(
    location="europe-west3",
    max_output_tokens=256,
    max_retries=20,
)
embedding_model = VertexAIEmbeddings()

testset_generator = TestsetGenerator(
    generator_llm=generator_llm,
    documents=splits,
    embedding_model=embedding_model,
    index_name=index_name,
    key="text",
)

Then, we call the generate method by passing two parameters:

synthetic_dataset = testset_generator.generate(
    num_contexts=10,
    num_questions_per_context=2,
)

Pretty simple, right?

If you’re interested in the implementation details, you can find them in the notebook.

We’re only halfway through it. Now, we need to use the RAG to predict the answers for each question and provide the context list used to ground the response.

The RAG is defined in a RAG class, so let’s initialize it first:

from rag import RAG

rag = RAG(
    index_name,
    "text-bison",
    embedding_model,
    "text",
)

Then, we iterate over the synthetic dataset by calling the predict method over each question and collect the predictions.

rag_answers = []
contexts = []

for i, row in synthetic_dataset.iterrows():
    question = row["question"]
    prediction = rag.predict(question)

    rag_answer = prediction["answer"]
    rag_answers.append(rag_answer)
    source_documents = prediction["source_documents"]
    contexts.append([s.page_content for s in source_documents])

synthetic_dataset_rag = synthetic_dataset.copy()
synthetic_dataset_rag["answer"] = rag_answers
synthetic_dataset_rag["contexts"] = contexts

and here’s what the final result looks like:

🏆 Congratulations if you’ve made it this far, now you’re ready to evaluate your RAG.

2—Popular RAG metrics 📊

Before jumping into the code, let’s cover the four basic metrics we’ll use to evaluate our RAG.

Each metric examines a different facet. Therefore, when evaluating your application, it is crucial to consider multiple metrics for a comprehensive perspective.

1. Answer Relevancy:
   The answer relevance metric is designed to evaluate the pertinence of the generated answer in relation to the provided prompt. Answers that lack completeness or include redundant information receive lower scores. This metric utilizes both the question and the answer, generating values between 0 and 1. Higher scores signify better relevance.
   
   Example

   ❓Question: What are the key features of a healthy diet?

   ⬇️ Low relevance answer: A healthy diet is important for overall well-being.

   ⬆️ High relevance answer: A healthy diet should include a variety of fruits, vegetables, whole grains, lean proteins, and dairy products, providing essential nutrients for optimal health.
   

2. Faithfulness

   This metric evaluates the factual consistency of the generated answer within the provided context. The calculation involves both the answer and the retrieved context, with the answer scaled to a range between 0 and 1, where higher values indicate better faithfulness.

   For an answer to be deemed faithful, all claims made in the response must be inferable from the given context.
   

   Example:
   ❓Question: What are the main accomplishments of Marie Curie?
   📑 Context: Marie Curie (1867–1934) was a pioneering physicist and chemist, the first woman to win a Nobel Prize and the only woman to win Nobel Prizes in two different fields.

   ⬆️ High faithfulness answer: Marie Curie won Nobel Prizes in both physics and chemistry, making her the first woman to achieve this feat.

   ⬇️ Low faithfulness answer: Marie Curie won Nobel Prizes only in physics.
   

3. Context precision
   Context Precision is a metric that evaluates whether all of the ground-truth relevant items present in the contexts are ranked higher or not. Ideally, all the relevant chunks must appear at the top ranks. This metric is computed using question and the contexts, with values ranging between 0 and 1, where higher scores indicate better precision.
   

4. Answer correctness

   This metric measures the accuracy of the generated answer in comparison to the ground truth. This assessment utilizes both the ground truth and the answer, assigning scores within the 0 to 1 range. A higher score indicates a more accurate alignment between the generated answer and the ground truth, indicative of superior correctness.
   

   Example:
   🟢 Ground Truth: The Eiffel Tower was completed in 1889 Paris, France.

   ⬆️ High answer correctness: The construction of the Eiffel Tower concluded in 1889 in Paris, France.

   ⬇️ Low answer correctness: Completed in 1889, the Eiffel Tower stands in London, England.

3—Evaluate RAGs with RAGAS 📏

To evaluate the RAG and compute the four metrics, we can use Ragas.

Ragas (for Rag Assessment), is a framework that helps you evaluate your RAG pipelines.

It also provides a bunch of metrics and utility functions to generate synthetic datasets.

To run Ragas over our dataset, you first need to import the metrics and convert the dataframe of synthetic data to a Dataset object.

from datasets import Dataset
from ragas.llms import LangchainLLM
from ragas.metrics import (
    answer_correctness,
    answer_relevancy,
    answer_similarity,
    context_precision,
    context_recall,
    context_relevancy,
    faithfulness,
)

synthetic_ds_rag = Dataset.from_pandas(synthetic_dataset_rag)

Then we need to configure Ragas to use VertexAI LLMs and embeddings.

This step is important since Ragas is configured to use OpenAI by default.

metrics = [
    answer_relevancy,
    context_precision,
    faithfulness,
    answer_correctness,
    answer_similarity,
]

for m in metrics:
    m.__setattr__("llm", ragas_vertexai_llm)
    if hasattr(m, "embeddings"):
        m.__setattr__("embeddings", vertexai_embeddings)

answer_correctness.faithfulness = faithfulness
answer_correctness.answer_similarity = answer_similarity

Finally, we call the evaluate function on the synthetic dataset and specify the metrics we want to compute:

from ragas import evaluate

results_rag = evaluate(
    synthetic_ds_rag,
    metrics=[
        answer_relevancy,
        context_precision,
        faithfulness,
        answer_correctness,
    ],
)

Once the evaluation is done, you can print the result directly.

or you can convert it to a dataframe to inspect each of these metrics for every question.

Conclusion

Generating a synthetic dataset to evaluate your RAG is a good start, especially when you don’t have access to labeled data.

However, this solution also comes with its problems.

Some of the generated answers:

• May lack diversity

• Are redundant

• Are plain rephrasing of the original text and need more complexity to reflect real questions that need reasoning

• Can be too generic (especially in very technical domains)

To tackle these issues, you can adjust and tune your prompts, filter irrelevant questions, create synthetic questions on specific topics, and use Ragas as well for dataset generation.

What to read on Substack?

The Kaitchup – AI on a Budget

Mixtral-8x7B: Understanding and Running the Sparse Mixture of Experts by Mistral AI

Mistral AI released this week their new LLM: mistralai/Mixtral-8x7B-v0.1 (Apache 2.0 license) Mixtral-8x7B is a sparse mixture of 8 expert models. In total, it contains 46.7B parameters and occupies 96.8 GB on the hard drive. Yet, thanks to this architecture, Mixtral-8x7B can efficiently run on consumer hardware. Inference with Mixtral-8x7B is indeed sign…

Read more

3 years ago · 9 likes · Benjamin Marie, PhD

Ahead of AI

Understanding Large Language Models

Note: Next to the monthly Ahead of AI series that discusses the latest research and trends, I plan to post some additional articles related to machine learning and AI once in a while. I am curious to hear what you think — do you like this idea, or should I stick to the main newsletter series? Please let me know in the comments…

Read more

3 years ago · 592 likes · 27 comments · Sebastian Raschka, PhD

Share The Tech Buffet

Hello 👋 this is Ahmed, your host at The Tech Buffet. If you’re new to this newsletter, it’s your go-to place for ML engineering insights, coding best practices and tool reviews.
You’ll also see a lot of content about LLMs here. No hype, no BS. I promise.

Subscribe now

This issue is about a tool I started using recently to track the performance of RAG-based applications. It’s called TruLens.

TruLens is an open-source package that helps you evaluate and monitor your LLM applications.

It comes particularly in handy if you build RAGs and struggle to find their optimal parameters.

To help with this, TruLens tracks your RAG experiments, compares them with custom metrics, diagnoses errors, and allows you to land on an optimal configuration.

In this issue, we’ll see how this works for a simple and practical use case: Answering questions about Albert Einstein given a short text about him.

This is a practical tutorial that you can follow step by step. (it’s also best read on a wide screen like a laptop or a tablet).

Let’s dive in.

Disclaimer: I’m not affiliated with TruLens and this is not a sponsored post.

This issue is broken into 5 sections.

1️⃣ Setup

To prepare the environment and the data needed to build the RAG, we must go over some preliminary steps.

• We first install these libraries:

!pip install trulens_eval==0.18.1 chromadb==0.4.18 openai==1.3.1

• We set an OpenAI key as an environment variable to be able to use GPT-3.5 as LLM.

import os
os.environ["OPENAI_API_KEY"] = <key>

👉 You can also plug in any open-source LLM from HuggingFace. Learn more about that here.

• We define a simple text about Albert Einstein to build our RAG on:

einstein_info = """
Albert Einstein (/ˈaɪnstaɪn/ EYEN-styne;[4] German: [ˈalbɛɐt ˈʔaɪnʃtaɪn] ⓘ; 14 March 1879 – 18 April 1955) 
was a German-born theoretical physicist who is widely held to be one of the greatest and most influential
scientists of all time. Best known for developing the theory of relativity, Einstein also made important 
contributions to quantum mechanics, and was thus a central figure in the revolutionary reshaping of the 
scientific understanding of nature that modern physics accomplished in the first decades of the twentieth century.[1][5] 
His mass–energy equivalence formula E = mc2, which arises from relativity theory, has been called "the world's 
most famous equation".[6] He received the 1921 Nobel Prize in Physics "for his services to theoretical physics, 
and especially for his discovery of the law of the photoelectric effect",[7] a pivotal step in the development 
of quantum theory. His work is also known for its influence on the philosophy of science.[8][9] In a 1999 poll 
of 130 leading physicists worldwide by the British journal Physics World, Einstein was ranked the greatest 
physicist of all time.[10] His intellectual achievements and originality have made the word Einstein 
broadly synonymous with genius.[11]
"""

• And finally, we create a chromadb vector store in memory to store the embedding of the previous text.

2️⃣ Build a custom RAG with instrumentation

In this step, we define a custom RAG in a Python class and add TruLens instrumentation to each of its methods.

TruLens provides different instrumentation frameworks to allow you to inspect and evaluate the internals of your application and its associated records. This helps you track a wide variety of usage metrics and metadata along with the inputs and outputs of the application.

Let’s first look at the RAG implementation.

We will define three methods:

1. retrieve: to extract relevant texts as context from the vector store given an input query. (we have one text in our chromadb since this is a simple demo, but you get the idea)

2. generate_completion: to generate an answer given the query and the retrieved context.

3. query: to combine the two previous methods and provide the RAG’s final answer.
   

For each of these methods, we will add the @instrument decorator to track usage metrics and analyze the input and output.

3️⃣ Create feedback functions

Now that the RAG is created, we need to define the feedback functions that evaluate it.

Feedback functions score the output of the LLM application by analyzing the generated text.

TruLens provides different feedback functions designed for different purposes, but in our case, we’ll only use the three following ones:

✅ Response relevance: to measure the relationship of the final answer to the input question
✅ Context relevance: to measure the relationship between the provided context and the input question.
✅ Groundedness: to check if the answer is grounded in its supplied context

You can learn more about TruLens feedback functions here.

Back to code.

To define the three feedback functions, we first need an LLM. This one can be different from the one we used in the RAG to generate the answer. (We can, for example, pick GPT-4 for evaluation tasks since it’s bigger than GPT-3.5 and shows better reasoning capabilities).

from trulens_eval.feedback.provider.openai import OpenAI as fOpenAI

fopenai = fOpenAI()

Each feedback function is then defined separately.

👉 Question / Answer relevance between the overall question (the query argument) and answer (the output):

from trulens_eval import Feedback, Select
from trulens_eval.feedback import Groundedness

f_qa_relevance = (
    Feedback(
        fopenai.relevance_with_cot_reasons,
        name="Answer Relevance",
    )
    .on(Select.RecordCalls.retrieve.args.query)
    .on_output()
)

⚠️ Notice that the input question is referenced here as the query argument of the retrieve method defined in the RAG class. If you named this method fetch for example, the feedback function would be written as following:

f_qa_relevance = (
    Feedback(
        fopenai.relevance_with_cot_reasons,
        name="Answer Relevance",
    )
    .on(Select.RecordCalls.fetch.args.query)
    .on_output()
)

👉 Question / Statement relevance between the question (the query argument) and each context chunk:

f_context_relevance = (
    Feedback(
        fopenai.qs_relevance_with_cot_reasons,
        name="Context Relevance",
    )
    .on(Select.RecordCalls.retrieve.args.query)
    .on(Select.RecordCalls.retrieve.rets.collect())
    .aggregate(np.mean)
)

In this example, the context is defined as the returned value of the retrieve method, or simply, with the rets argument.

👉Groundedness:

grounded = Groundedness(groundedness_provider=fopenai)

# Define a groundedness feedback function
f_groundedness = (
    Feedback(
        grounded.groundedness_measure_with_cot_reasons,
        name="Groundedness",
    )
    .on(Select.RecordCalls.retrieve.rets.collect())
    .on_output()
    .aggregate(grounded.grounded_statements_aggregator)
)

Here’s the full code 👇

Now, we wrap the custom RAG with TruCustomApp and add the list of feedback functions for evaluation.

from trulens_eval import TruCustomApp

tru_rag = TruCustomApp(
    rag,
    app_id="RAG-einstein:v1",
    feedbacks=[
        f_groundedness,
        f_qa_relevance,
        f_context_relevance,
    ],
)

4️⃣ Run the RAG and track metrics

We’re ready to run some questions over the RAG.

To make sure instrumentation is enabled, you must call the query method with the tru_rag context manager.

with tru_rag as recording:
    rag.query("Who is Albert Einstein?")

Upon running this query, feedback functions and usage metrics are computed. You can visualize them by calling the get_leaderboard method.

tru.get_leaderboard(app_ids=["RAG-einstein:v1"])

We got 1 for every feedback metric. No surprise, our question was easy.

Let’s try another question that goes a bit beyond the provided context.

with tru_rag as recording:
    rag.query("Did Einstein contribue to the development of the atomic bomb?")

If we display the metrics again, we notice a drop in the average groundedness and the context relevance. That’s normal because the context we’re retrieving for the second query is irrelevant to the question and the answer.

5️⃣ Visualize metrics and metadata in the TruLens dashboard

How about diving into the RAG answers individually to diagnose precisely what happened and understand why metrics were computed as such?

This is where it’s useful to run the run_dashboard method.

If you visit the provided URL, you’ll see a Streamlit app that summarizes your RAG (RAG-einstein:v1).

In the first tab, you’ll see a leaderboard with global averaged metrics:

In the evaluations tab, you’ll get more interesting details: the feedback scores for each query and the corresponding LLM’s answer and reasoning.

Let’s monitor what happened for the second query:

• We can first see the RAG’s answer:

Then, we can go over each feedback metric to understand why it got its score:

• 👉 Answer relevance =1
  reason: The response directly addresses the prompt by stating that Einstein did contribute to the development of the atomic bomb. It provides specific details about his indirect contribution through his scientific work and theories, as well as his involvement in urging the United States to develop the bomb.

• 👉 Context Relevance = 0.2
  reason: The statement provides information about Albert Einstein's contributions to science, including his work on the theory of relativity and quantum mechanics. It also mentions his Nobel Prize in Physics for his discovery of the law of the photoelectric effect. However, it does not specifically mention his contribution to the development of the atomic bomb.

• 👉 Groundednes = 0

Statement Sentence: Yes, Einstein did contribute to the development of the atomic bomb.,

Supporting Evidence: NOTHING FOUND

Score: 0

Statement Sentence: Although he did not directly work on the Manhattan Project, which was the U.S. government's secret research project to develop atomic weapons during World War II, his scientific work and theories, particularly his equation E=mc2, laid the foundation for understanding nuclear energy and the potential for atomic weapons.,

Supporting Evidence: NOTHING FOUND

Score: 0

Statement Sentence: Einstein signed a letter to President Franklin D. Roosevelt in 1939, urging the United States to develop an atomic bomb before Nazi Germany could.,

Supporting Evidence: NOTHING FOUND

Score: 0

Pretty neat, right?

Conclusion

TruLens is an interesting tool to monitor and diagnose LLM’s behavior. It personally saved me a lot of hassle going through evaluations myself.

Additionally, it provides a useful dashboard that can be deployed and shared with colleagues and stakeholders.

If you found the RAG example we went through quite simple, note that TruLens also integrates with LangChain and LlamIndex when building more complex workflows.

That being said, it’d be all for me today.

Thanks for reading 🙏

Ahmed

Hello everyone 👋, Ahmed here. Welcome to The Tech Buffet, where I share practical insights to build industry-ready ML (and LLM) applications.

Subscribe now

Tuning your RAG to get optimal performance takes time as this depends on various interdependent parameters: chunk size, overlap, top K retrieved docs, embedding models, LLM, etc.

The best combination often depends on your data and your use case: you can’t simply plug in the settings you used in the last project and hope for the same results.

Most people don’t address this issue properly and pick parameters almost randomly. While some are comfortable with this approach, I decided to tackle the problem numerically.

Here’s where evaluating your RAG comes in.

In this issue, I’ll show you a quick method you can follow to efficiently and quickly evaluate your RAGs across two tasks:

1. Retrieval

2. Generation

By mastering this evaluation pipeline, you can iterate, perform multiple experiments, compare them with metrics, and hopefully land on the best configuration

Let’s see how this works 👇.

1—Create a synthetic dataset

Evaluating an LLM often requires annotating a test set manually. This takes time, requires domain expertise, and is prone to human errors.

Hopefully, LLMs can help us with this task.

Sample N chunks from your data. For each chunk, instruct an LLM to generate K tuples of questions and answers.
After the generation is complete, you will obtain a dataset of N*K tuples, each one having (question, answer, context).

Ps: The context here is the original paragraph and its metadata

Here’s the prompt I used in one of my projects:

Create exactly {num_questions} questions using the context and make sure each question doesn't reference
terms like "this study", "this research", or anything that's not available to the reader.

End each question with a '?' character and then in a newline write the answer to that question using only 
the context provided.
Separate each question/answer pair by "XXX"
Each question must start with "question:".
Each answer must start with "answer:".
CONTEXT = {context}

2—Run your RAG over each synthetic question

Once the dataset is built, you can use your RAG to perform predictions on each question.

This will generate answers backed by a set of retrieved sources.

For each question, make sure you list the retrieved documents: this will later be useful for evaluation.

3—Compute two evaluation metrics

Now, you can compute two scores for each question:

1. A retrieval score to assess the relevance of the retrieved documents
   This score can be binary (1/0 for each prediction) and tells whether the true source for each question is in the list of the retrieved sources.

2. A quality score to evaluate the generated answer given the question and the ground-truth answer. Again, an LLM can be used in this task to produce such evaluation over 5.

   
   Here is the prompt I used:

   Your job is to rate the quality of a generated answer
   given a query  and a reference answer.
   
   QUERY = {query}
   
   GENERATED ANSWER = {generated_answer}
   
   REFERENCE ANSWER = {reference_answer}
   
   Your score has to be between 1 and 5.
   You must return your response in a line with only the score.
   Do not return answers in any other format.
   On a separate line provide your reasoning for the score as well

After obtaining these scores for each question, average them over the dataset to obtain the final two metrics.

Resource:

• https://www.anyscale.com/blog/a-comprehensive-guide-for-building-rag-based-llm-applications-part-1#cold-start

Interested to learn more about RAGs? Feel free to check my previous issues:

• Why Your RAG is Not Reliable in Production

• Getting a RAG To Work Well Is Hard - 5 Blog Posts To Become a RAG Master

Get to know other newsletters I like on this platform 👇

The Kaitchup – AI on a Budget

Weekly news, tips, and tutorials on fine-tuning, running, and serving large language models on your computer. Each tutorial is published along with a notebook ready to run.

By Benjamin Marie, PhD

Ahead of AI

Ahead AI specializes in Machine Learning & AI research and is read by tens of thousands of researchers and practitioners who want to stay ahead in the ever-evolving field.

By Sebastian Raschka, PhD

This was a short issue for today. Happy Sunday! 👋

Hello there 👋🏻, this is Ahmed. I write The Tech Buffet to share weekly practical tips on improving ML systems. Consider subscribing to access exclusive content.

Subscribe now

Hello everyone,

In this issue, I’m providing you with a list of interesting reads about RAGs.

This list doesn’t cover introductory concepts nor does it showcase simple demos of 5-line Langchain snippets.

Rather, it discusses the limitations of RAG-based systems, offers valuable insights on deploying them in production, and shares a startup’s experience applying RAGs to the insurance sector.

This curated list is the result of thorough documentation I conducted while building conversational agents for the pharmaceutical industry.

I hope you find it useful to you or your teams.

Get ready to become a RAG master. ♜

Building RAG-based LLM Applications for Production

🔗 Link
👨‍💻 Authors: Goku Mohandas and Philipp Moritz

Disclaimer: One of the authors (Goku Mohandas) is an excellent technical writer and open-source developer. He’s particularly famous for his website (madewithml.com) that taught thousands of developers to productionize ML systems.

This post is a production guide for building RAG-based LLM applications.

It’s by far the best piece I’ve read on this topic. I’m currently using it as a guide to improve my existing RAG-based apps. If you too are getting serious about RAGs, this post is a must-read.

This tutorial is also well-written, detailed, and easy to follow. Plus, the corresponding code is available on Github.

This is a long read but here’s what you’ll learn if you go through it:

• 💻 Develop a retrieval augmented generation (RAG) LLM app from scratch.

• 🚀 Scale the major workloads (load, chunk, embed, index, serve, etc.) across multiple workers.

• ✅ Evaluate different configurations of our application to optimize for both per-component (ex. retrieval_score) and overall performance (quality_score).

• 🔀 Implement an LLM hybrid routing approach to bridge the gap b/w OSS and closed-source LLMs.

• 📦 Serve the application in a highly scalable and available manner.

• 💥 Share the 1st-order and 2nd-order impacts LLM applications have had on products and organizations

🙌 What did I like the most about this post?
While reading this guide, I learned about a powerful method to evaluate RAGs. It consists of building a synthetic evaluation dataset and then computing two metrics: one for evaluating the retrieval quality and another one for the generation quality.
I applied this evaluation framework to my current RAG-based projects and this helped me quickly iterate on my experiments: this was useful for determining the optimal parameters (chunk size, number of retrieved docs, embedding models, etc) instead of picking them randomly.

Learn more about RAG evaluation here.

Disadvantages of RAG

🔗 Link
👨‍💻 Author: Kelvin Lu

If you’re new to RAGs, it’s important to be aware of the limitations these systems hide. This will help you diagnose potential errors and manage unrealistic expectations.

In this timely post, the author provides a refreshing and brutally honest take on RAG's disadvantages. More specifically, he first discusses the semantic limitations of RAGs. Then, he emphasizes the lossy nature of their underlying processes, from chunking and embedding to retrieval and response generation.
Interestingly, he also explains why certain types of search like multi-hop Q&A are not suited to RAGs.

With these limitations in mind, the author suggests that hybrid LLMs combined with external knowledge bases like graph databases are the way to achieve harder-to-reach goals.

🙌 What did I like the most about this post?
Similar to any machine learning system, RAGs come with inherent limitations. Amid the excitement surrounding LLM applications, it becomes crucial to gain a clear understanding of their potential. This post aims to provide precisely that insight.

10 Ways to Improve the Performance of Retrieval Augmented Generation Systems

🔗 Link
👨‍💻 Author: Matt Ambrogi

After addressing some of RAG’s shortcomings, it’s useful to explore practical tips to improve their performance.

In this post, the author outlines 10 strategies to implement to bridge the gap between getting RAGs to work and getting them to work very well.

Here’s a one-sentence summary for each tip:

1. Inspect and clean the data your RAG relies on to ground an answer

2. Explore different index types: embedding-based, key-word-based and hybrid

3. The chunk size is a hyper-parameter that must be tuned properly

4. Tweak the system prompt to customize your RAG, control its answers, and avoid hallucinations

5. Retrieval is not limited to the k nearest neighbors: you can leverage chunks’ metadata for additional filtering

6. Create specialized indexes (each one for a specific task) and route your query to one of them

7. Rerank the retrieved documents based on relevance

8. Transform your query to help the LLM provide a better answer

9. Fine-tune the embedding model to align it further with your context

10. Debug LLM by using developer tools like Rivet

Some of these tips are easy to implement directly while others are already integrated in LlamaIndex.

🙌 What did I like the most about this post?
In a 9-minute read, the author condenses a wealth of insights that would typically take any ML practitioner weeks (if not months) of experimenting to uncover.

Gen AI: why does simple Retrieval Augmented Generation (RAG) not work for insurance?

🔗 Link
👨‍💻 Authors: Damien Philippon, Christophe Bourguignat

In this article, the authors, who are the founders of a French startup operating in the insurance sector, claim that RAG, on its own, doesn’t fully satisfy consumer expectations or meet regulations.

They explain this through multiple reasons, the most interesting one being: the inability of RAGs to understand the structure of insurance documents.

In fact, in their default setting, RAGs cannot accurately answer questions like “Am I covered if a storm damages the trees in my garden?” because this question needs to be addressed from multiple angles, each one backed with its related policy in a different section of a +70-page contract.

A generic RAG would answer this question incorrectly because it retrieves a quick mention of the trees somewhere on page 12 (‘your trees and ground plantations’ if they were planted at least two years before the disaster.’) without connecting this information with the other conditions of the policy that are on page 11, 66, 13, 52, etc.

More juicy details in the post. 👆

🙌 What did I like the most about this post?
It’s uncommon to get tangible and honest feedback from companies, especially when evaluating over-hyped technologies. This post provides a fresh perspective that anyone applying RAGs can benefit from.

Azure Cognitive Search: Outperforming vector search with hybrid retrieval and ranking capabilities

🔗 Link
👨‍💻 Author: Alec Berntson

This post comes from Microsoft’s tech blog and presents a new method to improve retrieval in RAG applications. This method uses hybrid retrieval + semantic ranking.

The author claims that it’s the most effective approach for improved relevance out of the box.

👉 Hybrid retrieval (L1): Performs both keyword and vector retrieval and applies a fusion step to select the best results from each technique.

👉 Semantic ranking is performed by Azure Cognitive Search's L2 ranker which utilizes multi-lingual, deep learning models adapted from Microsoft Bing. The Semantic ranker can rank the top 50 results from the L1.

Retrieval comparison using Azure Cognitive Search in various retrieval modes on customer and academic benchmarks. See §6.1 How we generated the numbers in this post and §6.2 Search and Dataset configuration for Table 1 for the setup and measurement details.  

🙌 What did I like the most about this post?
Instead of applying the same retrieval method over and over, this post provides a new approach that beats the existing state-of-the-art.

Thanks for reading 🙏

This is a new format I’m experimenting with in the newsletter. Please tell me what you think. (was it helpful? boring? too short?)

Anywho, I hope you learned one thing or two and explored the articles I shared with you.

Have a good day and until next time! 👋

Ahmed

Hello there 👋🏻, I’m Ahmed. I write The Tech Buffet to share weekly practical tips on improving ML systems. Join the family (+600 🎉) and subscribe!

Subscribe now

In this issue, we’ll explore 3 different ways you can index your data when building RAGs.

These methods are worth experimenting with as they can boost the performance and accuracy.

Let’s see how.

Reminder on data indexing in typical RAGs

In the default implementation of a RAG system, documents are first split into chunks.

Then, each chunk is embedded and indexed into a vector database.

In the retrieval step, the input query is also embedded and the most similar chunks are extracted.

In this setup, the data (i.e. the chunks) we retrieve is the same as the data we index.

This is the most natural and intuitive implementation.

🚨 However, this doesn’t have to be always the case.

We can index the chunks differently to increase the performance of such systems

1—Index chunks by their subparts 🧩

Instead of indexing the whole chunk directly, we can split it again into smaller pieces (e.g. sentences) and index it with those multiple times.

Why is this useful?

Imagine dealing with long and complex chunks that discuss multiple topics or conflicting information. Using them in a typical RAG will likely generate noisy outputs with some irrelevant content.

If we separate these chunks into smaller sentences, each sentence will likely have a clear and well-defined topic that matches the user query more accurately.

When retrieving the chunk, this method makes sure to get a relevant context to the query and a broader context (not present in the indexing sentence) that will be useful for the LLM to generate a comprehensive answer.

2—Index chunks by the questions ❓︎they answer

Instead of indexing the chunks directly, we can instruct the LLM to generate the questions they answer and use them for indexing. This is a simple approach.

When submitting the query, the RAG will therefore compare it to the most relevant questions that the data answers. Then, based on these questions, it will retrieve the corresponding chunks.

This indexing method is useful because it aligns the user's query/objective with the core content of the data.

If a user doesn’t formulate a very clear question, such an indexing method can reduce ambiguity. Instead of trying to figure out what chunks are relevant to the user’s question, we directly map it to existing questions that we know we have an answer for.

3—Index chunks by their summaries 📝

This indexing method is similar to the previous one. It uses the chunk summary for indexing instead of the questions it answers.

This is typically useful when the chunks have redundant information or irrelevant details that are not useful to the user’s query.

It also captures the essence of the information that the user is looking for.

That’s it for today!

I hope you’ve enjoyed this issue and learned something useful about improving RAG indexing.

If you know about other indexing methods that improve RAG systems, please tell me about them (in the comments or via a reply to this email).

Interested in building RAGs for your company? Have a look at my next issue where I cover a curated list of top articles from the industry.

Until next time! 👋

Ahmed

Hello again 👋 it’s Ahmed. I write The Tech Buffet to share practical tips and tutorials helping you build industrial-grade ML applications.
Subscribe for exclusive content 👇

Subscribe now

If you use Python, you probably use a linter or a code formatter that checks the quality of your codebase.

This enforces industry standards and normalizes the code quality across the developers of the same team.

I’ve personally always used Black, isort, and Flake8 in large codebases.

In this issue, I want to share a promising alternative that is 10 to 100 times faster than those tools with additional benefits.

It’s called Ruff

What is Ruff?

Ruff is an extremely fast Python linter and code formatter, written in Rust.

If you don’t know what Rust is, it is a multi-paradigm, general-purpose programming language that emphasizes performance, type safety, and concurrency.

The main selling point of Ruff is its speed. It is insanely fast. In large codebases, this makes a huge difference by speeding up the developer’s workflow.

Multiple benchmarks have been conducted (I read this one on Reddit) and all agree that this tool is 10-100x faster than existing linters (like Flake8) and formatters (like Black).

You should consider moving to Ruff

Ruff has taken the Python community by storm and is now becoming the go-to solution that combines linting and formatting in the same place.

Here are its main features:

• 🐍 It can be installed with pip or poetry

• 🛠️ It provides support for pyproject.toml

• 🤝 It’s compatible with modern versions of Python (3.12 at this time of writing)

• ⚖️ It’s 99% compatible with Black and 30x faster

• 📦 It has a caching mechanism and reanalyzes unchanged files only

• 🔧 It spots linting errors like traditional linters and fixes them too

• 📏 It has over 700 built-in rules

• ⌨️ It integrates well with the developer’s tool (Jetbrain, Vscode, etc.)

• 🌎 It’s mono-repo-friendly

Ruff is under active development and is supported by a large community of developers.

It’s also used in major open-source projects:

Getting started

You can install Ruff with pip (or any other package manager):

pip install ruff

And you can start using it for lining or formatting from the terminal.

• To run Ruff as a linter, you can use one of these commands:

If Ruff detects linting errors, you can ask it to fix them with this command:

ruff check . --fix

• To run Ruff as a formatter, you can use one of these commands:

You can also use Ruff in a pre-commit hook via ruff-pre-commit.

Not familiar with pre-commit hooks? Check out this issue when I cover them in details.

Configuration

To determine the appropriate settings for each Python file, Ruff looks for the first pyproject.toml, ruff.toml, or .ruff.toml file in the file's directory or any parent directory.

To configure Ruff, let's create a pyproject.toml file in our project's root directory. In this example, we’ll a specific rule named line-too-long and defined by the E501 code.

[project]
# Support Python 3.10+.
requires-python = ">=3.10"

[tool.ruff]
# Set the maximum line length to 79.
line-length = 79

[tool.ruff.lint]
# Add the `line-too-long` rule to the enforced rule set.
extend-select = ["E501"]

Running Ruff again, we see that it now enforces a maximum line width, with a limit of 79:

❯ ruff check .
numbers/numbers.py:5:80: E501 Line too long (90 > 79)
Found 1 error.

For a full enumeration of the supported settings, see Settings.

If you want a detailed tutorial on setting up Ruff for your project, have a look a this link.

Resources:

• https://github.com/astral-sh/ruff

• https://www.reddit.com/r/Python/comments/16ig4wu/ruff_format_new_tool_to_format_python_files_that/

• https://docs.astral.sh/ruff/tutorial/

• https://blog.jerrycodes.com/ruff-the-python-linter/

This was a quick issue! Thanks for reading. 👋

Share

Hello, this is Ahmed! 👋

This issue is part of a Python series dedicated to sharing tips and best practices to improve your code.
If you’re interested in this type of content (+ the additional coding material)
consider subscribing.

Subscribe now

Python decorators are powerful tools that help you produce clean, reusable, and maintainable code. They provide interesting functionalities such as caching, automatic retry, rate limiting, logging, or turning your classes into supercharged data containers.

I’ve long waited to learn about these abstractions and now that I’ve acquired a solid understanding, I’m writing this practical guide to help you, too, grasp the concepts behind decorators.

No big intros or lengthy theoretical definitions today.

This issue is rather a documented list of 12 helpful decorators I regularly use in my projects to extend my code with extra functionalities.
We’ll dive into each decorator, look at the code, and play with some hands-on examples.

If you’re a Python developer, this post will extend your toolbox with helpful scripts to increase your productivity and avoid code duplication.

Less talk, I suggest we jump into the code now 💻.

Share

1 — @logger (to get started)✏️

If you’re new to decorators, you can think of them as functions that take functions as input and extend their functionalities without altering their primary purpose.

Let’s start with a simple decorator that extends a function by logging when it starts and ends executing.

The result of the function being decorated would look like this:

To write this decroator, you first have to pick an appropriate name: let’s call it logger.

logger is a function that takes a function as input and returns a function as output. The output function is usually an extended version of the input. In our case, we want the output function to surround the call of the input function with start and end statements.

Since we don’t know what arguments the input function uses, we can pass them from the wrapper function using *args and **kwargs. These expressions allow passing an arbitrary number of positional and keyword arguments.

Here’s a simple implementation of the logger decorator:

Now you can apply logger to some_function or any other function for that matter.

decorated_function = logger(some_function)

Python provides a more Pythonic syntax for this, it uses the @ symbol.

11 decorators left to review. Let’s move to the next one.

Read more

Hey there 👋 this is Ahmed. With The Tech Buffet, I share digestible tips and concepts to help you industrialize stable ML and LLM applications.

This issue is one of a kind. It covers hallucinations 🤡.

Not exactly these hallucinations, but the ones Large Language Models face.

If you’ve ever played with ChatGPT, Bard, Claude, or even one of their open-source brothers (Llama2 or Mistral 7B) you’ve probably noticed these AI systems produce misleading or incorrect information.

This is what the AI community agreed to define as hallucinations.

In this issue, I explain why hallucinations happen, how you can quantitatively evaluate them, and go over the different strategies to mitigate them.

Let’s dig in 🔍

Why do LLMs hallucinate?

When talking about hallucination in the context of LLMs, it’s important to keep in mind that this phenomenon happens not because these systems can “imagine”, “perceive” or “dream”.

First, it's essential to grasp that LLMs are pre-trained to predict tokens. They lack a concept of true or false, instead relying on probabilities for text generation. While this approach enables some unexpected reasoning capabilities, like passing the BAR exam or USMLE, it's a result of their probabilistic, token-by-token reasoning.

Additional training steps, such as instruct tuning and RLHF in modern LLMs, introduce some "bias towards factuality" but don't alter the fundamental underlying mechanism and its limitations.

LLMs draw knowledge from vast sources like the internet, books, Q&A databases, and Wikipedia. Their responses tend to be biased towards what they've encountered most frequently. If you ask a medical question without careful prompting, you might receive an answer aligned with established medical literature or random forum discussions.

Summing up this in two sentences: LLMs hallucinate because they lack reasoning and understanding capabilities. They spit text by extrapolating a prompt and struggle to cite their sources.

How to quantify LLM hallucinations?

Evaluating the LLM hallucinations with a number is important when conducting iterative experiments.

This is where FActScore comes in handy.

FActScore is a modern metric that serves for both human and model-based assessments.

It dissects an LLM's generation into "atomic facts," and the overall score is calculated as the sum of the accuracy of these atomic facts, each holding equal importance.

Accuracy, in this context, is a straightforward binary measure indicating whether the atomic fact is supported by the source. To determine this metric, the authors employ various automation techniques that leverage LLMs and vector stores.

Here’s how the FActScore works in practice. 👇

You first instruct your LLM to generate content. Let’s say, a biography of Bridget Moynahan.

“Bridget Moynahan is an American actress, model and producer. She is best known for her roles in Grey’s Anatomy, I, Robot and Blue Bloods. She studied acting at the American Academy of Dramatic Arts, and …”

Next, you split this text into atomic facts.

• Bridget Moynahan is American.

• Bridget Moynahan is an actress.

• Bridget Moynahan is a model.

• Bridget Moynahan is a producer.

• She is best known for her roles in Grey’s Anatomy

Then, you verify each fact with an external source (e.g. Wikipedia). Think RAG here.

If the fact is supported by the source, this accounts for 1, otherwise it’s 0.

• Bridget Moynahan is American. ✅ → 1

• Bridget Moynahan is an actress. ✅ → 1

• Bridget Moynahan is a model. ✅ → 1

• Bridget Moynahan is a producer.❌ → 0

• She is best known for her roles in Grey’s Anatomy ❌ → 0

The final score is therefore (1+1+1)/ 5 = 0.6

Here’s the full example from the original paper👇

How to mitigate hallucinations?

This is a complex problem that you can address from multiple angles.

👉 Product-Level Recommendations:

Hallucinations in Language Models (LLMs) can be a challenge, but with the right strategies, you can minimize their impact. Here are some product-level recommendations to help you navigate this.

These recommendations challenge the way you intend to apply and interact with LLMs.

1. Purposeful Design: Start by structuring your use case to naturally reduce hallucination risks. For instance, for content generation applications, focus on opinion pieces over factual articles.

2. Empower User Editability: Allow users to fine-tune AI-generated content. This not only enhances scrutiny but also bolsters content reliability.

3. User Accountability: Clearly communicate that users bear responsibility for the content generated and published.

4. Citations and References: Implement a feature that includes citations, providing a safety net for users to verify information before sharing.

5. Operational Flexibility: Offer various operational modes, such as a "precision" mode that utilizes a more accurate model, albeit at a higher computational cost.

6. User Feedback Loop: Create a feedback mechanism for users to report inaccuracies, harmful content, or incompleteness. This user data is invaluable for refining the model in future iterations.

7. Output Control: Mindfully manage the length and complexity of generated responses. Longer and more complex outputs have a higher potential for hallucinations.

8. Structured Input/Output: Consider implementing structured fields rather than free-form text, particularly useful in applications like resume generation. Predefined fields for educational background, work experience, and skills can be a game-changer.

Addressing hallucinations is crucial for LLMs' responsible use. Implementing these recommendations can enhance user experiences and content reliability.

👉 General prompting guidelines

It’s no secret that adjusting your prompts dramatically improves the answers of your LLM.

Here are some principles I applied that proved to be effective:

• Assertive Tone: Improve model compliance with ALL CAPS and highlight key directives.

• Context Matters: Provide more background information to anchor the model's responses. For example, attributing a “role” to the LLM generally helps direct the generated text toward a desired response.

• Refinement Steps: Review the initial output and make necessary adjustments.

• Inline Citations: Ask the model to back its claims with references.

• Selective Grounding: Identify situations where grounding is essential versus optional.

• Reiteration: Emphasize key instructions at the end of prompts.

• Echo Input: Request the model to recap essential input details for alignment.

• Algorithmic Filtering: Utilize algorithms to prioritize relevant information.

👉 Chain Of Thoughts (CoT)

The concept of a 'Chain of Thought' was first introduced in the paper titled 'Chain-of-Thought Prompting Elicits Reasoning in Large Language Models' by researchers at Google.

The underlying principle is straightforward: since Large Language Models (LLMs) are primarily trained to predict tokens rather than engage in explicit reasoning, you can encourage reasoning by providing explicit step-by-step instructions.

Here's a simple example from the original paper:

There’s another method to apply CoT called zero-shot CoT where you simply ask the LLM to “think step by step”.

👉 Grounding the answer with RAG

RAG is the acronym for Retrieval Augmented Generation.

The RAG framework is a simple yet powerful framework to mitigate hallucinations. Its primary purpose is to augment an LLM with a factual and verified context drawn from an external database.

Here’s how RAG works:

It first takes an input question and retrieves relevant documents to it from an external database. Then, it passes those chunks as a context in a prompt to help an LLM generate an augmented answer.

That’s basically saying:

“Hey LLM, here’s my question, and here are some pieces of text to help you understand the problem. Give me an answer.”

Hallucinations are what prevent most companies nowadays from deploying LLM-based solutions to production.

To control this issue, it’s important to

• Understand that this phenomenon is inherent to the way LLMs are trained. Hallucinations happen from pure statistics, not intrinsic cognitive abilities

• Be able to quantitatively evaluate it to conduct experiments efficiently

• Mitigate hallucinations with a multi-faceted strategy

I hope this issue addressed these 3 points in the most comprehensive way.

That’d be all for me today.

Until next time 👋

Today, we’ll explore an open-source no-code tool that helps you build and visualize LLM applications.

If you spend time designing RAGs, chatbots, or sophisticated agents, this tool will speed up your prototyping.

↳ Here’s the agenda 📋

• A quick intro of the solution

• Setup and configuration

• Demo: Create a chatbot over PDFs

• My honest opinion

Flowise: an interface for building LLMs

The LangChain framework made it very easy to build LLM applications.

By abstracting objects like chains, retrievers, or document loaders, it gave us the power to compose complex workflows that solve interesting tasks: chatbots over documents, semantic search engines, personal assistants, etc.

Flowise went further to ease this process by providing an interactive UI on top of LangChain.

By using Flowise, it’s now possible to build the same chains directly from the browser by selecting the right components and connecting them.

It does look good, doesn’t it? That’s the power of LangChainJS combined with TypeScript and JavaScript.

Setup and configuration

To start Flowise on your computer you can either install it from source code or use it in a Docker container. (I recommend the second option)

↳ Option 1— Install it from source

To install Flowise, make sure you have NodeJS >= 18.15.0 set up on your computer and run this command:

npm install -g flowise

After it’s installed, you can start Flowise with this command:

npx flowise start

You can also specify the username and password as arguments:

npx flowise start --FLOWISE_USERNAME=user --FLOWISE_PASSWORD=1234

The app is now accessible on http://localhost:3000

↳ Option 2— Run Flowise in a Docker container

To run Flowise using Docker, go to the Docker folder at the root of the project.

Then copy the .env.example file, paste it into the same location, and rename it to .env.

Run this Docker command to start the Flowise server:

docker-compose up -d

With that, the app is accessible on the same URL: http://localhost:3000

Ok this is cool, but show me the app now! 🤯

When you open Flowise, you’ll first see a screen with the ChatFlows you’ve already created (a ChatFlow is a workflow you build with Flowise)

To create a ChatFlow that represents an LLM chain, click on Add New, then click on the plus sign on the left.

This will open up a list of nodes that you can drag and drop to the interface and connect them with each other.

If you’re already familiar with LangChain, these nodes correspond to the same components you already use:

• Document Loaders

• Embeddings

• Prompts

• Retrievers

• …

If you click on Document Loaders for example, you’ll find many options that you can pick depending on your data format (CSV, PDF, Markdown, etc.)

If you click on Embeddings, you’ll find different embedding providers as well. You can find for example AWS, GCP, Azure, OpenAI and HuggingFace

As for the vector stores, you also have a lot of options (Chroma, Weaviate, Pinecone …)

With Flowise, you can either feed your vector store directly in a chatflow or you can load an existing one that you created elsewhere. That’s up to you.

Demo: Create a chatbot over PDFs

Here’s a common use case: your company asks you to build a chatbot over internal PDF documents (contracts? invoices? research papers?) to gather insights and test LLM capabilities.

Building this prototype in Flowise is easier than ever.

You first start by importing the PDF loader.

Your PDF is quite long so you need to split it into chunks. Therefore, you need to import a Character Text Splitter. You can specify the separator (\n), the chunk size (500), and the chunk overlap (150).

This text splitter must be connected to the PDF file loader.

The chunked documents must now be inserted in a vector store. Let’s import one.

In this demo, we’ll use Chroma.

As you see, the Chroma database is missing the embedding model so let’s add that.

I’m going to pick one from OpenAI.

Since we need an API key to perform calls to OpenAI, we can add and save these credentials in Flowise. (They’re encrypted)

Now the last part: we need to create a Conversational Retrieval QA chain that takes as inputs a chat model and the Chroma database.

So let’s go ahead and add those.

Here’s the final flow. The first time I made it, it took me probably 3 minutes.

That’s pretty much it.

In this demo, I’m going to chat with the first chapter of this book: Effective Python: 90 Specific Ways to Write Better Python

To do this, you first need to upload the book directly to the PDF loader.

To chat with it, you simply need to click on the purple icon (on the right) to open an interactive chat.

Here’s an example question:

How awesome is that?

Notice that Flowise can also display the items the LLM used when generating the answer. This helps diagnose the results.

and this doesn’t stop, here comes the best part:

Once your chatbot is built and tested, you can have access to an endpoint to interact with it programmatically and embed it in other applications.

It’s as simple as writing these lines of code.

Before I move to the last section of this post, I wanted to mention that this demo is built locally. In a real setting though, you’d need to deploy Flowise on the cloud so that teammates can easily collaborate on the same ChatFlow.

Check this doc to learn more about the deployment options.

My honest opinion

Flowise is a powerful tool that makes prototyping chains very easy. It provides a nice UI to assemble blocks together and quickly test them.

I also like the ability to embed the ChatFlow by using the provided endpoint.

As this solution is relatively new, there are some improvements that could be made.

• An improved stacktrace to diagnose bugs

• Displaying a status to the user when a heavy workload (e.g. embedding) is being performed or when a large file is being ingested

• A more detailed documentation

Will I use this solution?

In the short term, absolutely.

However, my commitment in the long run is less certain. This hesitancy is rooted in the fact that Flowise is closely intertwined with LangChain's core components. Although LangChain is a great package, I occasionally find its architecture overly complex. At times, constructing chains or RAGs from the ground up seems more practical. Hence, I might consider exploring alternative frameworks in the future.

Perhaps, if the community someday reaches a consensus on a standardized approach for representing LLMs, solutions like Flowise could enjoy broader adoption.

That’s all for today. I hope you discovered a tool that would make your life easier for building and prototyping.

Thanks for reading,

Until next time! 👋

Ahmed

Hello again! 👋

This issue is part of a Python series dedicated to sharing tips and best practices to improve your code.
If you’re interested in this type of content (+ the additional coding material)
consider subscribing.

Subscribe now

As a data scientist, I manipulate paths and files on a daily basis to read and write data.

To do this, I typically use the os.path Python module to perform operations such as joining paths, checking the content of a directory, or creating folders.

In this issue, I’m challenging this practice by introducing another path management library called Pathlib.

We’ll see how this library works, how it differs from the os.path module, what features and benefits it provides, and when you should (or not) use it.

If you’re a Python developer, learning about Pathlib will make your code lighter, more flexible, and easier to write and maintain.

Let’s dig in.

Read more

Hello there 👋🏻, I’m Ahmed. I write The Tech Buffet to share weekly practical tips on productionizing ML systems. Join the family (+400 🎉) and subscribe!

Subscribe now

With the rise of LLMs, the Retrieval Augmented Generation (RAG) framework also gained popularity by making it possible to build question-answering systems over data.

We’ve all seen those demos of chatbots conversing with PDFs or emails.

While these systems are certainly impressive, they might not be reliable in production without tweaking and experimentation.

In this issue, we explore the problems behind the RAG framework and go over some tips to improve its performance.

These findings are based on my experience as an ML engineer who’s still learning about this tech.

RAG in a nutshell ⚙️

Let's get the basics right first.

Here’s how RAG works:

It first takes an input question and retrieves relevant documents to it from an external database. Then, it passes those chunks as a context in a prompt to help an LLM generate an augmented answer.

That’s basically saying:

“Hey LLM, here’s my question, and here are some pieces of text to help you understand the problem. Give me an answer.”

You should not be fooled by the simplicity of this diagram.

In fact, RAG hides a certain complexity and involves the following components behind the scenes:

• Loaders to parse external data in different formats: PDFs, websites, Doc files, etc.

• Splitters to chunk the raw data into smaller pieces of text

• An embedding model to convert the chunks into vectors

• A vector database to store the vectors and query them

• A prompt to combine the question and the retrieved documents

• An LLM to generate the answer

If you like diagrams, here’s another for you.

A tad more complex but it illustrates the indexing and retrieval processes.

Phew!

Don’t worry though, you can still prototype your RAG very quickly.

Frameworks like LangChain abstracted most of the steps involved in building a RAG and it became easy to prototype those systems.

How easy is that? 5-line-of-code easy.

Of course, there’s always a problem with the apparent simplicity of such code snippets: depending on your use case, they don’t always work as-is and need very careful tuning.

They’re great as quickstarts but they’re certainly not a reliable solution for an industrialized application.

The problems with RAG

If you start building RAG systems with little to no tuning, you may be surprised.

Here’s what I noticed during the first weeks of using LangChain and building RAGs.

Read more