I did not start with that sentence. I started with a project, and the sentence is what I had left after it.

The project is agent-harness-ops, and the trilogy I just finished on AI That Ships is the writeup. Not a tutorial I invented and then coded up. The other way round: I built the thing, hit every wall in it, and the three articles are the walls, in the order I hit them.

So this post is both. The code, and the writing it produced.

The project

DevVoice turns a GitHub README into a reviewed X thread, LinkedIn post, and dev.to article. Five agents: an extractor that pulls grounded facts from the source, three platform writers, and a reviewer that fact-checks the drafts back against the original. A human approves before anything is final.

Stack: FastAPI, Celery, Redis, PostgreSQL, LangChain DeepAgents, Langfuse for tracing, React and nginx on the front. Model providers are swappable, with Ollama as the default so it costs nothing to run.

It lives on two branches, and the split maps onto the articles.

master, the local version The harness and the backend. Clone it, copy .env.example, docker compose up --build, and it runs on your laptop. Parts 1 and 2 are this branch, explained.

aws-deployment, the AWS version The same application plus everything needed to run it in the cloud. Terraform for VPC, ECS Fargate, RDS, ElastiCache, ALB, IAM, Secrets Manager, CloudWatch. GitHub Actions over OIDC, so no AWS credentials are stored anywhere. Bedrock through the task role, so no model API key exists either. Two environments from one folder using Terraform workspaces. make aws-setup once, git push after that. Part 3 is this branch, explained.

Run master first. Read aws-deployment when you want the second half.

Part 1: System Design for Agent Systems

Building the five agents taught me that almost none of the engineering is in the model. It is in the harness around it.

What the article covers: tools, prompts, memory, orchestration, and where a human sits in the loop. Then the evaluation framework, because none of this is testable the way ordinary code is.

What I got wrong first, and fixed:

• I reached for parallel agents because they sound faster. They are not. They create a coordination problem. Sequential subagents won on clear data flow, early exit, and debuggability.

• I let a dynamic value drift above the static block in my prompts. Nothing errored. The cache hit rate went to zero and the bill quietly doubled.

• I gated on importance. Everything looks important, so the gate fired constantly and I learned to click approve without reading. Gate on irreversibility instead.

• I trusted the model to decide it was finished. That is the least reliable stop condition there is. Add a step budget, a wall-clock budget, and a token budget. You own three of the four exits.

Read Part 1

Part 2: Designing the Backend for Agent Systems

The first version of DevVoice was one synchronous endpoint that ran the agent and returned 90 seconds later. It is the natural thing to write and it is wrong in six ways, each of which becomes an incident on a different day.

What the article covers: the eight layers under the harness. Identity, API, queue, workers, the model-call seam, tool execution, state and sessions, delivery.

What building it taught me:

• POST /jobs returns 202 in milliseconds. Persist first, then enqueue. Reverse that and a worker picks up a job the store has never heard of.

• Idempotency is not optional. In a normal API a duplicate submit is untidy. Here it is a second bill.

• Intermediate states are the entire UX. “running” for 90 seconds reads as broken. extracting → drafting → reviewing reads as working.

• I sized workers with web-service reflexes and left roughly 10x on the floor. An agent worker is blocked on a socket 70 to 95% of its life.

• One file imports the provider SDK. Caching, tier routing, retries, circuit breaking and fallback all live behind that seam. Routing by task took a run from $0.300 to $0.087.

• Retry the transport, never the reasoning.

Read Part 2

Part 3: Deployment of Agent Systems to AWS ECS

Then I put it on AWS, and found a category of bug that cannot exist on a laptop.

What the article covers: twelve steps from laptop to production, Terraform describing the infrastructure, GitHub Actions doing the deploying, dev and prod. No prior AWS experience assumed.

What the deployment taught me:

• Three containers on Fargate: frontend, api, worker. Only the frontend is reachable from the internet. The worker has no address at all.

• Give the worker roughly double the API’s memory and CPU. It holds an entire agent run for the life of the job.

• One Terraform folder, two workspaces. Never copy the folder per environment. Copies drift, and the drift is always found during an incident.

• Restricting an agent in the prompt is a request. Restricting it in the task role is a fact.

• A green pipeline is not proof. I had a deployment that looked perfectly healthy and could not call the model, because nothing before the first real job ever tries.

• One model passed every quick test, then in the real pipeline listed the same directory forty times without converging. Test with an end-to-end job, not a single call.

Read Part 3

What survived

Swap the model. Swap the framework. Swap what the agent does entirely. All three parts hold, because their shape follows from the workload rather than the model: slow, I/O-bound, non-deterministic, priced per unit.

Those four properties are not going anywhere. Design for them from the first commit and deployment is configuration. Design for the web shape and deployment is a rewrite.

The articles are free to read, no membership needed. The code is at github.com/kmeanskaran/agent-harness-ops: master to run it locally, aws-deployment to put it on AWS.

Follow me on X: @kmeanskaran

They solve 500 LeetCode problems, build five Gen AI chatbots, collect ten certificates, fine-tune Qwen models and still hear, “We’ll get back to you.”

After interviewing with startups and MNCs, getting rejected, receiving offers, conducting interviews and sitting through countless interview rounds, I realized something:

Companies don’t hire the most knowledgeable candidate. They hire the candidate who reduces their risk.

This guide is everything I wish someone had told me before my first interview.

1. How HIRING actually works

This the most common interview process for ML Engineering (When I say ML this applies to AI as well, in later section we will discuss about the core differences).

General MLE interview pipeline

1. Resume: HRs (non-tech) match your resume with the job description, while technical reviewers evaluate whether your skills align with the team’s current technical requirements and product needs.

2. Recruiter Screen: A short screening call to verify your experience, discuss your background, notice period, salary expectations, and occasionally ask a few basic technical questions.

3. Technical Round: Coding interviews covering DSA, machine learning fundamentals, project discussions, and sometimes take-home assignments or implementing ML algorithms from scratch.

4. System Design rounds for ML: You’ll be asked to design an end-to-end ML system, discuss your take-home assignment, or solve production-scale ML, Agentic AI, and distributed systems problems (especially for mid-to-senior roles).

5. Hiring Manger: A discussion about your experience, role expectations, compensation, joining timeline, and how well you fit the team.

6. Culture/CEO/Founder: Evaluates your communication, attitude, ownership, and overall alignment with the company’s culture and values.

This article covers what you can do on the technical side of the interview to get shortlisted, crack the interviews, and land the job.

2. Start with Projects (even you are senior)

The most important stage of preparation is knowing where to start. Honestly, there is a lot of noise on the internet. Some people talk about tools, some fake career stories, some genuinely get great opportunities at a young age, and some get placed in SF startups.

But you are sitting in front of your laptop, dreaming about their success stories. I know this should be a technical article, but what matters most is having the right mindset and clarity.

I hope you already have the fundamentals of AI/ML

Stage 1

To get your first job, you need to build SERIOUS BORING PROJECTS.

Let me be clear. If you’re looking for a job, it’s not necessary to build a product with real users. If you can, that’s a cherry on top. But here, I’m covering what works for everyone.

A boring project means stopping the habit of building fancy projects with weak engineering. An Agentic AI chatbot that audits documents but has no queue workers or caching, or a VLM that solves sign language but has high latency, isn’t enough.

You need to understand:

• The ML lifecycle

• How orchestration works

• How to design the backend for AI projects

• How to use caching and improve latency

A project like forecasting stock prices with LSTMs sounds boring today. But what if you add Agentic AI to generate reports? What if you can train multiple stocks with around 40% less compute?

around this idea, with a complete engineering pipeline deployed on AWS.I built

Stock-Agent-Ops

Simply build two end-to-end projects: one focused on LLM fine-tuning or traditional ML, and another on Agentic AI. Ask Claude or ChatGPT to simulate a real engineering pipeline, then build and deploy it on AWS, GCP, or Azure.

Stage 2

In this stage, improve your production, collaboration, and soft skills, especially if you’re already employed.

Candidates with these skills are valued the most because technical skills can always be learned, but communication and project ownership are essential.

Stage 2 becomes much more important in the later stages of interviews.

3. Resume Creation

We’re in 2026, and a resume is just a formal document that showcases your professional experience. It does not define your worth.

LinkedIn, Glassdoor, and Indeed are highly saturated. Recruiters usually post a job, review the profiles of early applicants first, and shortlist from that pool. If they don’t find a suitable candidate, they move on to the remaining applicants.

ATS optimization was a bigger concern a few years ago. Today, your focus should be on creating a resume that clearly showcases your technical skills, projects, and achievements. Learn how to write effective cold DMs and invest in personal branding. Over time, these will create more opportunities than simply clicking Easy Apply.

This article isn’t about resume preparation, but you can use my resume as an example.

4. Technical Round (Most Important)

A. DSA

The most common question is:

“Is DSA important?”

My answer is:

“Yes, but maybe not in the future.”

The real question is how much DSA do you need, and who’s interviewing you?

As of July 2026, almost all MNCs and around 40% of startups ask DSA questions (based on my experience). If you’re targeting large tech companies, then yes, DSA is important.

But there’s a catch.

Recruiters know AI can generate code. Live coding is no longer just about getting the correct answer. They want to understand how you think, communicate, and approach the problem.

In some of my interviews, I couldn’t fully solve the problem, but I still moved to the next round because of my approach.

For most ML engineering roles, focus on:

• Arrays

• Strings

• Patterns

• Two Pointers

• Sliding Window

• Linked Lists (important)

• Trees (important)

• Graphs (rare)

• Dynamic Programming (rare for most ML roles)

Master the first five topics well. They cover most Python data structure questions. Don’t spend months grinding every DSA topic.

B. ML Coding Rounds

The second most important area is ML coding.

Some research-focused roles may ask you to implement ML algorithms from scratch or solve tensor-level problems.

Personally, I’ve rarely been asked these questions.

If you’re applying for AI, ML, or MLOps engineering roles, these rounds are much less common. Always align your preparation with the role and its expectations.

C. Take-Home Assignments (The Future of Hiring)

Many startups, and even some MNCs, now prefer take-home assignments.

These could be a prototype, MVP, or a series of engineering tasks.

You have complete freedom to choose your tools, architecture, and implementation. Yes, you can use AI, and recruiters generally expect that.

The twist comes during the interview.

The interviewer is usually a senior engineer, and you can’t bluff your way through. They’ll ask questions about every design decision, trade-off, and implementation detail. An experienced engineer can quickly tell if you don’t understand your own project.

This is one of the best opportunities to make an impression.

Personally, I enjoy take-home assignments because they let me think about the system from a business and engineering perspective instead of solving isolated coding problems.

Remember, the interviewer wants to understand your technical thinking and whether you can fit into the existing engineering pipeline. Your reasoning matters far more than the buzzwords you use.

I’ve made the mistake of throwing around terms like AI, Celery, Kafka, and distributed systems just to sound impressive.

Don’t do that.

D. Unexpected Questions

The market is unpredictable, and many companies don’t even know exactly what they’re looking for.

Some may ask about SQL, backend development, statistics, APIs, or distributed systems.

Don’t try to learn everything.

Instead, shortlist the companies you want to apply to, study their products, and carefully read their job descriptions.

You’ll get a very good idea of what they’re actually looking for, and your preparation will become much more focused.

See this diagram.

5. ML System Design & MLOps

now we’re at the most important part of the process. This is the stage that can single-handedly land you in a pool of opportunities.

AI is becoming smarter. Companies are no longer struggling to write code for agents or neural networks.

We now have Kimi, Fable, Ops, GPT-sol, GLM, Grok, DeepSeek, and many more. Anyone can use these models to build a prototype or even a development-ready application.

But the real challenge is serving that AI application to real users. Maybe 10,000+ users per second.

You can refer to this post for learning resources. The resources I’ve listed are genuinely good and worth your time.

Overall interview prep guide by Chip Huyen and everyone must read this.
https://huyenchip.com/ml-interviews-book/

AI System Design vs ML System Design

Today both are slightly different. Let’s cover.

AI System Design

• LangGraph understanding

• About Tools, Context Management, RAG and ReAct framework

• In-depth design of Memory based on scenarios of product

• Subagents designing

• Prompt vs Semantic Caching based on scenario (HOTTEST TOPIC)

• Cost saving techniques

• Multi-agent orchestration

• Agent evaluations

• Observability of agents

• Latency vs Throughput

• Backend designing for AI applications

• Conversation Intelligence

ML System Design (Works for LLMs as well)

• General ML lifecycle

• Feature store (online vs offline)

• Feature Engineering pipeline

• Model training/LLM fine-tuning

• Offline Evaluation (Precision/Recall, F1 Score, MSE)

• Mode versioning and fallback

• Backend design with inference layer

• Online vs Batch inference

• Model quantisation

• Transfer Learning/PEFT/RL

Well, I may have missed something, but you can always ask AI for additional topics and learn them step by step within a single project. Never learn these topics one after another in isolation. Learn them while building the same project.

The sauce of MLOps

Honestly MLOps nothing but the summarisation above topics altogether.

MLOps Engineer = 40% AI/ML lifecycle + 25% Backend + 20% DevOps + 10% cloud + 5% Linux. Depends on organisation to organisation.

So you need,

• Docker containerisation

• Backend concepts like APIs/Websockets, rate limit, caching, auth, etc.

• Observability like Prometheus and Grafana

• Logging

• Networking and Load balancing

• Strong cloud fundamentals (AWS)

• Terraform

• Kubernetes HPA

• CI/CD

For MLOps this is my #1 resource.

@GokuMohandas covers every fundamental topic that every MLOps engineer should know. This is a must-read: madewithml.com

Additionally you should also learn the basics of system design.

Watch this Youtube Tutorial to understand these concepts in one shot:

6. Recruiters Expectations

This is the time when you need to be very active. Sometimes recruiters want you to attend interviews back to back, while others schedule one interview every week.

At this stage, you should satisfy both the technical requirements and the communication aspect of the interview.

The second most important factor is your joining date. If you’re exceptionally talented, recruiters might wait for 2–3 months, although that’s rare. Today, most companies want a strong candidate who can join within 30–45 days while still being employed.

High availability often attracts recruiters more than additional technical skills, especially in large tech companies or startups with a high volume of applicants.

The hiring process is unpredictable.

I believe luck plays a role here.

So yes, luck matters.

7. Bonus: My Experiences

When I attend an Interview

Most recruiters will ask about your expected CTC, joining date, and current work during the initial phone call. Be very clear with your answers. Decide your expected CTC beforehand and be ready to justify it.

A take-home assignment is a great opportunity to crack the interview, but you must understand the end-to-end lifecycle of your solution. If you fail to explain your decisions, you might lose the opportunity on the spot.

Always listen to the recruiter first before speaking.

For DSA, basic Python data structures and common patterns are enough. Practice easy to medium-level problems and solve them within a reasonable time. Even if you can’t finish the coding, keep explaining your thought process. Recruiters care about your approach. However, if another candidate has similar knowledge but stronger coding skills, they’ll have the advantage.

Talk more about your projects and real experience than the trendy tools and technologies you’ve seen on your X feed.

When I conduct an Interview

Here’s a polished version that keeps your message and style intact.

I’ve also been on the other side of the interview table.

Personally, I don’t ask DSA questions at all. My interviews are usually 50% AI/ML concepts and 50% system design.

I ask candidates to share their screen and walk me through one of their projects, from the high-level architecture down to the implementation details. Along the way, I ask a lot of questions about their approach, design decisions, trade-offs, and results.

These conversations tell me much more than a coding problem. I can understand how enthusiastic the candidate is, how they think, and whether they have the ability to solve real organizational problems.

That’s it for today guys.

Please feel free to add your experience in comment section.

Like, share, and join the cluster

Subscribe now

Part 1: The Agent Harness

What it is

An Agent Harness is the infrastructure layer that wraps a language model into a useful system. It is not the model. It is not the prompt. It is the scaffolding that answers every question the model itself cannot:

- Where does work product live between agent steps?

- What context does each agent see, and what is it explicitly excluded from seeing?

- How do agents coordinate without stepping on each other’s context windows?

- What happens when an agent produces bad output or a call fails?

- How is cost tracked, bounded, and controlled?

The LLM is a reasoning engine. The harness is the system that makes it useful. Every serious agent deployment is built on one explicitly designed or accidentally accumulated. The explicit version is faster, cheaper, and easier to debug.

The five components

A well-designed harness composes five concepts. Each one is a solution to a specific failure mode that emerges when you try to run agents in production.

Agent Harness: The Five Components

The five components are:

• Orchestrator: reads the brief, delegates in sequence, verifies completion, reports done

• Subagents: isolated execution contexts, each with one job and one output artifact

• Skills: per-agent knowledge documents that define role, output format, and rules

• Backend: the shared virtual filesystem where agents pass work product between steps

• Context Engineering: the discipline of controlling what each agent sees, when, and in what order

Understanding why each component exists is more useful than memorizing the API. The design decisions that follow from understanding the *why* are the ones that scale.

Part 2: Backends

The backend is the state management layer of the harness. It answers the question every multi-agent system must answer: where does work product live?

The problem with message-passing state

The naive approach is to pass agent outputs through conversation messages. Agent A produces structured insights and returns them. The orchestrator stores the response and passes it to Agent B as message content.

This creates three compounding problems.

• Context bloat. Every message in the orchestrator’s thread costs tokens on every subsequent call. After four agents have run and echoed their outputs back, the orchestrator is carrying thousands of tokens of content that only the next agent will need. You pay for it on every call until the job ends.

• No inspection surface. There is no way to read what Agent A produced without parsing the orchestrator’s conversation history. Debugging means wading through reasoning traces.

• Coupling. Agent B’s behavior depends on the exact format of Agent A’s response appearing in the conversation. A format change in A’s output schema breaks B. The coupling is invisible until it fails.

The virtual filesystem

The `StateBackend` solves all three problems by giving every job a virtual filesystem is an in-memory workspace scoped to the duration of the run.

Agents don’t pass content to each other through messages. They write files to the workspace and read from it.

Agents orchestration with skills.md

A typical workspace for a five-agent pipeline looks like this: a `brief.md` seeded at job start (the only input), then one file written per agent that extracted insights, drafts for each platform, and a final review notes file. When the extractor finishes, it writes its output file and confirms in one sentence. The orchestrator stores one line in its thread and not the content. The next agent reads directly from the file when it runs.

This single change reduces the orchestrator’s accumulated context by roughly 85% on a five-agent pipeline. It makes every intermediate output inspectable. And it decouples agents from each other where each agent reads a file by path, not a message by position.

Seeding and result assembly

The workspace starts empty. Before the orchestrator runs, seed it with everything the pipeline needs: skill files, shared context documents, and the per-job brief. Seeding separates the file system setup from the agent’s runtime where the workspace is fully defined before the first LLM call is made.

When the pipeline finishes, read the workspace to assemble the structured result. This is the only place where file content is read by the calling code.

Best practice: never read workspace content during the pipeline run from the orchestrator’s code. The orchestrator infers progress from file existence which files have which appeared not file content. Content is read exactly once, at the end.

Progress inference from the workspace

One under appreciated property of the file-based workspace: you can infer pipeline progress from which files exist. No explicit progress callbacks needed in the agent code. The workspace state is the progress state.

Part 3: Skills

What skills are

Every agent has a specific job. The job description is what to produce, what format it should take, what rules to follow which lives in a Skill file. Skills are markdown documents loaded into the agent’s context alongside its system prompt.

The key design decision is progressive disclosure: each agent loads only the skill it needs.

The naive approach is to include all skills for all agents. This is wrong for two reasons.

Token cost. Skills are static context loaded on every call. An X writer loading LinkedIn formatting instructions pays for those tokens every time it runs and contributing nothing to the output.

Focus degradation. Models occasionally apply instructions from the wrong context. An agent carrying instructions it doesn’t need will, with some non-zero probability, produce output influenced by those instructions. The more irrelevant context an agent carries, the worse this gets.

Each agent declares exactly which skill it loads. The skill resolver loads only the matching file. Five agents. Five skills. Each agent sees only its own.

Tool scoping follows the same principle

Tools are an extension of the skills concept. Like skills, they add tokens (tool definitions count toward input) and add behavioral surface area.

Give agents only the tools they can actually use. The reviewer agent gets a fact-checking tool and it’s the only one that verifies external claims. No other agent gets it. A writer that can’t call external APIs gets no tools. A tool it can’t use is overhead: tokens paid, behavioral risk incurred, no upside.

Best practice: the skill and tool set of an agent should be the minimum required to do its specific job. Expand scope only when a specific task demands it.

Part 4: Subagents and Isolated Contexts

Why shared context windows fail

When you run a multi-agent pipeline in a single conversation thread, every agent accumulates the full history. By the time the reviewer runs, it’s carrying the orchestrator’s planning reasoning, every writer’s output confirmation, and any back-and-forth from corrections. You pay for every one of those tokens. Quality degrades because the reviewer is reasoning in a context full of content that has nothing to do with its job.

Subagents run in isolation

Each subagent gets a fresh conversation: its own system prompt, its own skill, and only what it explicitly reads from the workspace.

Shared Thread of agents vs Isolated Contexts (Subagents)

The orchestrator maintains a flat list of agent descriptions. When it decides to delegate, it picks by description. The subagent runs in its own context, does its work, writes to the workspace, and its context is garbage-collected when done.

The cost difference is significant. A five-agent pipeline run entirely in one shared thread accumulates 15,000–30,000 tokens of history by the final step. The same pipeline with isolated subagents keeps each agent’s context at 2,000–5,000 tokens. The orchestrator’s thread stays lean because it accumulates file paths and one-line confirmations, not content.

Best practice: Design subagents to have exactly one output artifact in the workspace. One job, one file. This makes progress tracking trivial and outputs inspectable.

Part 5: Context Engineering

Context engineering is the discipline of controlling what enters an agent’s context window, when, and in what order. It has more impact on cost and quality than any other engineering decision.

The static-before-dynamic rule

This is the foundational rule. It must be followed without exception.

Static content is anything identical across many requests: system prompts, skill files, tool definitions, shared context documents. Dynamic content is anything that changes per request: job IDs, user input, timestamps, per-job parameters.

The correct ordering is: tool definitions first (most stable), then system prompt, then skill files, then conversation history, then the current user message (fully dynamic, never cached).

Provider-level prompt caching stores computed tensor representations of the prefix up to the first dynamic content. A cache read costs 10% of normal input price. A cache miss costs 100%.

Every violation of static-before-dynamic breaks the cache prefix. Common violations: putting a job ID or timestamp in the system prompt, embedding user-specific data in a skill file path, including environment-specific flags in system messages, or changing tool definitions between requests. The fix is always the same that move the dynamic value to the user turn message.

Durable memory vs. per-job context

Not all context ages the same way.

Durable memory is stable across every job: project conventions, behavioral guidelines, how agents should handle edge cases. This lives in a shared document loaded as memory at graph construction time. It gets computed and cached once per worker process.

Per-job context is task-specific: the user’s README, the requested platforms, the tone. This belongs in a brief document seeded into the workspace at job start where the only dynamic input to the pipeline.

The test for which category a piece of context belongs in: would it be identical across 1,000 different jobs? If yes, it’s durable memory. If it changes per job, it belongs in the brief and should never appear in the system prompt.

Thread compaction

For long-running pipelines, conversation threads grow. Older turns are less relevant than recent turns but still cost tokens on every subsequent call.

Thread summarization middleware handles this automatically. When the thread exceeds a configurable token threshold, older turns are compacted into a summary paragraph. The last N messages stay verbatim and recent context is the most relevant.

Best practice: Don’t set the summarization threshold too low. Summarizing at 80% of context capacity gives the agent room to work without constantly compacting. Summarizing at 40% wastes tokens on summary overhead.

Part 6: The Orchestrator

Coordination, not execution

The orchestrator’s role is exactly one thing: read the brief, delegate in sequence, verify completion, report done.

It explicitly does not produce content. This is a hard architectural constraint, not a guideline.

The orchestrator has the broadest context in the system. If it starts reasoning about domain-specific tasks like writing content, making factual judgments, formatting outputs and it will produce adequate results at the cost of long reasoning traces filling its context window, confusion between its coordination role and the domain role it just assumed, and bypassing the quality controls that subagents implement. Any time the orchestrator is tempted to produce domain-specific output, a subagent is missing from the design.

Build once, reuse always

The orchestrator’s construction is expensive: loading skill files from disk, initializing the model client, compiling the graph. Cache it at the process level. Build once per worker, reuse across every job.

A worker processing 40 jobs per hour builds the graph once and reuses it 40 times. A module-level singleton is Python’s simplest and most reliable pattern here.

Enforcing pipeline invariants in the prompt

The orchestrator’s system prompt is where pipeline rules are codified: only generate platforms listed in the brief, always run verification last, pass the job ID explicitly to every subagent, never write draft files directly, confirm file existence before reporting done.

These are coordination rules, not hints. Make them explicit and direct. The orchestrator’s prompt should read like a technical runbook, not a creative brief.

Part 7: The Caching Stack

Caching in agent systems has more leverage than in traditional applications because you can avoid work at multiple levels. Each level has different cost savings, different hit rates, and different implementation complexity.

3-layer caching

Layer 1: Provider prompt caching

Anthropic’s prompt cache stores computed KV tensor representations of the stable prompt prefix. Subsequent requests with identical prefixes read from cache at 10% of normal input token price.

The prerequisite is strict adherence to the static-before-dynamic ordering. Cache control is applied transparently on every system message — the call site doesn’t change.

Critical constraints engineers miss: the minimum token threshold is 1,024 tokens. Content below this fails to cache silently; no error, no warning, just a cache miss and full-price billing. Tool definition changes invalidate the entire cache hierarchy. You get at most four cache breakpoints per request.

Layer 2: Redis LLM response cache

Prompt caching reduces token cost but doesn’t eliminate API latency where you still make an HTTP call and wait for a response. A Redis response cache operates upstream of the API entirely: a cache hit means no HTTP call, no latency, zero cost.

Every LLM call in the system, orchestrator and all subagents automatically checks Redis before making any API call. The cache key is a hash of the full serialized message list combined with the model configuration. Including the model configuration in the key means a model upgrade creates new keys automatically.

Version your keys on prompt changes. Without versioning, a broken prompt gets cached and served for hours. A version bump at deploy time flushes the entire cache without touching Redis directly.

TTL by environment: Development at 5 minutes (prompt edits visible immediately), Staging at 1 hour (stable enough to catch regressions), Production at 24 hours (maximize cost savings).

Layer 3: Content identity cache

For systems where the same source material recurs popular open-source repos submitted by different users, documents processed multiple times and a content-identity cache can eliminate the most expensive pipeline step entirely.

Hash the raw source content. The same document with different user-specified parameters hashes to the same key because the source content is identical. This cache operates on content identity, not prompt identity. It bypasses the LLM entirely on a hit: no API call, no tokens, no latency. TTL can be much longer even seven days is reasonable for most content.

Part 8: Token Optimization

Tokens are the cost unit of LLM systems. Every inefficiency compounds across every user, every request, every retry.

Estimate before you execute

Never run a job without estimating its token cost first. A rough estimator uses character count divided by four (a reliable approximation for English text) plus fixed overhead for skills and context files. Log this for every job. After a week of production traffic you have real P50/P95 data. The numbers that let you set alert thresholds with confidence instead of guesswork.

Validate and truncate at the boundary

Validate input size before the job enters the queue. When input is oversized, truncate rather than reject some users with large inputs should still get results, just from the most information-dense portion of their content.

Snap truncation to a structural boundary (paragraph break, section heading) so the model doesn’t receive mid-sentence content. Append a truncation marker so the model knows the document is incomplete.

Model routing by task

Not every agent in your pipeline needs the most capable (and expensive) model. Structured extraction from a markdown document is something a smaller model handles well. Multi-document cross-referencing with judgment calls benefits from stronger reasoning.

Route cheaper models to extraction, classification, and format validation. Reserve capable models for final review and complex multi-step reasoning. Done correctly, this reduces total pipeline cost by 40–60% with no quality reduction on the overall output.

Part 9: The Async Job Architecture

Why you need a job queue

Agent pipelines take 45–120 seconds for non-trivial work. HTTP connections timeout in 30 seconds by default. Even if they don’t, holding a connection open per active job is a poor use of resources.

The correct architecture: accept the request immediately, return a job ID, run the pipeline asynchronously. The client polls for status. The result is written to a fast store the moment the worker completes. Total overhead for the poll loop: milliseconds.

Agent Orchestrator

Celery for LLM workloads

LLM tasks are I/O-bound, not CPU-bound. A worker thread spends most of its time waiting for API responses. This means you can run far more concurrency than CPU cores. The `gevent` pool uses cooperative multitasking where threads yield during I/O waits, allowing other tasks to run. A 4-core machine running 32 concurrent LLM jobs is reasonable when each job spends 70–80% of its time waiting for API responses.

The dual-store pattern

Redis is fast but ephemeral. Postgres is durable but slower. Use both for different purposes.

Redis handles the real-time polling use case like clients check status every few seconds and need sub-millisecond responses. Postgres handles the historical use case: user job history, billing, debugging jobs from yesterday.

The write pattern: write to both on every state change. The read pattern: check Redis first, fall back to Postgres if the key has expired. Never make Redis your source of truth. TTL expiration is silent.

Part 10: Development Workflow

Local model first, cloud model for validation

Development to Production

Separate development iteration from cost by building your harness to accept any LangChain-compatible model. Use a local Ollama model during development is free, fast, no API key needed.

Local output is lower quality than frontier models. It is good enough to verify that files are written to the correct workspace paths, the orchestrator delegates in the correct sequence, result assembly parses workspace files into the expected structure, and error handling works as designed.

When a skill file change needs quality validation, flip to the cloud provider for a single test run. Switch back immediately. This separation makes the iteration cycle for prompt and skill development essentially free.

Testing the harness, not the model

Unit tests for an agent harness should test harness behavior, not model output. The model is non-deterministic; the harness is not.

What to test: workspace seeding produces the expected file structure, result assembly correctly reads each file type, token estimation returns correct totals for known inputs, truncation snaps to paragraph boundaries correctly, cache key generation is deterministic, and state transitions follow the defined valid transition map.

What not to test at the unit level: whether the model produces good content. That’s integration testing with a real model, run periodically, not on every commit.

Part 11: Observability

Observability in LLM systems is harder than in traditional systems because the most important failure mode is qualitative degradation (invisible without the right tooling). A slow API call shows up in latency metrics. An agent that produces subtly wrong content doesn’t.

Structured logs

Every log line should be machine-parseable. Consistent field names across all log lines like job ID, status, step, elapsed time, whether the response was cached means you can grep, filter, and aggregate across your entire log history without a structured logging system. With this format, extracting P95 latency is a shell one-liner against your log file.

LLM call tracing

Structured logs give you job-level visibility. A tracing layer gives you call-level visibility: the full prompt, the response, actual token counts, per-call latency broken down by time-to-first-token and generation time, and the complete call tree across orchestrator and subagents.

When a user reports wrong output, you open the trace for their job ID and see exactly what prompt produced the problem. Without this, debugging hallucinations or incorrect agent behavior is guesswork.

Cache hit rate as a cost signal

Track cache performance explicitly. A sudden drop in LLM response cache hit rate is a signal that a prompt changed which usually means a dynamic value leaked into a previously static section, a model was upgraded, or a skill file was accidentally modified. Alert on this. It’s a cost event that’s easy to miss until the billing cycle closes.

Summary: Design Principles

Every decision in this guide comes from one of five principles.

Minimize accumulated context. Agents that carry less context are cheaper, faster, and more focused. Every component like the workspace backend, subagent isolation, thread compaction serves this principle.

Static before dynamic, always. Prompt caching is the highest-ROI optimization in a deployed agent system. It requires static content to come before dynamic content in every prompt, without exception.

Scope knowledge to role. Agents should know exactly what they need to do their job. Skills, tools, and context files should be narrowed to the minimum. Unnecessary context costs tokens and degrades focus.

Separate concerns cleanly. Orchestrators coordinate. Subagents execute. Backends hold state. These roles should not overlap. When they do, debugging becomes significantly harder.

Estimate, validate, and bound before spending. Token costs compound. Input validation, pre-execution estimation, and hard ceilings prevent runaway spend from becoming a production incident.

The infrastructure described here is not glamorous. None of it shows up in a demo. All of it is the difference between an agent that works on your laptop and a system that serves real users reliably.

Closing Thoughts

This guide is written from the other perspective: the one where you have real users, real costs, and a system that has to work at 3am when you’re not watching it.

The five components don’t solve interesting AI problems. They solve boring infrastructure problems. The boring problems are the ones that kill production systems.

I will be publishing one project on Agent Harness and Ops with video demo. Agent-Harness-Ops project deployment will be on

@Railway

to understand CI/CD lifecycle.

The model is the easy part. It was always the easy part.

Follow

@kmeanskaran

for more deployment and Ops article on AI/ML.

Further reading:

• Externalization in LLM Agents: Memory, Skills, Protocols and Harness Engineering (arXiv 2604.08224)

• Prompt Caching — Anthropic API Docs

• Don’t Break the Cache: Prompt Caching for Long-Horizon Agentic Tasks (arXiv 2601.06007)

• Optimizing Sequential Multi-Step Tasks with Parallel LLM Agents (arXiv 2507.08944)

• Context Window Overflow — Redis Blog

• Taming the AI Inference Queue: Redis, Celery & RabbitMQ at Scale

The model is not the problem. The architecture is.

This is exactly what agent harnesses are built to fix. And once you understand the difference, you will see why the harness is not an optional upgrade. It is a fundamentally different way of building with AI.

What a Classic Agent Actually Is

A classic AI agent is a language model sitting behind a chat interface. You send a prompt. The model generates a response. That is the entire loop.

Every request is stateless. The model has no awareness of what it did in the previous step. It cannot break a task into parts. It cannot write to a file, run a script, check whether its output was correct, or try again if something went wrong.

This works well for:

• Answering a specific question

• Summarizing a document

• Drafting a short piece of text

It breaks down immediately when the task requires:

• Multiple dependent steps

• External tool calls

• Quality validation

• Memory across steps

• Recovery from partial failure

A classic agent is a single shot. The harness is what turns that into a reliable workflow.

What an Agent Harness Is

An agent harness is the system built around a model. It is the layer that gives a language model the ability to plan, act, remember, delegate, and recover.

The model provides intelligence. The harness provides execution.

LangChain’s DeepAgents is one of the clearest implementations of this pattern. It is an open source agent harness built on LangChain and LangGraph, designed specifically for long-running, multi-step tasks. It ships with planning tools, a virtual filesystem, subagent delegation, context engineering, persistent memory, skills, sandboxed code execution, and human-in-the-loop support all out of the box.

The stack underneath it is layered deliberately:

• LangGraph is the graph runtime that handles durable execution, streaming, and stateful workflows

• LangChain’s create_agent is a minimal harness on top of LangGraph, good for lightweight agents

• DeepAgents is the full harness on top of create_agent, bundling everything a long-running agent needs

You use DeepAgents when you want the complete harness without building it yourself. You drop to create_agent when you want something thinner. You drop to LangGraph directly when you need a fully custom execution graph.

Why You Should Prefer the Harness

Before going into each component, it is worth being direct about why this matters.

A classic agent fails not because the underlying model is bad but because the model has no scaffolding to do sustained work. It cannot hold state across steps, cannot verify its own output, cannot delegate to a focused subagent, and cannot recover gracefully when something breaks midway through a complex task.

The harness solves all of these. It turns a capable model into a capable system. If your task has more than two or three sequential steps, involves any external data source, or needs results you can trust rather than results you have to manually verify, the harness is the correct choice. Classic agents are for demos and simple assistants. Harnesses are for work that actually ships.

The Components in Detail

1. Planning

The first thing a harnessed agent does when given a complex task is create a plan. DeepAgents ships with a built-in write_todos tool that forces the model to decompose the objective into steps before taking any action.

This might sound trivial. It is not. Without explicit planning, a model tends to jump straight into execution and lose coherence after the first few steps. With a visible todo list, the agent knows exactly where it is in the task, what comes next, and what is still outstanding. It can also adapt the plan mid-task as it learns more.

For a research-to-article workflow, the plan might look like:

1. Extract key sections from the source paper

2. Research supplementary background on the topic

3. Draft the article from structured notes

4. Review the draft against the source for accuracy

5. Finalize and format for publication

Each step is tracked. If step 3 fails, the agent does not restart from step 1. It retries from where it failed, with the filesystem holding everything that was already done.

2. Backends and Virtual Filesystem

The virtual filesystem is one of the most important components in the harness and the one most people overlook.

A classic agent has no persistent workspace. Every piece of context lives in the prompt, and as the task grows, the context window fills up and older information gets dropped. For anything beyond a short task, this is a serious problem.

DeepAgents exposes a filesystem surface to the agent through built-in tools (ls, read_file, write_file, edit_file, glob, grep). These tools operate through a pluggable backend. The backend is what decides where files actually live and how they persist.

DeepAgents ships with several built-in backends:

StateBackend (default) is ephemeral. Files are stored in LangGraph agent state for the current thread. This is the scratch pad: great for intermediate results that the agent writes during a task and reads back later. Files persist across multiple agent turns on the same thread via checkpoints, and they are shared between the main agent and its subagents.

FilesystemBackend reads and writes real files under a configurable root directory on your local machine. You specify the root, and the agent has access to everything under it. Best for local development CLIs and coding assistants where the agent needs to interact with your actual project files.

StoreBackend uses the LangGraph cross-thread store, which means files persist across threads, not just within one. This is the backend for long-term storage: memories, instructions, or reference data that should survive across multiple conversations.

CompositeBackend is the router. It lets you point different filesystem paths to different backends. For example, the default workspace can be ephemeral (StateBackend) but anything under /memories/ goes to the durable StoreBackend. This is maximally flexible and is how most production setups are configured.

The read_file tool natively supports multimodal content across all backends. Images (.png, .jpg, .gif, .webp), video, audio, PDFs, and PowerPoint files are all returned as multimodal content blocks that the model can actually read and reason about.

The practical effect is that intermediate work gets offloaded to files rather than staying in the context window. Research notes go to a file. The draft goes to a file. Reviewer feedback goes to a file. The context stays clean, and no work is lost between steps.

3. Context Engineering

Context engineering is the difference between an agent that works for five minutes and one that works for five hours.

The official LangChain docs define it as providing the right information and tools in the right format so your agent can accomplish tasks reliably. Deep agents have access to several kinds of context, and the harness manages all of them:

Input context is what goes into the agent’s prompt at startup: the system prompt, loaded memories, loaded skills. This is static and applied at the beginning of each run.

Runtime context is configuration passed at invoke time: user metadata, API keys, connection details. It propagates automatically to subagents.

Context compression is the automatic mechanism that keeps the agent within its context window limits as a task runs long. As conversation history grows, the harness summarizes it. When tool results come back large, the harness offloads them to the filesystem and lets the agent read them back in pieces. Skills and memory are loaded on demand, not all at once.

Context isolation uses subagents to quarantine heavy work. A research subagent processes ten documents in its own context window and returns only a clean summary to the main agent. The main agent never sees the raw noise, which means it stays sharp and does not lose track of earlier steps.

Long-term memory is persistent storage across threads using the virtual filesystem, which feeds back into input context on future runs.

This layered approach is what lets a deep agent run for a long time, across many steps and many tool calls, without degrading. A classic agent has no equivalent. It fills up and fails.

4. Subagents

Complex tasks are almost always made of distinct, separable responsibilities. A research task needs a researcher. A writing task needs a writer. A review task needs a reviewer. Trying to do all of these inside one agent with one context window is inefficient and unreliable.

A harnessed agent solves this by spawning subagents. Each subagent runs in its own isolated context window, with its own tools and system prompt, focused on exactly one job. The main orchestrator sees only the final result from each subagent, not all the intermediate steps that subagent took to get there.

This has two major benefits. First, context stays clean. The orchestrator is not drowning in research noise when it is trying to write. Second, subagents can run in parallel where tasks are independent, cutting total execution time. DeepAgents supports async subagents for exactly this: fire off multiple tasks concurrently and collect results when they finish.

Subagents share the filesystem backend with their parent. Any file a subagent writes is available to the main agent and to other subagents. This is how they coordinate: one subagent writes research notes to a file, another reads those notes and drafts the article.

In DeepAgents, any LangGraph CompiledStateGraph can be passed in as a subagent. This means custom orchestration logic plugs in alongside the built-in harness defaults without any friction. You can also configure subagents with different models, different tools, and different permissions than the parent agent.

5. Memory

Memory in a harness operates at two levels.

Short-term memory is task-scoped. It holds everything relevant to the current job: the plan, the current draft, the working notes, the review feedback. This lives in the filesystem (typically the StateBackend) and is cleared or archived when the task completes.

Long-term memory persists across sessions and across threads. User preferences, writing style rules, domain knowledge, past corrections. This uses the StoreBackend or a CompositeBackend with a durable route (like /memories/). The agent loads what is relevant from the memory store at the start of a session and carries that context forward.

The LangChain docs describe memory as what lets agents learn and improve across conversations. The harness loads memories into the system prompt as input context, and the agent can also write new memories to the filesystem during a run. This means the agent is not just consuming persistent knowledge but actively updating it based on what it learns.

Classic agents have neither form of memory. Every session starts from zero. Every interaction requires the user to re-explain context that should already be known. Long-term memory is what turns a capable agent into one that actually learns how you work and improves over time.

6. Skills

Skills are reusable agent capabilities that provide specialized workflows and domain knowledge. They follow the Agent Skills specification and are loaded from the filesystem when relevant.

Think of a skill as a detailed briefing for a specific type of work. A research skill tells the agent how to structure extracted notes, what metadata to capture, and how to handle conflicting sources. A writing skill defines tone, format, target reading level, and citation style. A reviewing skill defines quality criteria, what to flag, and how to return feedback.

DeepAgents supports both community skills (from the Agent Skills ecosystem at agentskills.io) and LangChain’s own skill library (langchain-skills on GitHub), which provides ready-to-use skills that improve agent performance on LangChain ecosystem tasks. You can also create your own.

Skills are not hardcoded into the agent. They are files that can be updated based on real usage. If a reviewing skill produces consistently weak feedback on a certain type of content, you update the skill file and every future reviewer subagent automatically improves.

This is what makes harnessed agents maintainable at scale. You are not rewriting prompts scattered across different parts of your code. You are editing a single skill file that propagates everywhere it is used.

7. Sandboxes

Agents generate code, interact with filesystems, and run shell commands. Because you cannot predict what an agent might do, the execution environment should be isolated so it cannot access credentials, files, or the network on your host system. Sandboxes provide this isolation.

In DeepAgents, sandboxes are a type of backend. Unlike standard backends (State, Filesystem, Store) which only expose file operations, sandbox backends also give the agent an execute tool for running arbitrary shell commands. When you configure a sandbox backend, the agent gets all the standard filesystem tools plus shell execution inside a secure boundary.

This is especially important for:

• Coding agents that need to run shell commands, use git, clone repositories, and run build/test pipelines

• Data analysis agents that need to load files, install dependencies, and run scripts

• Any agent that executes untrusted or generated code

DeepAgents supports multiple sandbox providers: Modal, Daytona, Deno, and a local VFS option for development. Sandboxes can be scoped per thread (each conversation gets its own sandbox) or per assistant (shared across conversations). Files can be seeded into the sandbox before the agent starts, and artifacts can be retrieved after execution completes.

The core principle is simple: the agent can do anything inside the sandbox but nothing outside it. Your host system, credentials, and network stay protected.

8. Human-in-the-Loop

Not every decision should be made autonomously. Some actions carry real-world consequences: sending a message, writing to a production database, deleting a file, deploying code. These should have a human approval step before execution.

DeepAgents supports this through LangGraph’s interrupt capabilities. You configure which tools require approval using the interrupt_on parameter, and the harness handles the rest.

Each tool can be configured independently:

• True enables interrupts with default behavior: the human can approve, edit, or reject the tool call

• False disables interrupts for that tool

• Custom configuration lets you restrict the allowed decisions (for example, approve and reject only, no editing)

agent = create_deep_agent(
    model="anthropic:claude-sonnet-4-6",
    tools=[delete_file, read_file, send_email],
    interrupt_on={
        "delete_file": True,
        "read_file": False,
        "send_email": {"allowed_decisions": ["approve", "reject"]},
    },
    checkpointer=checkpointer,
)

When the agent hits an interrupt, it pauses, surfaces the tool call details to the user, and waits. The user can approve it as-is, edit the arguments before execution, or reject it entirely. If rejected, the agent revises its approach and tries again.

Subagent interrupts are also supported: if a subagent’s tool requires approval, the interrupt bubbles up to the human rather than being silently executed.

This is not an add-on. It is a first-class primitive in the harness. And it is what makes harnessed agents trustworthy for tasks that classic agents could never be given access to in the first place.

9. Tools

A classic agent can generate text. A harnessed agent can take actions.

DeepAgents can call any tool you define: plain Python functions, LangChain @tool-decorated functions, tool dicts, and tools from any MCP (Model Context Protocol) server. You pass them directly to create_deep_agent via the tools= parameter, and they sit alongside the built-in harness tools.

The schema is inferred automatically from your function signature and docstring. You do not need to define a separate schema in most cases.

from deepagents import create_deep_agent

def internet_search(query: str, max_results: int = 5):
    """Run a web search"""
    return search_client.search(query, max_results=max_results)

agent = create_deep_agent(
    model="anthropic:claude-sonnet-4-6",
    tools=[internet_search],
)

MCP support is where this gets powerful. MCP is an open protocol that connects agents to a growing ecosystem of external servers (databases, APIs, file systems, browsers, and more) through a standard interface. Instead of writing custom integration code for each service, you point your deep agent at an MCP server and it gets all the tools that server exposes. One interface, any service.

On top of your custom tools, every deep agent inherits a set of built-in harness tools: ls, read_file, write_file, edit_file, glob, grep for filesystem operations, execute for shell commands (sandbox backends only), task for spawning subagents, and write_todos for planning. These are always available without configuration.

Where the Harness Wins: Real Use Cases

1. Deep Research Pipelines

A classic agent summarizes one document. A harnessed agent surveys ten sources, cross-references findings, identifies contradictions, and synthesizes a structured report. Each source gets its own research subagent. Notes are stored to the filesystem. A synthesis subagent works from accumulated files rather than from memory. The output is reproducible and reviewable.

2. Autonomous Coding Agents

Fixing a bug means reading the relevant files, understanding the surrounding context, writing the fix, running tests, reading the output, and iterating. A classic agent can write code. It cannot run it, read the test results, and loop back. A harnessed agent with a sandbox backend can do the full loop: read, write, execute, observe, revise. All inside an isolated environment that protects your host system.

3. Multi-Stage Content Pipelines

Research, outline, draft, fact-check, review, format. Each stage is a separate subagent with a specific skill loaded. Intermediate files persist between stages. A failure in the review step does not mean rerunning research from scratch.

4. Iterative Workflows with Feedback Gates

Some tasks are loops by nature. A design agent that produces a spec, routes it for review, gets feedback, revises, and repeats needs state across iterations. The harness tracks iteration count, stores each version, loads previous reviewer comments as context, and pauses for human sign-off when needed via interrupt_on.

Comparing the Two Directly

Execution

Classic Agent → One prompt, one output

Agent Harness → Multi-step workflow with planning and todo tracking

Memory

Classic Agent → None

Agent Harness → Short-term (StateBackend) + long-term (StoreBackend) persistent memory

Tools

Classic Agent → Very limited

Agent Harness → Custom functions, MCP servers, APIs, databases, file I/O

Context Management

Classic Agent → None

Agent Harness → Auto-summarization, offloading, compression, isolation via subagents

Subagents

Classic Agent → No

Agent Harness → Isolated context, parallel execution, async support

Skills

Classic Agent → No

Agent Harness → Modular, reusable, updateable, community ecosystem

Code Execution

Classic Agent → No

Agent Harness → Sandboxed shell via Modal, Daytona, Deno, or local VFS

Human Oversight

Classic Agent → None

Agent Harness → Per-tool interrupt configuration with approve/edit/reject

Failure Recovery

Classic Agent → None

Agent Harness → Retry from last successful step, filesystem holds completed work

Model Flexibility

Classic Agent → Usually tied to one provider

Agent Harness → Any tool-calling model (OpenAI, Anthropic, Google, open-weight via Ollama/vLLM)

The Bottom Line

A language model is a reasoning engine. It is powerful but it is stateless, toolless, and memoryless by default.

An agent harness is what makes that reasoning engine useful for sustained, real-world work. Tools, backends, context engineering, subagents, memory, skills, sandboxes, human-in-the-loop, planning. Each component addresses a specific failure mode of the classic agent. Together they close the gap between a capable model and a capable system.

The model gives intelligence. The harness gives execution.

I am currently building a one complete end-to-end Agent Harness and will deploy on Railway using CI/CD and AgentOps setup.

Follow me on X for more update

Signing off…

References:

LangChain Deep Agents Documentation: http://docs.langchain.com/oss/python/deepagents

The Anatomy of an Agent Harness: https://www.langchain.com/blog/the-anatomy-of-an-agent-harness

Note: I am using AI and ML interchangeable

Building software today is not hard anymore.
With tools like Claude Code and Codex, you can build something meaningful in 48 hours.

I did the same with

@desysflow 🌀.
It generates mermaid diagrams, HLD, LLD, and even non tech reports for stakeholders. Check this repo:

github.com/kmeanskaran/desysflow-oss

So building is solved.

But shipping is not.

That is where most people fail.

Even if you have access to powerful AI tools, you still need to understand some core systems to actually deploy and run your AI projects in production.

You do not need everything.
You just need the right fundamentals.

Why AWS

AWS has everything.

That is exactly the problem.

If you try to learn all of it, you will never ship.

Instead, focus only on what helps you deploy.

A. Backend Design

This is where your AI system actually becomes a product.

Most people overcomplicate this part. It is actually simple if you think in terms of flow.

1. REST APIs are simple, but not enough

For most ML systems, your backend starts with one main endpoint.

/prediction

User sends input, model returns output. Simple.

Then people think, let me add /train.

This is where things go wrong.

Training is not an API problem. It is a system problem.

Training takes time. Sometimes minutes, sometimes hours.
If you try to handle that inside a REST API, your system will break or timeout.

So instead of thinking “how to expose training”, think “how to orchestrate training”.

That shift changes everything.

2. Rate limiting is not about security, it is about survival

In AI systems, every request has a cost.

If your /prediction endpoint is open without control, you are basically giving away money.

One script, one loop, and your bill is gone.

Even worse, if training jobs are triggered multiple times, your infrastructure will get overloaded.

So rate limiting becomes mandatory.

Not just per user, but per API key and per IP.

It is not an optimization. It is protection.

3. You need to stop thinking synchronously

Most ML tasks are not instant.

Training, batch jobs, heavy inference. These things take time.

If your API waits for everything, your system will feel slow and unreliable.

This is where a distributed task queue comes in.

Instead of doing the work immediately, you push it to a queue.
Workers pick it up and process it in the background.

Now your API becomes fast, and your system becomes scalable.

This pattern alone will take you very far.

4. Caching is your unfair advantage

One thing you will notice in real systems is repetition.

Same prompts. Same inputs. Same outputs.

If you recompute everything every time, you are wasting compute and money.

Caching solves this.

Store previous results, and return them when the same request comes again.

This is extremely powerful in:

• LLM applications

• Recommendation systems

• Repeated inference scenarios

It makes your system faster and cheaper at the same time.

5. Not every prediction should be real time

This is a mistake many beginners make.

They assume everything must be instant.

But some predictions are heavy. Some depend on pipelines. Some take time.

Instead of forcing real time, make it asynchronous.

Accept the request, return a job ID, process it in the background.

Now your system is stable, and users still get results.

6. Model versioning is what saves you later

In the beginning, you have one model.

Then you improve it. Then you retrain it. Then you tweak features.

Now you have multiple versions.

If you are not tracking versions, you are in trouble.

You will not know:

• Which model is live

• Which one performed better

• How to rollback

Model versioning brings control.

It lets you experiment without breaking production.

7. Queue and Worker System

A task queue alone is not enough. You need workers to actually execute the jobs.

Think of it like this.

Your API does not do the heavy work.
It only pushes tasks to a queue.

Workers are the ones that:

• Pick tasks from the queue

• Run training jobs

• Execute heavy inference

• Process data pipelines

This separation is powerful.

You can scale workers independently based on load.
More jobs → add more workers.
Less load → reduce workers.

This keeps your system efficient and prevents your API from getting overloaded.

In real ML systems, this pattern is everywhere.
API for control. Queue for scheduling. Workers for execution.

8. Monitoring is not optional

Once your system is live, things will break.

Not because your code is bad, but because real world data is messy.

You need to see:

• API latency

• Errors

• Model performance

• Resource usage

Without monitoring, you are guessing.

With monitoring, you are engineering.

9. Feature store solves a silent problem

One of the most common hidden issues in ML systems is mismatch.

Your training data pipeline and your production pipeline are slightly different.

That small difference breaks your model performance.

A feature store ensures consistency.

You compute features once and reuse them in both training and inference.

This reduces bugs and improves reliability.

10. CI/CD makes you fast

If you are manually deploying every change, you will slow down very quickly.

Use GitHub Actions to automate your pipeline.

Push code → run tests → build Docker → deploy

Now your system evolves continuously.

You spend less time deploying and more time improving.

11. Security is usually ignored, until it hurts

Most ML engineers ignore security in the beginning.

That works until it does not.

You need:

• Authentication

• Authorization

• API key control

• Secure secret handling

And most importantly, never expose your model endpoints openly.

Treat your model like an asset.

B. Containerization

This is what makes your system portable.

Docker simplifies everything

Instead of worrying about environments, dependencies, and setups, you package everything into a container.

Now your model runs the same everywhere.

This is especially important in ML because dependency issues are very common.

With Docker, you remove that uncertainty.

CUDA and GPU workloads

If you are using GPUs, things get more complex.

CUDA setup is painful if done manually.

Docker solves this as well.

Using NVIDIA CUDA images, you get pre configured environments.

This makes GPU deployment predictable and reproducible.

C. Observability

Once deployed, you need visibility.

Track your experiments

Use tools like MLflow and Weights & Biases.

They help you track:

• Metrics

• Parameters

• Model performance

This is how you move from guessing to knowing.

Monitor your system

Use Prometheus and Grafana.

They give you visibility into:

• System health

• Resource usage

• API performance

Your model will degrade

No model stays perfect.

Data changes. Users change.

This leads to:

• Data drift

• Concept drift

If you do not detect this early, your model silently fails.

So always monitor data and plan retraining.

D. Kubernetes

You do not need Kubernetes on day one.

But when you scale, it becomes useful.

It helps you:

• Deploy services

• Auto scale

• Handle failures

In ML, it is especially useful for multi node setups and GPU workloads.

E. Terraform

Setting up infrastructure manually does not scale.

Terraform lets you define everything as code.

Now your infrastructure is:

• Reproducible

• Version controlled

• Easy to scale

F. AWS Usage

You only need a few services to get started.

• IAM for access control

• S3 for storage

• EC2 for compute

• ECR for Docker images

• EKS when you scale

• Billing alerts to stay safe

• AWS Bedrock for managed models

That is enough for most projects

Why SageMaker should not be your first choice

Amazon SageMaker is powerful, but it comes with tradeoffs.

It pushes you into:

• Notebook based workflows

• AWS specific ecosystem

• Higher costs

In early stages, flexibility matters more.

If you use Docker based deployment, you stay portable.

You can move to any cloud or even on prem later.

That freedom is important.

G. Minimal Setup

You do not need a complex system to start.

Just use:

• FastAPI

• Docker

• Redis

• One EC2 instance

• S3

That is enough to deploy real AI systems.

H. Flow Diagram

User → API
     → Rate Limit
     → Cache
     → Queue
     → Worker
     → Model
     → Storage
     → Response

F. I made an end-to-end project on MLOps

Stock Agent Ops:

github.com/kmeanskaran/stock-agent-ops/

Which generates the Bloomberg style financial report by giving ticker id of US stocks as input. Entire ML pipeline generates including classic LSTM for time-series forecasting and agentic AI for generating report.

Read these articles:

Part 1: Designing a Production-Grade Agentic MLOps System

Part 2: Deploying a Production-Grade Agentic MLOps System on AWS

Building AI is easy now.

Everyone can do it.

But deployment is where real engineering starts.

That is where systems break, costs rise, and complexity shows up.

If you focus on these fundamentals, you will be ahead of most people.

Start simple.
Ship fast.
Learn from production.

That is how you actually become an AI engineer.

Follow @kmeanskaran on X for more Applied AI/ML content

This guide walks through the full stack with a focus on what actually matters when you are building systems. You can anytime

1. The Core Mental Model

At the simplest level, an LLM does one thing:

It predicts the next token given previous tokens.

Everything else is built around making this prediction accurate, efficient, and useful.

Pipeline:

Text → Tokens → Embeddings → Transformer → Probabilities → Tokens

2. Tokenization and Embeddings

Before anything reaches the model, text is converted into tokens. These are integer IDs representing subwords or characters.

Tokens are then mapped into embeddings, which are dense vectors. These vectors capture semantic meaning and are the actual inputs to the model.

From an engineering perspective:

• Token count directly affects cost and latency

• Better tokenization improves performance in code and reasoning tasks

3. Positional Encoding (RoPE)

Transformers do not understand order by default. If you shuffle words, the model would treat them the same without positional information.

RoPE (Rotary Positional Encoding) solves this by encoding relative position using rotation in vector space.

Instead of adding position as a separate signal, RoPE rotates embedding vectors based on position.

Why this matters:

• Captures relative distance between tokens

• Generalizes better to long context

• Used in modern models like LLaMA

Engineering insight:

RoPE helps the model understand how far apart tokens are, not just their absolute position.

4. Self Attention: The Core Mechanism

Self attention is the heart of transformers.

Each token looks at all other tokens and decides which ones matter.

Mathematically, attention computes a similarity between tokens and uses it to combine information.

Intuition:

• Query asks a question

• Key represents what each token contains

• Value is the actual information

The model computes how much each token should attend to others and aggregates relevant information.

5. Causal Attention: Enabling Generation

In generation tasks, the model must not see the future.

Causal attention ensures that each token only attends to previous tokens.

This makes the model autoregressive, meaning it generates text one token at a time.

Without causal masking, the model would cheat by looking ahead.

6. Multi Head Attention and Its Variants

Instead of using a single attention mechanism, transformers use multiple heads.

Multi Head Attention (MHA)

Each head learns a different relationship:

• Syntax

• Semantics

• Long range dependencies

This improves representation power.

Multi Query Attention (MQA)

All heads share keys and values.

Benefit:

• Reduces memory usage

• Faster inference

Grouped Query Attention (GQA)

Heads are grouped, and each group shares keys and values.

This balances performance and efficiency.

From an engineering perspective:

• MHA is powerful but heavy

• MQA and GQA are optimized for production

  

7. Transformer Block

A transformer is built by stacking blocks.

Each block contains:

• Attention layer

• Feed forward network

• Residual connections

• Layer normalization

Flow:
Input → Attention → Add → Norm → FFN → Add → Norm

Residual Connections

They add the input back to the output of a layer.

This stabilizes training and allows deeper networks.

Layer Normalization

Normalizes activations to keep training stable.

8. Feed Forward Network and SwiGLU

After attention, each token passes through a feed forward network.

This is where computation happens independently per token.

Modern models use SwiGLU activation instead of ReLU.

Why SwiGLU matters:

• Better gradient flow

• Improved performance

• More expressive transformations

From an engineering point of view:
Attention gathers information
FFN processes it

9. Training: From Data to Intelligence

Training starts with pretraining.

Pretraining

Objective: predict next token

This is done on massive datasets using cross entropy loss.

What the model learns:

• Language structure

• Facts

• Patterns

• Basic reasoning

Training Challenges

• Distributed systems

• GPU utilization

• Data quality

• Memory constraints

Better data often matters more than bigger models.

10. Fine Tuning and Alignment

After pretraining, the model needs to be shaped.

Supervised Fine Tuning (SFT)

Train on instruction response pairs.

This teaches:

• Format

• Style

• Behavior

Instruction Tuning

Expose the model to diverse tasks.

This improves generalization.

Alignment Methods

RLHF

Uses human feedback and reinforcement learning.

DPO

Directly learns from preferred vs rejected responses.

GRPO

Learns by comparing multiple outputs within a group.

Key idea:
Alignment shapes behavior, not knowledge.

11. Parameter Efficient Fine Tuning

Full fine tuning is expensive.

LoRA

Adds small trainable matrices while freezing the base model.

Benefits:

• Low memory usage

• Fast training

QLoRA

Combines LoRA with quantization.

Enables training large models on small hardware.

12. Quantization: Making Models Deployable

Quantization reduces precision to save memory.

Formats:

• FP16

• INT8

• INT4

Benefits:

• Lower memory usage

• Faster inference

Tradeoff:

• Slight accuracy loss

Common methods:

• GPTQ

• AWQ

• QLoRA

Quantization is critical for production systems.

13. Inference: The Real System

Inference is where everything runs. Mainly focus on vLLM

Loop:
Input → Predict token → Append → Repeat

KV Cache

Stores intermediate values to avoid recomputation.

Reduces compute but increases memory usage.

FlashAttention

Optimizes attention computation by reducing memory movement.

PagedAttention

Manages KV cache using fixed size memory blocks.

Prevents fragmentation and improves efficiency.

Continuous Batching

Dynamically processes requests to maximize GPU usage.

Speculative Decoding

Uses a smaller model to speed up generation.

14. Decoding Strategies

The model outputs probabilities. Decoding converts them into tokens.

Options:

• Greedy

• Sampling

• Top k

• Top p

• Temperature

These control creativity and determinism.

15. Reasoning Models

Reasoning models generate intermediate steps.

Techniques:

• Chain of thought

• Self consistency

• Tool usage

Tradeoff:

• Better accuracy

• Higher cost and latency

16. Training Tools and Practical Stack

To work as an engineer, you need tools.

Hugging Face

• Model loading

• Training pipelines

• Datasets

Unsloth

• Faster LoRA and QLoRA training

• Lower memory usage

• Optimized kernels

vLLM

• High-performance LLM inference

• PagedAttention for efficient KV cache

• Continuous batching for better throughput

• Optimized GPU utilization for production

Typical workflow:

• Load base model

• Apply LoRA

• Train with Unsloth

• Evaluate

• Export for inference

• Serve using vLLM

17. The Real Engineering Insight

To build LLM systems, you must understand tradeoffs:

• Accuracy vs latency

• Memory vs speed

• Cost vs quality

Most real world work is about balancing these.

18. Final Mental Model

An LLM system is made of layers:

Model layer:

• Attention

• Transformer blocks

Training layer:

• Pretraining

• Fine tuning

• Alignment

System layer:

• KV cache

• FlashAttention

• PagedAttention

• Batching

Optimization layer:

• LoRA

• Quantization

Conclusion

Learning LLMs as an engineer means going beyond theory.

You need to understand:

• How attention works

• How models are trained

• How behavior is aligned

• How systems are optimized

I am recently learning about fine-tuning LLMs, distributed training, reasoning model and inference engineering till deployment.

Upcoming articles, will be more on LLM engineering and inference. I am going to design an entire LLM inference pipeline and update on X.

Follow me on X.

In Part 1: Designing a Production-Grade Agentic MLOps System, we saw how to build Stock-Agent-Ops — an Agentic MLOps system that predicts the next 7 days (technically 5 trading days) of closing stock prices. It also generates sentiment analysis, line plots, and recent news related to the stock for financial reporting.

Refer to GitHub repo, I moved AWS infra on aws-deployment branch.

I recommend you read that article first to understand how we designed this system (me, ChatGPT, and AntiGravity) from scratch. Although both articles are kept at a high-level abstraction; otherwise, this would easily turn into a 200-page eBook.

To learn more about the complete system design and deep AWS technical details, read the following README files:

1. A complete ML System Design

2. AWS Infra Design

Pre-AWS Theme

In this article, we are not going to discuss what AWS is or explain its services. Instead, we will focus on how to connect the dots and deploy the system quickly with minimal setup.

In the local version, I’ve been running the entire application using Docker Compose. Currently, I have three main Docker images:

1. FastAPI Backend

(ML + Agentic AI + classic microservice structure + caching + rate limiting)

2. Frontend UI

(Streamlit UI for the financial dashboard)

3. Monitoring UI

(Streamlit dashboard for data drift detection, agent observability, and visual logs for the entire system)

Apart from these, we also have other Docker images for supporting tools and services:

• Redis (for caching)

• Qdrant (vector database)

• Prometheus and Grafana (for overall system observability)

I’ve used Docker Compose for the local setup to run everything together — you can refer to Part 1 for that. Additionally, I’ve used local Ollama models for LLM tasks and embeddings.

Now, we need to prepare the application for AWS deployment. First, we must map everything from the code level to the infrastructure level. We will start by fixing the code-level components for AWS compatibility, and then move to infrastructure decisions like Docker and Kubernetes.

This is a best practice to ensure seamless integration from local development to production. It’s not just about running Docker on the cloud and we must also adapt internal logic. For example, in our case, we currently rely on a local Ollama setup, which cannot remain the same in production.

Let’s now look at each deployment step in detail.

Set AWS

It is an easy step. Let me break it down:

• Create an AWS account (Free Tier — $100 credits if eligible)

• Install AWS CLI

• Configure an AWS IAM user and user group

• Add the access keys to your local terminal

You can simply copy-paste these steps into ChatGPT to get more precise, step-by-step instructions tailored to your system.

Replace Ollama with AWS Bedrock

In the Agentic AI part, we used Ollama models:

• LLM: gpt-oss:20b-cloud

• Embedding model: nomic-text-embed

This is exactly what I meant by moving from the code level to the infrastructure level. At the code level, we need to replace Ollama with AWS Bedrock.

In this project, I’ve used:

• LLM: openai.gpt-oss-20b-1:0

• Embedding model: amazon.titan-embed-text-v1

Everything else at the code level remains the same. Only the model provider changes from a locally hosted Ollama setup to managed foundation models via AWS Bedrock.

Kubernetes Service Mesh

The k8s/ folder defines how our application runs inside EKS. This is where we convert our local Docker Compose setup into a production-ready Kubernetes environment.

Everything runs inside a dedicated mlops namespace so the system stays isolated and organized. For sensitive data like API keys and AWS credentials, we use Kubernetes Secrets. In CI/CD, these are injected from GitHub Secrets, so nothing sensitive is hardcoded.

Our main services, fastapi, frontend, and monitoring-app are deployed with defined CPU and memory limits to keep the cluster stable. We also configure basic health checks so Kubernetes can automatically restart any container that becomes unhealthy.

For storage, we use persistent volumes backed by AWS EBS. This ensures that vector data in Qdrant and cached data in Redis remain safe even if pods restart.

For observability, Prometheus collects metrics from the backend, and Grafana visualizes system performance and health.

This structure keeps the system clean, stable, and production-ready without making it overly complex.

We will push Docker images to AWS Elastic Container Registry (ECR), provision EC2 instances, and spin up an AWS Elastic Kubernetes Service (EKS) cluster with two worker nodes running on EC2.

Don’t worry we’ll use Terraform to provision and configure everything in one go. No manual setup, no console clicking. One command, and the entire infrastructure comes to life.

Infrastructure as Code (Terraform)

The terraform/ directory is the blueprint of our cloud environment. Instead of manually creating AWS resources, we define everything as code and let Terraform provision it in a repeatable and structured way.

Terraform handles the core infrastructure: networking, the Kubernetes cluster, container registry, and identity management.

In vpc.tf, we define the networking layer. Workloads run inside private subnets so they are not directly exposed to the internet. A NAT Gateway allows secure outbound communication (like pulling models or talking to AWS Bedrock), while an Application Load Balancer in public subnets manages inbound traffic safely.

In eks.tf, we provision a managed EKS cluster (v1.29) with t3.xlarge node groups. The 16GB RAM is important for running the FastAPI backend with heavy ML libraries. Each node is configured with 50GB EBS storage to avoid disk pressure during large image pulls.

In ecr.tf, we create private ECR repositories for each microservice to enable secure and fast image pulls within AWS.

In iam.tf, we configure IRSA using an OIDC provider. Instead of injecting AWS keys into pods, services assume IAM roles directly with scoped permissions for resources like Bedrock and CloudWatch.

Once defined, everything is controlled through Terraform commands:
1. terraform apply provisions or updates the full infrastructure.
2. terraform destroy removes it cleanly when needed.

This makes the entire environment reproducible, manageable, and production-ready from the infrastructure layer itself.

GitHub CI/CD Pipeline

CI/CD (Continuous Integration and Continuous Deployment) ensures that every code change moves automatically from development to production without manual intervention.

In this project, GitHub Actions handles the entire automation. The .github/workflows/deploy.yml file acts as the orchestrator for every update pushed to the repository.

Detail CI/CD structure

Whenever new code is pushed, the pipeline starts by syncing infrastructure. It runs terraform apply to ensure any infrastructure changes such as updated variables or instance upgrades are reflected in AWS before deploying the new version.

Next comes the Docker build stage. The images are built from the project root. This is critical because the backend needs access to modules from both the backend/ and src/ directories. Building from the root ensures imports work correctly and prevents runtime errors.

Once built, the images are tagged with the specific Git SHA. This makes every deployment traceable to a commit. The images are then securely pushed to private ECR repositories.

After pushing, the workflow dynamically updates the Kubernetes manifests. Using a simple sed replacement, repository placeholders in the YAML files are replaced with the actual AWS Account ID registry URL. This keeps the configuration environment-aware without hardcoding values.

Finally, deployment happens using kubectl apply, followed by kubectl rollout restart. This ensures rolling updates, allowing the new version to be deployed without downtime.

In simple terms, the CI/CD pipeline automates infrastructure synchronization, Docker image creation, registry updates, and Kubernetes rollout, turning every commit into a production-ready deployment.

Running the application

After executing the CI/CD pipeline, you can see all the Docker images running inside the Kubernetes cluster from the EKS dashboard.

At this point, you just need to access the monitoring dashboard, frontend application, and Grafana dashboard to verify everything is working as expected.

The complete application workflow and overall system design have already been covered in:

1. A complete ML System Design

2. AWS Infra Design

Now the important thing is, the applications are running on an EKS cluster with 2 t3.xlarge nodes. This setup is powerful but it costs a lot. If you leave it running, the bill can increase very fast.

Make sure to configure AWS Billing Alerts before testing anything. Set a budget limit so you get notified if usage crosses a threshold. Otherwise, the bill can skyrocket.

After running and testing the application, monitor the pricing regularly and destroy all AWS resources to stop burning your wallet.

Destroy All Services

After running all services on AWS we must destroy them before turning off your computer. So we will destroy all services, as our CI/CD is perfectly configured. So you can re-run the workflow to cold start entire system. We will use terraform to destory AWS services and recheck using AWS commands. I have written bash script to nuke AWS. See this directory of bash scripts.

Final Takeaway

• Always run your setup on a local machine before moving to AWS.

• For faster, scalable, and accurate deployment, start from the code level and then move to infrastructure.

• Always set a billing alert. I usually set it to $10 for PoC projects.

• Spin up EKS with minimal nodes first. Don’t overprovision in the beginning.

• Once your basic AWS setup works without errors, scaling becomes much easier.

• A data drift admin dashboard is extremely important to monitor the ML model’s performance behind the scenes.

• MLOps is not limited to the ML service. It also involves backend, infrastructure, monitoring, and deployment workflows.

• Start using caching for ML inference to save both cost and response time.

• Infra + ML Engineer will be a high-demand role very soon.

• Terraform is one of the best tools you can use for building reliable Ops pipelines.

• Always destroy AWS services when you are not using them.

• Machine Learning is iterative. Don’t chase a perfect solution from day one.

• Full-stack ML now means Data Engineering + ML + Backend + DevOps + Frontend + Cloud. AI can generate code, but it still struggles at infrastructure-level thinking.

If you like this project and article then follow me on X and subscribe to my Substack newsletter

Thanks for reading this blog and supporting this project. Up next, I’ll be sharing more about MLOps and how to set up Agents at scale.

If you’d like to know more about me or need help solving ML + Ops problems, feel free to reach out. I’d be happy to contribute to your team.

Here’s my website: kmeanskaran.com

Signing off!

“What is MLOps? and how can I learn?”

“How to work outside Jupyter Notebook?”

“What tools do I need?”

These are some of the most frequently asked questions by ML practitioners who want to design and build end-to-end machine learning systems, including MLOps.To address this, I designed a production-grade, end-to-end ML and Agentic AI project. This is Part 1, where I explain how to think about system design and decision-making while working on real-world ML projects.

In Part 2, I will cover deployment on AWS.

For more such updates, follow me on X.

BEFORE READING THIS BLOG:
1. Watch this live demo video
2. See the system design doc

Introduction

When we say Machine Learning, many people think only about model.fit() and Jupyter notebooks. That is exactly where the difference between research ML and applied ML begins.

If you want to work in startups, big tech, or on the applied side of ML where systems directly drive business value, this article is written for you.

Let’s begin.

In this project, I am designing a Stock Agent that generates a Bloomberg-style financial report for the next five days of stock performance. It includes metrics such as Open, High, Low, Close, and other derived indicators.

Here is the interesting part. We are not using LLMs or any generative AI model to predict stock prices. Instead, we use an LSTM-based time series model for forecasting. This makes the project a strong combination of classical machine learning and Agentic AI.

Throughout this article, we will focus on how to design this problem for a production-grade environment. We will also cover edge cases and system-level considerations that most people tend to ignore, but which matter the most in real-world ML systems.

DISCLAIMER:

This project focuses on building an end-to-end machine learning system rather than maximizing prediction accuracy. While I do consider the quality and performance of the ML model, this work is intended as a proof of concept, with primary emphasis on system design and engineering decisions.

You are free to experiment with models other than LSTM or apply hyperparameter tuning to improve performance. This system should not be used for making real stock trading decisions. It is created purely for learning and educational purposes.

I am human and I do make mistakes. If you notice any issues or have suggestions for improvement, please feel free to share them in the comments.

The Problem

Design a system that predicts five-day stock performance and delivers a clean UI where users can input a stock ticker. The system generates a Bloomberg-style report that includes next week’s forecasted prices, current market news, sentiment classification such as Bullish, Bearish, or Neutral, and a visual chart for the forecast.

That defines the problem clearly.

Now the next step is to design the framework and structure the code into well-defined modules, with each component handling a specific responsibility. In real-world production systems, different teams work on different modules, and clean separation is critical for scalability and maintainability.

Let’s start designing the system.

How to Design

Looking at the requirements, we need two core outputs: forecasted stock prices and a structured financial report. This naturally leads us to two major ML components in the system.

1. A classical machine learning model for time series forecasting.

2. An Agentic AI layer to generate a financial report based on the model outputs and market context.

For time series forecasting, the most commonly used models are SARIMAX, Prophet, and LSTM. I chose LSTM, and here is why.

We are working with stock market data, which we source from the well-known Yahoo Finance API. Since we are forecasting US stocks, it is important to understand broader market behavior. Every stock market has an index that reflects overall trends, economic signals, and technical patterns. In the case of the US market, the S&P 500 serves this role.

Because LSTM is a neural network, it allows us to apply transfer learning. We first train an LSTM model on the S&P 500 index, which acts as the parent model. This model learns general market behavior. By freezing its weights, we then train child models for individual stocks such as Apple or Nvidia. This helps child models benefit from broader market knowledge while adapting to stock-specific patterns.

For experimentation, tracking, and storing model artifacts, we use MLflow along with DagsHub.

Now that the core decisions are clear, let’s look at the high-level system design.

Feeling confused? I have already documented a complete, detailed system-level design and explanation in the GitHub repository. For deeper technical details, I recommend reading that first.

To train the model, we begin with feature engineering and preprocessing. This step focuses on generating meaningful features that directly impact forecasting performance. After that, we move into model training using LSTM along with transfer learning, which we discussed earlier.

Moving forward, we need a robust inference pipeline. This pipeline uses an online feature store powered by Feast to serve weekly stock predictions in a reliable and scalable manner.

So far, we have covered training and inference. However, real engineering begins after this point, especially when we integrate Agentic AI and design the backend systems that support it.

Let’s go through these components one by one.

Where to use Agentic AI?

A common question I often get is why we do not use Agentic AI directly for both forecasting and report generation. After all, it is easy to integrate Agentic AI with the Yahoo Finance API and let it do everything.

This is a good question, and it highlights where many startups and engineering teams fail.

LLMs and Agentic AI are not the solution to every AI problem. They are excellent at tasks such as question answering with given context, summarizing information, and extracting or scraping textual data from the internet. However, they are not well suited for learning continuous and complex temporal patterns.

Stock price forecasting is exactly that kind of problem. In fact, Agentic AI should not be used for forecasting at all.

Instead, the right approach is to use a dedicated time series model for prediction and then pass the forecasted outputs to an Agentic AI layer. That is precisely what we do here.

The LSTM model generates the future price predictions. These outputs are then provided to the Agentic AI, whose responsibility is to gather relevant market news for the given stock ticker, analyze sentiment, and generate a structured financial report. Additionally, I use a critic agent to evaluate the quality and consistency of the generated report.

This is how classical machine learning using LSTM and Agentic AI are integrated in a practical, production-oriented system

If you think I contribute in your learning, then feel free to sponsor me

Backend—The backbone

There is no point in repeating everything from scratch, as I have already defined everything from A to Z in the system design document.

But let me share the key points:

• We need a microservice architecture for this project, as we are connecting multiple independent components. ML systems are hard to debug, and a monolithic architecture is not recommended.

• Keep parent and child training and prediction APIs separate.

• Connect Redis for caching forecast outputs.

• Use Qdrant to store generated stock reports in a vector database for ticker-based filtering and caching. Here, we do not need semantic caching, as we are caching reports for individual stocks.

• After receiving a stock name request, first check whether the model output is present in Redis. If not, train the model from scratch. While doing this, check whether parent model weights are present. If not:

  train the parent model → train the child model using parent weights → generate predictions → store them in Redis → call the Agentic AI API to fetch data from Redis → generate a financial report → serve it to the UI → store the report in Qdrant.

• We also provide additional APIs, such as hard delete, to remove all model weights and flush both Qdrant and Redis.

• The backend is asynchronous, but this is not ideal when handling a large number of concurrent requests. That is why we use rate limiting with different request limits for training and inference endpoints.

• Finally, we use Docker to containerize the backend application.

  ---

Observability

This is one of the most important parts of an ML system. In most ML pipelines, it is rarely discussed or properly implemented. However, as you move toward infrastructure-level and enterprise-grade ML, this becomes a necessity.

In simple terms, observability is used to monitor the overall health of the system. For ML systems, this also includes data drift and model drift detection. Since we are using Agentic AI, evaluation of agent outputs is also required. Along with that, monitoring backend and system health is mandatory.

For this project, we focus only on data drift detection for the parent model. This is because we do not keep the same parent model for more than a week, and child models can be retrained whenever required. This keeps the setup simple.

For general system monitoring, I am using Prometheus and Grafana.

Frontend and UX

Here, we have two primary frontends:

1. A user interface for generating stock reports, focused on end users.

2. A monitoring dashboard for training the parent model, detecting data drift, evaluating generated reports, and viewing system logs, focused on the ML team.

These are UIs for this project.

Future improvements

• The LSTM model is underfitted and needs improvement.

• Model versioning is implemented in the code but is not actively used.

• Cron jobs are required for parent model training and data-drift-based retraining.

• Better evaluation mechanisms are needed for Agentic AI outputs.

  ---

Key Takeaways

• Always think from a business perspective.

• The more fancy tools you use, the harder debugging becomes.

• Use AI for coding, but you need to be very strong in system design.

• AI agents are not the solution to every problem, but you should always evaluate where they fit.

• Write code outside Jupyter notebooks.

• I built this project with the help of ChatGPT, AntiGravity, and Grok, without following any tutorial.

  ---

his is just Part 1. In Part 2, I will deploy the entire system on AWS using services such as EC2, ECR, and EKS. I will also use Terraform for infrastructure provisioning. That will be covered in the next article, which I will release soon.

If you liked this blog post, I am confident you will enjoy my other content as well. I can predict that too. 😉

• Visit my website

• Follow me on X

• Follow me on LinkedIn

• Buy me a coffee

Until next time :)

In simple terms, prices are not fixed. They move based on demand, supply, competition and timing. When demand is low, prices fall to push more sales. When demand rises, prices increase because customers are willing to pay more. Dynamic pricing tries to match the right price to the right moment.

Most of these decisions come from algorithms. These algorithms use data like sales history, past prices, seasonality, competitor prices and time patterns. Then they estimate the price that gives the highest revenue or profit. As more data comes in, the algorithms update themselves and get better with time.

Today we’re going to look at what dynamic pricing is and how big product-based companies adjust prices every second. I’ll also share my experience building such systems.

Real Examples of Dynamic Pricing in Action

1. Uber Pricing: How Uber Adjusts Fares in Real Time

Uber uses one of the most advanced real-time pricing systems in the world.
Here is how it works in a technical sense:

• Uber constantly measures demand (ride requests) and supply (available drivers) in small geographic zones called geohashes.

• When demand is higher than supply, the system triggers dynamic pricing. Fares increase so that more drivers come online and move toward that area.

• Uber used to rely on simple surge multipliers. Modern versions now use more advanced models such as Bayesian inference, gradient boosting models, and auction-style pricing logic.

• The system also studies rider behavior and driver acceptance. For example, if many riders reject the fare, the algorithm adjusts the multiplier downward.

• Uber sometimes uses per rider pricing, which means two people standing near each other may see different prices based on predicted acceptance.

• Once supply increases or demand cools, the prices return to normal automatically.

Uber’s goal is not only revenue. The system tries to restore balance in the network so that rides are available when people need them.

2. Google Ads Smart Bidding: Dynamic Pricing For Ad Auctions

Google Ads uses dynamic pricing in the form of automated bidding for every ad auction. This is known as Smart Bidding.

This is what happens under the hood:

• When an ad auction begins, Google evaluates hundreds of real-time signals for that impression. These include device, location, time, browser, keywords, user history, demographics, and remarketing lists.

• Google has machine learning models trained on huge amounts of historical conversion data. The models predict the probability of conversion and expected value for that particular user.

• Based on your goal (for example Target CPA or Target ROAS), Smart Bidding sets the optimal bid for that exact auction.

• The bid can be higher for users with high conversion probability and lower for users with low probability.

• The system continuously learns from new data and updates the probability estimates.

This makes Smart Bidding one of the best examples of dynamic pricing in the digital space. The “price” here is the bid amount for each impression.

How Dynamic Pricing Systems Work Internally

Dynamic pricing systems usually have two main parts:

1. A demand forecasting model

2. A pricing engine that uses optimization or reinforcement learning

Let us break this down.

If you feel my content adds value to your ML journey, show some support to your centroid. It helps me make your K-Means cluster even better.

Demand Forecasting Models

A dynamic pricing engine first needs to forecast demand. It must answer the question:
If I set this price today, how much will I sell?

Different models are used here.

Classical Time Series Models

Models like ARIMA, SARIMA or Exponential Smoothing use historical patterns to predict demand. They work well when data is stable and follows trends or seasonality.

Machine Learning Models

Tree models like Random Forest or Gradient Boosting accept many features such as:

• Price

• Time of day

• Day of week

• Holidays

• Competitor prices

• Marketing spend

• Product attributes

• Inventory levels

They capture complex relations and usually perform well for e-commerce or retail data.

Deep Learning Models

Neural networks like RNN, LSTM and Transformers can learn long term sequences and patterns in sales data. These models help when patterns shift or when the dataset is large.

The output of forecasting is a demand curve. This curve tells the system how demand changes for each possible price.

Pricing Algorithms: Optimization and Reinforcement Learning

Once the system can forecast demand, it chooses a price using one of two strategies.

1. Greedy Optimization (Simple and Widely Used)

This method looks at immediate revenue:

Revenue = Price × Predicted Demand

The system checks a range of possible prices and picks the one that gives the highest revenue right now. It is fast and easy to use, but it ignores long term effects.

2. Reinforcement Learning (Advanced and Future Focused)

Reinforcement learning treats pricing as a sequence of decisions.
Today’s price affects customer behavior tomorrow.

RL works like this:

• State: market conditions (demand, supply, inventory, competitor prices, time)

• Action: choose a price

• Reward: revenue or profit

• Goal: maximize long term revenue

RL algorithms learn through experience. They explore different prices, observe results and slowly build an optimal pricing policy.

Common RL methods here include:

• Q Learning

• Deep Q Networks

• Policy Gradient

• Actor Critic

• Multi Armed Bandits

Companies often train these RL agents inside simulated environments to avoid experimenting on real customers. Once trained, the RL agent is deployed and monitored.

Building a Full Dynamic Pricing System

A real production-ready system usually follows these steps:

1. Data Collection

Collect historical sales data, prices, competitor data, product attributes, holidays and other context.

2. Forecast Demand

Train time series or ML models to predict demand for different prices.

3. Choose Pricing Strategy

Pick between greedy optimization or RL based on product and business goals.

4. Add Constraints

Add rules like minimum margin, price ceilings, inventory limits or regulatory boundaries.

5. Train and Validate

Test the model on past data.
If using RL, train in a simulated environment built from demand models.

6. Deploy and Monitor

Deploy the pricing engine.
Run A/B tests.
Track revenue, conversion, customer impact and model drift.
Retrain regularly.

Why Dynamic Pricing Matters Today

Dynamic pricing is becoming a core skill for ML engineers, product teams and business operations. It reflects how modern companies make decisions in real time using data and machine learning.
Tech companies like Uber, Amazon and Google rely heavily on dynamic pricing and bidding systems to run large scale marketplaces and advertising platforms.

A well designed pricing system can significantly increase revenue, improve availability, balance supply and create a better customer experience. At the same time, it requires caution because pricing decisions affect user trust and long term business stability.

Dynamic pricing sits at the intersection of machine learning, economics and strategy. It will only grow more important in the coming decade.

How’s my Experience

I designed an RL-based dynamic pricing system that predicts the optimal price of a product by analysing the past price-to-units relationship, time-series patterns, and forecasted demand. Let me break it down.

At AiHello, our eCommerce automation platform builds multiple AI-driven services to help Amazon sellers make smarter decisions. One of these services focuses on setting product prices in a way that increases sales at slightly higher price points, while still avoiding over-pricing. Since we already have a demand-forecasting service in our data warehouse, we can directly use its output to simulate and estimate the impact of different price levels.

Statistically, there are many correlations between pricing features and demand, but dynamic pricing cannot rely only on metrics like MSE or R2. The real behaviour of a pricing system can only be validated in production. This is why A/B testing or deploying the model for a small set of products is essential.

Reinforcement Learning then becomes the final layer of optimisation. Around 80% of the work depends on strong statistical and machine-learning fundamentals, while the remaining 20% comes from the RL module. RL helps explore all possible price scenarios and choose the one that achieves higher sales while maintaining healthy price efficiency.

See my X post about this project experience

That’s it for today. I’m looking forward to writing more exciting blogs on ML, MLOps, and GenAI. See you soon!

Share

Subscribe now

But ML projects involve both data and code. Along with programming, you need to understand, clean, and reason about data.
That’s what makes ML both challenging and exciting.

Most people believe ML is just about training models in Jupyter Notebooks. Yes, that’s a part of it, but not the whole story. Model training is important, but it’s not everything. The real complexity lies in what happens before and after training: data understanding, feature engineering, evaluation, and deployment.

When I’m assigned a new ML project, I usually start with just two things: the business problem statement and the data.
The problem statement tells me why this ML service is needed in the product, and the data tells me what I’m working with. It’s usually messy, huge, and confusing at first glance.

I talk with the manager or CEO to understand the business side. They know why this ML service matters and what the expectations are. My job is to translate that into a working ML pipeline.

Once I understand the problem and the data, I start building the project step by step.

1. Understand Data and Problem Statement

In this stage, talk with your product manager and CEO. They will explain the problem statement and what they expect from the ML system.
Write everything down, even if you don’t understand it immediately. They won’t talk in technical terms, and that’s fine. Translating those expectations into technical understanding is your responsibility.

Create a Google Doc or Notion page for the project outline. Keep updating it as you gain clarity. Discuss with teammates and managers regularly until you fully understand the business objective and the data.

When it comes to data, study the schema, update frequency, and column data types. If your dataset is in SQL, write a simple query with a LIMIT clause to fetch a few thousand records. Load it into a Pandas DataFrame and explore using df.head(), df.info(), and df.describe(). This helps you identify null values, data inconsistencies, and unexpected types.

Jupyter Notebook is perfect for this step because it helps you explore, visualize, and document everything together.

2. Statistics and Feature Engineering

Statistics and probability are essential in understanding data before you move to model training.
Start with the basics: mean, median, standard deviation, and count for each feature. Plot boxplots to find outliers, and use histograms or line plots to understand the distribution. Check skewness to see if the data needs normalization or transformation.

For text data, check if there are missing or incomplete values and whether the dataset is imbalanced. If the data is multilingual, identify the languages present. For image data, check resolution consistency and missing labels. In such cases, image augmentation techniques like flipping, rotation, or cropping can improve the dataset.

For tabular data, study how features correlate by plotting heatmaps or using correlation coefficients. You can also apply techniques like PCA or SelectKBest to reduce dimensionality and improve model performance.

The more time you spend understanding and engineering your features, the better your results will be. A well-prepared dataset often performs better than a complex model trained on messy data.

3. Model Selection

Model training is the heart of ML, but it shouldn’t consume all your time. Choose your model wisely based on the nature of your data.

• If your data is tabular and the task is classification or regression, start with XGBoost or LightGBM.

• For time series forecasting, try Prophet or ARIMA models.

• For NLP tasks, BERT or DistilBERT work well. If you’re working on generative or LLM-based tasks, try Qwen or Llama models.

• For image classification, YOLO or EfficientNet are reliable choices.

The real magic lies in the pipeline, not in the model.
I’ve seen many engineers pick complicated models too early. The code becomes messy, and debugging turns into a nightmare. Especially during backpropagation, small issues can waste hours.

Always start simple. Focus on solving the business problem first. You can add complexity later if needed.
Use tools like MLflow or Weights & Biases for tracking your experiments and model versions.

4. Evaluate the Pipeline, Not Just the Model

ML isn’t only about accuracy or F1-score. A model that performs well in a notebook might fail in production. That’s why you should evaluate the entire pipeline, not just the model.

Track common metrics like MSE, Precision, Recall, and F1-score, but go further. Perform A/B testing to see how your model impacts real business outcomes. Measure latency, throughput, and memory usage.

Try to break your pipeline intentionally. Feed invalid inputs or extreme values and see how it behaves. This helps you understand its robustness and limitations.

Also, evaluate how your ML system affects business KPIs such as conversions, retention, or cost savings. Metrics without real-world meaning don’t add much value.

5. Importance of Edge Cases

You’ve probably read this online many times:

The model behaves differently in production.

It’s absolutely true!

In production, your model faces noisy, incomplete, or completely new data. That’s why you need to think beyond technical performance and consider real-world behavior.

When testing your pipeline or API, use different kinds of inputs and note the outcomes. If the ML service meets the desired criteria, move it to production. Once live, monitor it using A/B tests, performance dashboards, and feedback loops.

You can use tools like Prometheus and Grafana for latency and performance tracking, or Evidently AI for detecting data drift. Retraining pipelines using Airflow or Prefect can help keep your model updated as data changes.

Always note why something failed, not just that it failed. Continuous improvement is the key to building stable and scalable ML systems.

Machine Learning projects succeed when you combine data understanding, thoughtful feature engineering, clean pipelines, and strong monitoring. The goal isn’t to build the flashiest model but to solve a real business problem efficiently.

Focus on clarity, structure, and results. Everything else will follow.

From my experience

• Communicate with the team, managers, CEO (if you’re into startup), or SMEs.

• Data is everything! Better data → better model performance.

• Do not overcomplicate your ML project. It’s so difficult to debug.

• Follow iterative approach for your projects.

• Pipeline is important than model for business.

• You don’t need fancy tools and frameworks for better result.

• Use AI for coding and ask it to help you in constructing the architecture.

For better technical understanding about designing ML project, read this OG book:

• Designing Machine Learning Systems by Chip Huyen

Thanks, for reading this article!

I summarised a simple framework to follow for your next ML project. So even if you are a beginner, this is a good way start looking at your ML projects.

Subscribe to my newsletter for such more interesting blogs in AI/ML and MLOps.

Subscribe now

Machine learning projects are never a single-step job. They include several stages like data collection, data cleaning, feature engineering, model training, testing, and deployment. Each stage depends on the successful completion of the previous one.

When you do all this manually, it becomes tiring and error-prone. Imagine having to re-run the entire process every time you change a small part of your code or dataset. You would have to remember the order of steps, monitor logs, and keep track of outputs. This is manageable for small experiments, but once the project grows it becomes a headache.

That’s where orchestration solves the problem.
Orchestration means automating and managing the entire workflow of your machine learning project so that tasks run in the correct order, automatically, without manual effort.

Think of orchestration as a “smart manager” for your ML workflow.

• It knows which tasks need to run first and which depend on others.

• It can stop the pipeline if something fails, retry the step, or skip unnecessary tasks.

• It can run your workflows on schedule (cron jobs): daily, weekly, or after a trigger.

For example, consider a simple ML pipeline:

1. Load your dataset

2. Clean and transform it

3. Train the ML model

4. Evaluate model accuracy

5. Deploy it if accuracy is high enough (yes, we can create if-else conditions in pipelines).

If any step fails, the orchestration tool can stop the workflow or retry automatically.
This helps avoid wasted time, broken pipelines, or manual reruns.

With orchestration, your ML system becomes smooth, reliable, and scalable. You can schedule workflows, monitor them in real-time, and even handle multiple projects together.

2. Why Orchestration Makes ML Projects Faster

Orchestration makes machine learning workflows more efficient and reliable. Instead of manually running scripts, everything happens automatically in a defined sequence.

It saves time because once you set up the workflow, you can schedule it or trigger it when new data arrives. It also reduces errors since each task runs with clear dependencies, no missing steps or wrong order.

Example scenario: Let’s say you want to retrain your recommendation model every Sunday at 11 PM. With orchestration, you can schedule this workflow once. Every week, it will automatically fetch the latest data, train the model, check accuracy, and push it to production — all without human help.

This brings speed, safety, and control to your ML workflow. These three things every production-grade ML system needs.

3. How to Design Orchestration DAGs

A DAG (Directed Acyclic Graph) is the heart of ML orchestration. It defines how your tasks connect and in what order they should run. The term “acyclic” means tasks move forward, they never loop back.

In simple words, a DAG is like a flowchart for your ML pipeline. Each box is a task, and arrows show which task depends on which.

For example, a basic ML DAG might look like this:

1. Load data

2. Clean data

3. Train model

4. Evaluate performance

5. Deploy if accuracy is above threshold

You can also add conditions (if-else) inside DAGs. For instance, deploy only if accuracy > 85%, else trigger a retraining step.

When designing DAGs, keep these things in mind:

• Each task should do one specific job.

• Always define dependencies clearly (what runs after what).

• Handle failures and retries properly.

• Add scheduling for recurring workflows (like weekly retraining).

A good DAG turns your ML process into a repeatable, fault-tolerant pipeline that’s easy to track and scale.

But,

What’s the difference between DAGs Orchestration and Kubernetes Orchestration?

While both DAGs and Kubernetes handle automation, they serve different purposes.
A DAG (Directed Acyclic Graph) in ML orchestration focuses on workflow logic that is defining the order in which tasks like data cleaning, training, and deployment run. It’s about what runs and when.
Kubernetes, on the other hand, handles infrastructure orchestration like managing containers, scaling pods, and ensuring resources are available. It’s about where and how things run.

In simple terms:

• DAGs orchestrate the ML workflow

• Kubernetes orchestrates the system that runs the workflow

Practice building your machine learning projects in a modular way and execute the entire pipeline as a flow from data ingestion to deployment. This approach helps you standardize your process and make your projects production ready.

4. Orchestrate ML workflow using Prefect

Modern data workflows often involve multiple dependent steps — data ingestion, model training, evaluation, and conditional decisions. Managing all of this manually can get messy. This is where Prefect comes in.

Prefect is a modern workflow orchestration framework that lets you build, schedule, and monitor data pipelines directly in Python without the complexity of static DAG definitions or YAML files.

It turns normal Python functions into tasks and automatically manages their execution order, retries, caching, and logging all while keeping your code clean and intuitive.

Key Features

• Pure Python orchestration — no DSLs or complex configs.

• Dynamic DAGs built at runtime (not hardcoded).

• Retry, caching, and scheduling with one line of code.

• Easy observability via the Prefect UI (local or cloud).

• Flexible execution — run locally, in Docker, or on Kubernetes.

To get started, follow the Prefect installation guide from the official documentation.

Here’s a simple Prefect flow that trains a regression model and decides whether to retrain based on its performance:

from prefect import flow, task
from sklearn.datasets import load_diabetes
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error
import pandas as pd


@task
def load_data():
    data = load_diabetes(as_frame=True)
    df = data.frame
    print(f”Dataset loaded with shape: {df.shape}”)
    return df


@task
def train_model(df: pd.DataFrame):
    X = df.drop(columns=[”target”])
    y = df[”target”]
    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
    model = LinearRegression()
    model.fit(X_train, y_train)
    preds = model.predict(X_test)
    mse = mean_squared_error(y_test, preds)
    print(f”Model trained. MSE = {mse:.3f}”)
    return mse


@task
def evaluate_performance(mse: float):
    if mse < 3000:
        print(f”Great performance! MSE = {mse:.2f}”)
    else:
        print(f”Needs improvement. MSE = {mse:.2f}”)


@flow(name=”Intro ML Orchestration Flow”)
def diabetes_flow():
    df = load_data()
    mse = train_model(df)
    evaluate_performance(mse)


if __name__ == “__main__”:
    diabetes_flow()

Understanding the Flow

1. load_data() downloads the diabetes dataset and returns it as a DataFrame.

2. train_model() trains a linear regression model and computes Mean Squared Error (MSE).

3. if–else branching inside the flow checks the MSE:

   • If the model performs well → trigger a success notification.

   • Otherwise → print a retraining message and handle fallback logic.

4. notify() logs or alerts based on the outcome.

What’s powerful here is that you can write plain Python conditions, and Prefect still manages the orchestration, dependencies, and logging behind the scenes automatically building a DAG (Directed Acyclic Graph) of tasks.

Prefect vs. Airflow — The Simplicity Advantage

Unlike Airflow, which requires static DAG definitions, special operators, and a complex setup, Prefect lets you orchestrate workflows using pure Python code.
No YAMLs, no boilerplate just decorators and logic you already know.

You can run it locally, visualize it in the Prefect UI, and scale it to production all without changing a single line of code.
That’s why Prefect is often called “the Airflow you’d actually enjoy using.”

Useful Links

• Prefect Homepage

• Installation Guide & Docs

• Core Concepts

I hope this gave you a clear idea of how orchestration works and how easy Prefect makes it to build production-ready ML pipelines. I wrote this for beginners who want to move beyond notebooks and start thinking like engineers.

This isn’t a sponsored post I just stumbled upon Prefect recently and loved how simple it feels.

If you’re new here, hit subscribe! I’ll be sharing more posts on ML, LLMs, MLOps, and career insights that actually make sense in the real world.

Thanks, see you soon ;)

Follow me on X: @kmeanskaran

Subscribe now

Everyone knows it is a well-known term in the tech industry. It is often seen as DevOps for machine learning, right? But it is not that straightforward. In this blog, I will explain why you need MLOps for ML projects and how you can design a system using MLOps.

Why MLOps?

We all use Jupyter Notebook or Google Colab to code our machine learning projects. We install the dependencies, import them, write the code, and run it step by step in cells.

In more ML terms:

1. Install dependencies or libraries like NumPy, Pandas, and Torch.

2. Import them in the top cells.

3. Import data using Pandas, clean it, apply normalization techniques, and do a train-test split.

4. Import models from Torch or Scikit-learn.

5. Train the model.

6. Evaluate the model and plot a confusion matrix or regression line.

7. Save the notebook and the model, then feel like we know machine learning.

From point 1 to point 6 everything works well, but look at point 7. We save the models in binary files (.pkl, .pth), but these files do not perform any real-time tasks. So how can we say our ML project is complete?

This is where MLOps comes in. Before moving forward, let me explain. MLOps is a general term, but when you work on a project you usually work with pipelines. A pipeline is a sequence of modules or functions that move data and actions from start to end. So instead of learning MLOps only in general terms, let us dive deeper and understand what a pipeline is for machine learning projects.

Frame the problem statement

The first step is to understand the business objective. Ask what the product is expected to achieve. If it is a B2C product that customers will use directly, low latency becomes a priority because users expect fast responses. If it is a B2B product meant to automate business tasks or support decision-making, then accuracy and reliability are more important than speed.

Next, look at the data. This is where you will find both the problems and the solutions. As a Machine Learning Engineer, I highly recommend learning statistics and feature engineering because they are key to building strong models.

Once you understand the business expectations and the patterns in the data, you reach the most critical stage of the ML pipeline. At this point, you can start experimenting with state-of-the-art algorithms, unless your company is focused on research. In my practice, if the data is tabular (classification, regression, or even time-series), I usually start with XGBoost. In 2025, XGBoost remains one of the most powerful algorithms for classic ML problems. For text data, you can use BERT or Sentence-BERT, and for text generation you can simply load pre-trained LLMs like Llama, Qwen, and others.

The approach should be to first explore the data and do simple model training. Then evaluate the model’s performance and discuss the metrics with your team.

Remember, machine learning metrics matter only to the ML team. The business side cares about numbers and growth. So even if your model shows 90% accuracy with good precision and recall, it means the model is ready for A/B testing, not for direct production.

Turning a notebook into pipeline

This is where actual MLOps begins. Many beginner ML engineers think machine learning is only about building a model and saving it on their system. In real-world projects, we take the messy notebook code and convert it into a clean, modular Python format.

Here’s what you can do:

1. Break each part of your project into separate pipelines. For example, have one for importing data, another for cleaning data, one for feature engineering, one for splitting the data into train, test, and validation sets, and others for training and evaluation.

2. Write separate Python files for each module, and make sure you have a good understanding of Python modules and object-oriented programming (OOP).

3. Create a main script that runs everything in sequence.

These are some of the most important things to consider before diving into MLOps.

Complete MLOps cycle

Here’s the 10 essential steps of MLOps lifecycle👇

1. Problem Definition & Data Collection: Every ML project starts with a clear goal, like predicting sales or detecting fraud. Once the problem is defined, we collect raw data from databases, APIs, sensors, or logs. Data is the foundation, so gathering the right and reliable data is important.
Tools: SQL, MongoDB, Kafka, Google BigQuery, APIs.

2. Data Cleaning & Preprocessing: Raw data often has missing values, duplicates, and errors. Cleaning makes it reliable by filling gaps, removing noise, and standardizing formats. Preprocessing also includes normalizing and splitting data into training, validation, and testing sets.
Tools: Pandas, NumPy, PySpark

3. Data Versioning & Storage: Data keeps changing over time, so versioning is needed to track changes and ensure reproducibility. Storing processed datasets securely also makes collaboration easier. This step avoids confusion between different dataset versions.
Tools: DVC, Git-LFS

4. Model Development: Here, data scientists experiment with ML algorithms and architectures. They train models, compare performance, and tune parameters to achieve the best results. This step is like the “research” part of MLOps.
Tools: PyTorch, TensorFlow, Scikit-learn, HuggingFace, XGBoost

5. Experiment Tracking: Many experiments are run with different settings, so tracking results is necessary. Tracking helps compare metrics, hyperparameters, and outcomes to choose the best model. This avoids confusion and makes research organized.
Tools: MLflow, Weights & Biases, Comet.

6. Model Validation & Testing: Before deployment, models must be validated on unseen data to check accuracy, fairness, and robustness. Testing ensures the model is not biased and works well under real-world conditions. This step prevents failures later.
Tools: pytest

7. Model Packaging & CI/CD: Once the model is ready, it is packaged into a deployable format (like Docker containers). CI/CD pipelines automate testing, integration, and deployment, reducing manual work. This makes the system reliable and repeatable.
Tools: Docker, GitHub Actions, Jenkins, CircleCI.

8. Model Deployment: Models are deployed into production so users or applications can use them. Deployment can be batch (scheduled jobs) or real-time (API-based). Proper scaling is also important to handle many requests.
Tools: FastAPI, Flask, Kubernetes, AWS Sagemaker, GCP Vertex AI.

9. Monitoring & Logging: After deployment, models need monitoring to check performance, accuracy, and system health. Logs capture errors and unusual patterns, while monitoring helps detect model drift and data changes. This ensures reliability.
Tools: Prometheus, Grafana, ELK Stack

10. Continuous Training & Feedback Loop: Over time, data and user behavior change, so models must be retrained regularly. Continuous training uses new data to keep models updated. Feedback from users also helps improve accuracy and usefulness.
Tools: Airflow, Kubeflow, Prefect, MLflow Pipelines.

Bonus:

MLOps is not just about tools. It is more about the practices you follow in your machine learning project to develop, deploy, and monitor systems quickly. As data keeps growing, we need to automate projects at the same pace, and this is where MLOps becomes important. The pipeline shown above is not fixed. It depends on your company and the tools being used.

If you are a beginner, I highly recommend learning Python modular coding and Docker at the very least. Learning FastAPI is also a great choice, and understanding how system design works in software engineering is essential. Concepts like response models, APIs, and rate limiting are very important for machine learning projects too.

In applied ML, where MLOps is most relevant, you need to be stronger in software engineering skills than in pure ML.

Sources:

1. Book: Designing Machine Learning Systems by Chip Huyen

2. Marvelous MLOps Youtube Playlist

3. YouTube playlist on complete MLOps in Hindi

I am writing more about MLOps, so you can subscribe to my Substack K-Means Karan where I plan to share detailed technical content on MLOps. I am also building a project on MLOps for beginners and intermediate engineers.

Do follow me on Medium for more content. You can also connect with me on X and LinkedIn, where I share thoughts on ML, MLOps, and my career journey.

Thanks for reading, share this article, and see you soon!

Choosing ML as journey, failed 12 interviews, laid off, freelancing, technical writing, and remote AI lead. Let’s unzip my journey.

Table of Contents

1. Pre Tech Journey

2. The First Code

3. Choosing Machine Learning

4. First Interview for ML Intern

5. First Internship

6. Unemployment and MLOps

7. Laid off within a Month

8. Freelance and Build in Public

9. From Remote Hire to Leading AI Team

10. Takeaways (Don’t miss this)

Pre Tech Journey

In 2019, I appeared for the JEE exam, hoping to get into a B.Tech college, but I didn’t make the cut. At that point, I had no clue about coding or even how a laptop worked. Still, everyone around me was talking about Computer Engineering, and their excitement pulled me in. I decided to pursue it too.

I filled out the form for a B.Sc. in Computer Science. Due to late admissions and limited seats, I had to pay a donation fee to get in. Eventually, I got admission into Garware College, under Pune University. It wasn’t a dream start, but it was a beginning.

By April 2020, I had learned a bit of C programming and SQL through college lectures. But I struggled and ended up scoring just 8 out of 50 in the programming exam. Then the pandemic hit. Colleges shut down, exams got cancelled, and to my surprise, even students who had failed were promoted.

The First Code

In July 2020, I bought my first laptop, finally stepping into the world of coding. Until then, during my first year of college, I was learning to code using just pen and paper. I didn’t even know how to install software, but before the laptop arrived, I had already started learning Python.

Python quickly became my comfort zone. It’s still my primary language today. Looking back, I believe it’s better to stick with one language in the beginning instead of constantly switching. It helps you build depth and confidence.

With help from a college friend, I set up VS Code and got Python running. From there, I began following the Python playlist by Telusko on YouTube. I genuinely enjoyed coding in Python, but like many beginners, I got distracted. People around me kept saying, “You need C++ if you want to get a job in IT.” So I gave in and started learning C++.

But honestly, I never really clicked with it. Concepts like calloc, malloc, and pointers just didn’t make sense to me, no matter how many times I tried. Eventually, I went back to Python.

I began exploring data structures like arrays, stacks, and linked lists in Python. Even though I didn’t enjoy DSA much, I kept pushing through because of peer pressure. Everyone kept saying it was crucial for getting placed.

By the end of 2020, I had spent nearly five months learning libraries like Pandas, NumPy, and Matplotlib. I didn’t even know what machine learning was back then. I was just following Python tutorials and reading documentation on W3Schools, moving step by step with whatever I could find.

Choosing Machine Learning

One day, I was searching on Google and saw a list of the highest-paying jobs of the 21st century. One job that caught my eye was “Data Scientist.” I had heard this word before in a Python video, but now I wanted to learn more.

I searched for a roadmap to become a data scientist and found something new — Machine Learning. It sounded interesting. I started watching beginner videos on YouTube from channels like Great Learning and Intellipaat to understand what it was.

I began with simple topics like Linear Regression, Logistic Regression, Decision Trees, and K-Means. That’s also when I created my own online name k-means karan (you can follow me on X). I started using Jupyter Notebook to play with data. I was learning how to clean data, choose features, and train models.

After a few months, in mid-2021, I found Kaggle a website where you can practice real machine learning problems. I spent a lot of time there, and it helped me understand how to look at data and think about the problem before writing code.

Here is my GitHub where I share the projects I worked on:
🔵 github.com/karan842/Data-Science-Projects

After learning basic machine learning and some statistics, I wanted to go deeper. I started learning about neural networks. At that time, my math wasn’t strong, so it took me almost 4 months to understand how they work. Slowly, I learned about different types like ANN, CNN, RNN, and LSTM.

To learn the basics of statistics, machine learning, and neural networks, I followed Krish Naik’s YouTube videos. I also added neural network projects to the same GitHub repository.

Later, I started reading books on machine learning. Books helped me understand how models actually work inside. I believe reading technical books is a great way to build strong knowledge and improve your skills.

First interview for ML Intern

I still remember when I got my first interview call for a Machine Learning internship in May 2022. I was very excited and prepared for it for a whole week. But once the interview started, I realized how unprepared I actually was. I couldn’t even speak confidently during the introduction. I was nervous and didn’t know what to expect.

The interviewer asked me to share my screen and solve a problem in Python. It was a basic question about checking if two words are anagrams. I only managed to write a simple for loop and couldn’t figure out the full logic. After that, he started asking more technical questions. He asked about neural networks, XGBoost, CNNs, how inference works, and even Transformers and BERT. I couldn’t answer most of them.

Then another person joined the call. She started asking questions about data engineering, databases, OLAP, and OLTP. I had no idea what to say.

The next night, I received an email from HR with the usual rejection message:

“Sorry, we’re not moving forward at this time.”

I felt disappointed and lost a bit of confidence. But instead of giving up, I started preparing again. I used pen and paper to go through machine learning basics and started practicing algorithms using Python. To be honest, I still don’t enjoy solving problems on LeetCode, but I know it is important for interviews.

That interview did not go well, but just one week later, I got a message from Internshala.

First Internship

After a week, I got a call from the CEO of a fintech company based in Pune. He spoke kindly and offered me a position. The surprising part was, he didn’t ask any technical questions or even check my code. I had just finished my final semester exams of B.Sc. CS, so I accepted the offer and joined the startup.

The office was around 35 to 40 KMs from my home. I had to take two buses every day to reach there. On my first day, I met a senior, the HR, and the CEO. During our introduction, the CEO told me I would be working on the backend using Django.

I was shocked. The role was offered as a Data Science internship, but now I had to work on backend development. I didn’t know Django and had no interest in backend work. Still, I tried to give my best.

For the next three months, I struggled a lot. I was not able to perform well, and I knew it. Along with me, three other interns also left the company. Finally, I gathered the courage to call the CEO and told him I wanted to leave. He refused to give me my last month’s stipend and also did not provide a relieving letter.

Just four days later, I got another opportunity. I joined quickly because I hadn’t enrolled for a Master’s degree and didn’t want to stay unemployed.

But this new company turned out to be a scam. I was working in a small room with a single person who called himself the CEO. There were no real projects, no team, and no proper work. On the second day, I left the place. It was a scary and bad experience. Since I never received an offer letter, I had nothing in hand. No salary, no experience, nothing.

Unemployment and MLOps

After the internship experience, I was unemployed. Most of my friends had either started a new degree or were already working somewhere. That period was very tough for me. My daily routine was boring and repetitive — waking up early, going to the gym, updating my resume, and applying for every possible job on LinkedIn, Naukri, and Indeed.

One day while reading job descriptions, I came across a new word — MLOps. I didn’t know what it meant, so I searched for it on YouTube and started learning. I found a video by Krish Naik about MLOps and began following his content regularly.

At the same time, I tried applying what I learned. I took one of my Kaggle projects, an NLP-BERT notebook, and converted it into a basic MLOps workflow. It wasn’t a full MLOps setup, but it was my way of exploring the field step by step.

Here are the blog posts I wrote during that time:
🔹 Design ML System for NLP BERT + MLOps
🔹 ML in Production: A Practical Beginner Guide

These blogs also include GitHub repositories, so feel free to check them out.

Those unemployment days were hard, but I promised myself not to waste a single day. I kept learning, kept building. During that phase, I gave around 12 interviews and failed all of them. But I didn’t stop.

By January 2023, I was feeling mentally tired. That’s when I decided to go for my Master’s degree starting in June. I also made a habit of sharing whatever I was learning on LinkedIn. That habit is still helping me today I now get messages from CEOs and HRs noticing my work.

I’ll be sharing more about MLOps strategies and practical ideas in my upcoming blog posts. If you’re interested, don’t forget to subscribe to my Medium newsletter.

Laid off within a month

In February 2023, I finally got another internship offer, and I joined as soon as possible. During the technical interview, I secretly searched the internet for the syntax of some Pandas operations to pass a few test cases. Somehow, I cleared the round and felt excited to work with this new company.

On March 1, I joined a Zoom call with the India head. That call changed how I saw machine learning completely. During our conversation, I confidently said, “I’m a Kaggle notebook master with 5 medals.” He smiled and replied, “That’s toy machine learning. In the real world, you need to work on every part of the pipeline.” That one line hit me hard and gave me the push I needed to grow.

On my first day, I was given a CSV file and a task outline. I had to clean and process the data using a Pandas script so it matched the expected output. It looked simple, but I couldn’t solve it on my own. It took me two weeks, and I still failed to finish it properly. One day, I even went out without informing my manager. Because of that, I received a warning for being unprofessional.

March went by, and I was still stuck on the same task. It felt impossible. Back then, ChatGPT-3.5-turbo had just launched, and it wasn’t as helpful as it is today. I couldn’t upload files or explain my errors well, and I kept going in circles.

Even though my manager knew I was copying code, he still gave me another chance. He introduced me to a new tool called LangChain, which was very new at the time. I enjoyed exploring it, but relying on ChatGPT for coding didn’t help me much. I needed to understand the logic better.

Then came April 7, 2023 — a Friday morning. Around 10 AM, my manager called me. He told me that they were letting me go because of poor performance and lack of sincerity. The next day was my birthday. That Friday afternoon, I cried a lot. It felt like a heavy failure. I couldn’t believe that since finishing college, I still hadn’t landed a stable offer.

But looking back, that experience was a big lesson. I’m actually thankful I faced these rejections early in my career. They gave me the strength and confidence to handle bigger challenges later in life.

Freelancing and Build in Public

Just a week after my last internship ended, I got a call from two of my B.Sc. professors. They were working on their PhD research at Pune University and needed help with technical experiments. That call became my first real opportunity to earn money as a freelancer.

I started working with both professors at the same time, helping them with different research projects. They shared research papers and GitHub repositories, and my job was to run the code, test different models, and suggest the best technical solutions for their thesis work.

Until August 2023, I continued working with them. They paid me $150 for each experiment and later gave me $950 for completing the full project. Around the same time, I also worked with a Master’s student from Mumbai University to build a basic RAG (Retrieval-Augmented Generation) app.

In August, something unexpected happened. I received a message from the founder of one of Pune’s biggest data science companies. He was planning to build an AI startup and wanted to discuss the idea with me. I met him and his business partner, and we had a good conversation. But the very next day, I received another offer — a contract-based ML Engineer role at a small startup. Because I needed money urgently, I chose to join the startup and had to decline the AI startup offer.

Well, that’s life. You don’t always get to choose both.

The startup I joined turned out to be a blessing. I learned more than I could ever measure from MLOps on real-estate datasets with millions of entries to research work and neural networks. I was their youngest team member, and the CEO and his wife, who ran the company together, had over 30 years of experience in tech and AI, mostly in the US. They treated me like family and taught me lessons that felt more like guidance from a father figure.

At the same time, I had enrolled in a Master’s degree in Computer Science at the same college where I did my B.Sc. One of my freelance clients was a professor there, and with his help, I got admission in the last round without much effort.

Life became very busy. I was managing both college and office, changing buses every day, walking long distances, and still coding in the evenings. But that was a golden period for me. I was learning everything I had once only dreamed of — and that phase truly shaped my future.

I also continued sharing my journey on LinkedIn. That habit helped me get freelance projects and job opportunities. Later, I started writing on Twitter, which opened new doors in consulting and technical writing.

In December 2023, I got my first paid freelance project to write technical content about Generative AI and LLMs. You can find those articles in my previous Medium publications. I’ve realized I’m pretty good at technical writing and I hope you’re enjoying this story as much as I enjoy telling it.

From Remote Hire to Leading AI Team

In April 2024, I received an offer for a Junior Machine Learning role from a Canadian startup. I accepted it without thinking twice. At that time, I had just one year left to complete my M.Sc. in Computer Science. The job was fully remote and offered better pay than both my previous contract-based work and freelance writing combined. So, I joined.

Working remotely turned out to be a turning point in my career. I learned much more than just machine learning. I got hands-on experience with distributed training, reinforcement learning, and many other advanced topics. As time passed, I started picking up skills beyond coding — project management, task ownership, and even leading a remote team of engineers.

That part changed everything. Being able to work remotely and still lead a team made me more confident and trustworthy in the eyes of other employers and CEOs. Today, I’m proudly leading a team of AI engineers — all remotely.

Over the months, I also developed strong problem-solving skills. I started solving real-time business problems using ML, not just toy datasets. In September 2023, I traveled to Delhi for a Hackathon at Mastercard’s office. The trip was sponsored by my company, and it was a great chance to meet fellow developers. Since then, I’ve been attending AI/ML meetups regularly in Pune and Bengaluru to learn, connect, and grow.

Now, I finally have a stable job and income. But life has brought me new challenges at a different level. I often share these experiences on my Twitter so others can learn from them too. Remote work comes with its own set of struggles and excitement. I’ll write about that in a separate blog sometime soon.

Right now, I’m focusing on writing more technical blogs around Machine Learning, MLOps, and topics like Federated Learning. So stay tuned if you’re curious about what’s next.

Takeaways (Don’t Miss This)

Here are some of the key lessons I’ve learned along the way. If you’re just starting out, these might help guide your journey:

1. Machine Learning

• Don’t just stick to Jupyter Notebooks or Google Colab. Try to build complete ML pipelines from start to finish. Learn how to deploy your models and make them useful in the real world.

• Good data leads to better results. Spend more time collecting quality data and doing feature engineering instead of chasing complex or fancy models. (These days, LLM research feels like a leaderboard race, but basics still matter.)

• Focus on learning statistics, math, the basics of machine learning, deep learning, and transformers. These are the foundations that will stay relevant, even when today’s SOTA (state-of-the-art) models are replaced tomorrow.

2. Career

• Don’t just dream about a job title or a big company. Dream about the kind of work you actually want to do. That makes all the difference.

• You can have 1000 strong skills, but if you don’t share them online, nobody will know. Build your presence, talk about your work or you’ll miss real chances.

• CEOs are interesting people. If you’re ambitious, they’ll notice it and share real startup stories and life lessons that you won’t hear anywhere else.

• Stay healthy. If you don’t take care of your body, your confidence will drop. And in today’s world, people do respect those who look and feel active.

• Learn how to communicate well. Build your storytelling skills. Honestly, this is even more powerful than machine learning skills. The way you explain your journey, ideas, and impact can open more doors than any algorithm.

Thanks for reading my journey. I hope it didn’t feel boring because every line here comes from real struggle, learning, and growth.

I’ll be writing more technical blogs soon on topics like Machine Learning, MLOps, and real-world projects. If you’re interested in those, stay tuned.

Let’s connect on Twitter (X), where I regularly share my learnings, experiments, and experiences: @kmeanskaran

Also, feel free to visit my website if you want to know more about my work in detail. I’m always open to collaborations, feedback, and conversations.

Freelance, layoff, and remote AI Engineer — My tech journey was originally published in Level Up Coding on Medium, where people are continuing the conversation by highlighting and responding to this story.

“Nobody cares your AI model or architecture if your UI is ugly. — Unknown”

Disclaimer: This blog reflects my personal experience and isn’t a guarantee that this approach will work for everyone. It’s not a Next.js tutorial or a guide on how to learn Next.js. Instead, it’s about why Next.js is a valuable secondary skill for freelancing and how it can set you apart from other AI engineers by enabling you to build full-stack applications with practice. Note that, I already have an idea about React JS.

Client’s website: https://www.shreeshotblasting.com/

Introduction

When we say, “I am an AI/ML Engineer,” it usually means we’re backend engineers who work on AI models. Yes, that’s our role in the real application system. Let’s be honest: most AI engineers don’t like frontend development, and I used to be one of them. For four years of my coding journey, I only focused on AI and Machine Learning. It felt rewarding, but after getting a job and some expertise in one area, I saw the value in exploring other tech skills too. For AI Engineers (or ML Engineers — the terms are often the same), learning frontend development can be really worthwhile. In this article, I’ll show you how to learn frontend development from the view of a Python fan and an AI engineer who thinks frontend is actually pretty simple.

Why Next JS?

Streamlit might look like a toy project for quick prototypes, but if you’re serious about building quality, stand-out UI applications, it’s time to learn a real-world UI framework like Next.js. So, why is Next.js the best framework out there? It’s special because it takes the power of React.js is a popular library for building dynamic user interfaces and supercharges it with features like server-side rendering (SSR), static site generation (SSG), and a built-in routing system, all out of the box.

Unlike React.js, which is just a library focused on the view layer, Next.js is a full-fledged framework that simplifies development by handling optimizations, SEO, and performance without requiring you to stitch together countless third-party tools.

You might wonder: if React.js exists, why bother with Next.js? The answer is simple — React.js gives you a blank canvas, but Next.js provides a structured masterpiece, saving you time and effort while delivering faster, more scalable applications. It’s the difference between assembling a car from scratch and driving one off the lot, tuned for speed and reliability. That’s why Next.js stands out as the go-to choice for modern developers.

Where to Start?

I know most AI developers hate working on the frontend! We can easily understand complex things like the transformers architecture — some of us can even build it from scratch — but we still think HTML and CSS are the hardest and most boring parts (especially CSS). I’ll admit, even now, I don’t enjoy learning CSS properties. But thanks to tools like ChatGPT, Claude, and others, writing CSS scripts has become much easier.

Nowadays, you don’t need to learn everything from top to bottom. Just get a basic idea of how an application works. Spend 3–4 hours this upcoming weekend and watch a simple video. HTML and CSS might feel boring at first, but once you start learning a language like JavaScript (JS), you’ll love working with it. It’s nothing like a Python script — it’s exciting and fun!

You know what is common between JavaScript and Python devs?

Both hates Java😝.

Learning JavaScript is one of the best investments you can make right now — it’s like investing in gold! The whole internet runs on JavaScript. You can read a simple document (click here) to learn about JS syntax and everything you want to know. I think using AI tools is a better way to learn any language instead of spending hours watching YouTube videos or reading long documentation.

So, after spending two weekends learning HTML, CSS, and JS, it’s time to dive into the best JS library out there: React.js! React.js is where you should really focus your learning efforts. To get started with React.js, I recommend watching this video. Try building 2–3 small projects that go from basic to advanced concepts in React.js. Your goal is simple: understand how things work and how to connect the dots to create a full AI app. Once you finish this step, it’s time to move on to Next.js!

Next JS Ecosystem

After learning JavaScript, spend a day or less on TypeScript (TS), a superset of JS with added typing. Grasp its basics like types and interfaces in a few hours. Then, explore Tailwind CSS, a utility-first framework for efficient styling, and check out ShadCN UI for customizable UI components built on Tailwind. This gives you a quick, solid start in TS and modern front-end tools.

• TypeScript

• TailwindCSS

• Shadcn/UI

You don’t need to master everything — just aim to understand what TypeScript, Tailwind CSS, and ShadCN UI are and how they function. Your goal is to integrate them with Next.js.

Now, it’s time to dive into Next.js, a powerful framework! Start with this simple tutorial (click here) to explore its features, like app-based routing, and learn the difference between pages and layouts. This will give you a clear idea of Next.js and its related libraries.

The real fun begins here: leverage AI to handle coding and debugging with prompts, saving you hours of effort!

Magic of V0

Vercel has recently launched v0, a revolutionary UI engine that generates user interfaces using Next.js based on simple user prompts. This tool is a game-changer, saving developers countless hours by eliminating the need to build projects from scratch. Even with the free tier, users get a generous number of prompt requests, sufficient to create basic structures, sections, pages, and responsive navigation bars. v0 generates TypeScript files (.tsx) that leverage shadcn and Tailwind CSS, allowing developers to focus on understanding functionality rather than wrestling with syntax. Once your project is ready, you can easily clone it locally with a single npm command — details for which can be found via ChatGPT’s search mode or Vercel’s official documentation.

$150 Story

One of my family friends recently approached me about establishing an online presence and needed a website built. At the same time, my gym membership renewal was looming, and I had to manage my finances to cover it. I promised them I’d deliver a fully functional website within four days. Coincidentally, I was learning Next.js at the time and had just come across Vercel’s v0 UI engine through a post on X. Intrigued, I opened my laptop, switched off my phone, and dove straight into the task.

On the first day, I used v0 to create the basic structure of the website. I experimented heavily with prompts to design the layout and components, which saved me a ton of time. By the second and third days, I cloned the project locally from Vercel using a simple command and opened ChatGPT in another tab for additional support. From there, it was a breeze. I rebuilt individual components like Home, About, Industries, and Contact, then populated them with the company data provided by my client. I even synced the contact form with their Gmail account. Hosting the site on Vercel was seamless. Vercel’s CI/CD pipeline automatically built the Next.js app with every new commit to the branch.

After hosting, I shared the live site with my client for feedback. They suggested a few tweaks, which I quickly implemented. On the final day, they provided me with their domain name and GoDaddy login credentials. I spent two hours researching how SEO works with Next.js, then added a sitemap.xml file to ensure Google could crawl the site. After linking the hostname, the website went live in just 20 minutes.

In the end, they paid me $150 enough to renew my gym membership. While this was far below market rates, I offered them lifetime website updates, Google Business setup, and ongoing maintenance as part of the deal. In just four days, I leveraged AI tools like v0 and ChatGPT to deliver a professional, fully functional website for my client. This experience showed me the power of modern tools to streamline development and turn a tight deadline into a success story.

More in Next JS

Next.js offers an impressive array of functionalities, far beyond what I initially tapped into. So far, I’ve only used it to create a business landing page, but I’m now diving deeper into its potential for building AI-driven applications or platforms. My goal is to integrate my FastAPI backend which powers AI functionalities with a Next.js frontend to create a seamless user interface. Next.js even comes equipped with tools like NextAuth.js for authentication, making it a versatile choice for full-stack development. As I explore these capabilities, I’m excited to see how Next.js can bridge my AI backend with a dynamic, user-friendly frontend.

If you’re using Figma, you can simply upload your Figma file to Vercel’s v0 platform, and it will generate a UI for you automatically. This integration streamlines the process, turning your designs into functional interfaces with minimal effort.

Final Thoughts

As an AI engineer focused on training machine learning models and building multi-agent applications, I’ve come to appreciate the value of a clean, functional UI. Beyond my own projects, I’ve realized I can support small businesses by creating SEO-optimized landing pages to boost their online growth. While AI and ML remain my primary passions, Next.js has become a powerful secondary skill. I no longer spend hours coding from scratch instead, I leverage existing AI tools to generate websites in mere minutes. In the world of AI, nothing feels out of reach; it all boils down to your interest and commitment. My advice? Don’t confine yourself to a single tech stack embrace the flexibility to explore and adapt.

Explore new technologies and remember:

Focus on understanding the workflow and identifying where it’s needed — not on memorizing syntax.

I hope you enjoyed this blog! It’s my first time sharing a story from my tech career, and I’m excited to kick things off this way. Moving forward, I plan to write more about my experiences in tech, offering insights to help others figure out where to start — and, more importantly, how to finish their work efficiently. That’s all for today! If you have any questions or feedback, feel free to let me know.

Connect me on,

LinkedIn and X. Also check out MY WEBSITE which again I built using V0.

See you soon👋!

Every AI Engineer should learn Next JS. was originally published in Level Up Coding on Medium, where people are continuing the conversation by highlighting and responding to this story.

In this article, I’ll show you how to optimize your Python environment, especially for machine learning projects, using UV, the next-generation package manager. Managing dependencies in ML projects can often feel like a daunting task. We’ll also explore how to install CUDA-accelerated PyTorch in a virtual environment (uv) and run it seamlessly within a Dockerfile. Issues such as broken packages, version mismatches, and slow installations can significantly hinder your productivity during project development.

But don’t worry! We’ll explore how UV addresses these common challenges, ensuring faster, more reliable package management to keep your focus on building and innovating.

Installing UV

This tutorial is based on my experience with Ubuntu 22.04, and it should work as of today. For the latest information and updates, please refer to the official documentation of Astral UV.

For macOS and Linux:

curl -LsSf https://astral.sh/uv/install.sh | sh

For Windows:

powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"

You can also use `pip` to install UV!!

pip install uv

(Ironically, the only time PIP is still useful is to install UV! Once set up, UV takes over, speeding up your package management for python projects.)

After installing uv, you can check that uv is available by running the uv command:

uv

You should see a help menu listing the available commands.

Set up environment

Once you’ve completed the basic UV installation, you can easily specify the Python version for your project. Unlike traditional PIP, UV allows you to streamline this process without extra tools or steps. In contrast, PIP doesn’t support direct installation of newer Python versions using a simple command. (If you know a workaround for this, feel free to share it in the comments below!)

By running,

uv python install 3.12

You can easily install Python 3.12 or even set up multiple Python versions for development. Simply run the following command:

uv python install 3.11 3.12

This command installs both Python versions on your system, allowing you to seamlessly switch between them for development. To see all present Python versions in your system:

uv python list

Now let's initialize the project and environment.

uv init uv-sample

The above command initializes the UV environment for your project, with ‘uv-sample’ as the project name. Once executed, it generates several files within the project directory. But before exploring those files, let’s quickly set up a Python environment within the ‘uv-sample’ directory.

uv venv

After creating the .venv environment in your project folder, it’s time to explore the files inside the directory.

What UV offers you?

You’ll notice some familiar files like .gitignore and README.md (I’m sure you’re already familiar with their purpose!). Additionally, there are other files that are both intriguing and serve as the backbone of the UV environment, making it more efficient and faster.

• .python-version shows the current Python version of the project environment.

• pyproject.toml file is a configuration file in Python projects that defines project metadata, dependencies, and build system requirements, ensuring streamlined project setup and compatibility.

• The uv.lockfile records the exact versions of all dependencies in your project, ensuring consistent and reproducible environments across different setups. You will see this file after installing dependencies.

Install dependencies

Let’s install some of the Python packages, libraries, or frameworks. For that, simply run the below command by defining their names.

uv add fastapi uvicorn groq langchain-groq numpy requests

The main reason to use UV over pip is its parallelism. UV installs all the required components simultaneously, significantly speeding up the installation process while reducing the likelihood of conflicts.

After installing these packages, you can find them in pyproject.toml .

[project]
name = "uv-sample"
version = "0.1.0"
description = "Add your description here"
readme = "README.md"
requires-python = ">=3.12"
dependencies = [
    "fastapi>=0.115.6",
    "groq>=0.13.1",
    "langchain-groq>=0.2.2",
    "numpy>=2.2.1",
    "requests>=2.32.3",
    "uvicorn>=0.34.0",
]

Running Python file

After running the⁣ uv init <project-name> command, you will notice a hello.py file created by UV. As of today, this file contains some predefined content.

def main():
    print("Hello from uv-sample!")


if __name__ == "__main__":
    main()

Since the Python virtual environment is already activated, you can run the script using python hello.py. However, there's another way to execute the Python file using UV.

uv run hello.py 

These are some basic setup steps for UV. For more detailed information, please refer to their official documentation.

Create an API

Now, let’s create a simple FastAPI application that takes a user query and generates a response using the Groq API.

(I renamed hello.py to app.py )

from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
from langchain_groq import ChatGroq
import os 
import uvicorn

# Initialize FastAPI
app = FastAPI()

# Initialize ChatGroq LLM
llm = ChatGroq(
    model="mixtral-8x7b-32768",
    api_key='YOUR API KEY',
    temperature=0,
    max_tokens=None,
    timeout=None,
    max_retries=2,
    # Add additional parameters if necessary
)

# Define request schema
class ChatRequest(BaseModel):
    human_message: str

# Define response schema
class ChatResponse(BaseModel):
    response: str

@app.post("/chat", response_model=ChatResponse)
async def chat_endpoint(request: ChatRequest):
    """
    Handle user input and get a response from ChatGroq.
    """
    try:
        # Prepare messages
        messages = [
            (
                "system",
                "You are a helpful assistant. Response general questions from users.",
            ),
            ("human", request.human_message),
        ]
        
        # Call ChatGroq LLM
        ai_msg = llm.invoke(messages)
        
        # Return response
        return ChatResponse(response=ai_msg.content)
    except Exception as e:
        raise HTTPException(status_code=500, detail=str(e))

if __name__ == '__main__':
    uvicorn.run(app, host="0.0.0.0", port=5500)

To run the above code: open terminal, activate .venv and execute this command:

uv run app.py

You will see in the terminal that FastAPI is running on the server. Open a new terminal window and run the following command to test:

curl -X POST \
  'http://0.0.0.0:5500/chat' \
  -H 'Content-Type: application/json' \
  -d '{
    "human_message": "Hello, how are you?"
  }'

{"response":"Hello! I'm an artificial intelligence and don't have feelings, but I'm here to help you. How can I assist you today? Is there a specific question you have in mind?"}

That's how you can simply run the Python file.

Create Dockerfile for UV

FROM python:3.12-slim-bookworm

# The installer requires curl (and certificates) to download the release archive
RUN apt-get update && apt-get install -y --no-install-recommends curl ca-certificates

# Download the latest installer
ADD https://astral.sh/uv/install.sh /uv-installer.sh

# Run the installer then remove it
RUN sh /uv-installer.sh && rm /uv-installer.sh

# Ensure the installed binary is on the `PATH`
ENV PATH="/root/.local/bin/:$PATH"

# Copy the project into the image
ADD . /app

# Sync the project into a new environment, using the frozen lockfile
WORKDIR /app
RUN uv sync --frozen

# Export PORT
EXPOSE 5500

# Run file
CMD ["uv", "run", "app.py"]

In the Dockerfile above, I use a basic template for a Python application and manually install uv using commands. This approach offers greater flexibility for future integrations and reduces potential conflicts. Alternatively, you can use the python-uv Docker image. For more information on this, refer to their Docker installation guide.

Install PyTorch with CUDA

If your operating system is GPU-accelerated or you have already installed CUDA on your system, you can simply install PyTorch. Let’s proceed with that.

uv add torch torchvision

By running this command, you can install PyTorch and its related packages. The real advantage, however, is that uv enables parallel downloading, which speeds up the installation process and automatically resolves package dependencies. See this screenshot:

Note: Installation pace depends on your internet speed.

On my device, the installation took around 5 minutes. You can try the same using pip install torch torchvision and compare the installation speed and conflict resolution capabilities of both methods.

After the installation, you can check the pyproject.toml file for the updated dependencies:

[project]
name = "uv-sample"
version = "0.1.0"
description = "Add your description here"
readme = "README.md"
requires-python = ">=3.12"
dependencies = [
    "fastapi>=0.115.6",
    "groq>=0.13.1",
    "langchain-groq>=0.2.2",
    "numpy>=2.2.1",
    "requests>=2.32.3",
    "torch>=2.5.1",
    "torchvision>=0.20.1",
    "uvicorn>=0.34.0",
]

Magic of uv-sync

The⁣uv sync command is my favorite feature of uv. In traditional Python projects, we typically create a requirements.txt file and list dependencies one by one, sometimes specifying versions. After pulling or cloning a repository from version control (e.g., GitHub, GitLab), installing dependencies can be a tedious process. It often involves version mismatches, differing Python package versions, or incompatible pip versions.

I’ve encountered these issues many times in production. Python projects, especially those involving ML packages like transformers, torch, tensorflow, pillow, and others, often require multiple sub-dependencies. When⁣ pip attempts to install newer versions, it may uninstall older ones, leading to conflicts or broken package errors.

In such cases, uv is the perfect solution. If your source code is set up with uv, you simply need to create a virtual environment and run uv sync. This will quickly download all the dependencies, ensuring a smoother and conflict-free installation process.

Final Thoughts

I use uv in my latest ML projects, and it has significantly simplified my workflow. It allows me to focus more on the project itself rather than dealing with complex setup processes. I encourage you to try both the traditional pip method and the uv method, then decide which one suits your preferences better.

Remember, software engineering is a gray field—sometimes things work perfectly, and other times they don’t. While uv is a powerful tool, there might still be some cons or limitations you could encounter. For more details and updates, check their official website or GitHub repository for the latest improvements.

That’s all for today! This year (2025), I’m planning to write more engaging blog posts—not just on Python, LLMs, and MLOps, but also on broader tech explorations from my learning journey. Don’t forget to check out my new Substack newsletter, K-Means Karan.

If you enjoyed this blog, I’d love to hear your feedback in the comments. Give it an upvote and connect with me on LinkedIn or X. Let’s grow and learn together!

Boost ML Projects: Switch from PIP to UV was originally published in Level Up Coding on Medium, where people are continuing the conversation by highlighting and responding to this story.

This article is associated with team AiHello.

Tables of Contents:

• Introduction

• Embedding for Similarity Search

• What is SBERT?

• Installation and setup

• Training SBERT

• Various Methods for Training SBERT

• Test the Model

• Conclusion

• Resources

Introduction

Sentence Transformers is a widely recognized Python module for training or fine-tuning state-of-the-art text embedding models. In the realm of large language models (LLMs), embedding plays a crucial role, as it significantly enhances the performance of tasks such as similarity search when tailored to specific datasets.

Recently, Hugging Face released version 3.0.0 of Sentence Transformers, which simplifies training, logging, and evaluation processes. In this article, we will explore how to train and fine-tune a Sentence Transformer model using our data.

Embeddings for Similarity Search

Embedding is the process of converting text into fixed-size vector representations (floating-point numbers) that capture the semantic meaning of the text in relation to other words. How can this be used for similarity search? In similarity search, we embed queries into a vector database. When a user submits a query, we need to find similar queries in the database.

First, convert all textual data into fixed-size vector embeddings and store them in a vector database. Next, accept a query from the user and convert it into an embedding as well. Then, find similar search terms or keywords from the user query within the vector database and retrieve those embeddings that are closest. Is it simple? Yes, but to search for the closest embeddings, we need to use distance-based algorithms such as Cosine Similarity, Manhattan Distance, or Euclidean Distance.

What is SBERT?

SBERT (Sentence-BERT) is a specialized type of sentence transformer model tailored for efficient sentence processing and comparison. It employs a Siamese network architecture, utilizing identical BERT models to process sentence pairs independently. Additionally, SBERT utilizes mean pooling on the final output layer to generate high-quality sentence embeddings. For a comprehensive understanding of SBERT, I recommend referring to the detailed article.

Installation and setup

You can either use online notebooks such as Google Colab. I have also covered how to execute training code from script. For Google Colab, set your runtime environment to T4 GPU hardware.

!pip install -U "sentence-transformers[train]" accelerate datasets

Import dependencies

import os
import json
import torch
import datasets
import pandas as pd
from torch.utils.data import DataLoader
from sentence_transformers import (
    SentenceTransformer, models,
    losses, util,
    InputExample, evaluation,
    SentenceTransformerTrainingArguments, SentenceTransformerTrainer
)
from accelerate import Accelerator
from datasets import load_dataset

For this blog post, I am using Glue STS-B data and model sentence-transformers/all-MiniLM-L6-v2

data = load_dataset('sentence-transformers/stsb')
train_data = data['train'].select(range(100))
val_data = data['validation'].select(range(100, 140))

In the code block above, I’ve chosen samples of 100 for training and 40 for validation. This decision is due to the limited resources available in the free version of Colab. Feel free to adjust the range size or import the entire dataset as needed.

Let’s see random sample data from train data

# Example data from 5th record (taking randomly to just display)
print("Sentence 1: ", train_data['sentence1'][5], "\nSentence 2: ", train_data['sentence2'][5], "\nScore: ", train_data['score'][5])

Output:

Sentence 1:  Some men are fighting. 
Sentence 2:  Two men are fighting. 
Score:  0.85

This will be the format of our data: ‘sentence1’, ‘sentence2’, and ‘score’. The ‘score’ represents the degree of closeness or similarity between the two sentences. In cases where a label score is unavailable, you simply need to modify the loss function and evaluator accordingly.

Training SBERT

This is recommended way to train SBERT model form SBERT official site.

To train the SBERT model, you need to encapsulate the model building, evaluator, and training processes within the main() function. See this discussion.

Training code:

def main():
   
    # Get number of GPUs working
    accelerator = Accelerator()
    print(f"Using GPUs: {accelerator.num_processes}")

    # Sentence Transformer BERT Model
    word_embedding_model = models.Transformer('sentence-transformers/all-MiniLM-L6-v2')
    # Applying pooling on final layer
    pooling_model = models.Pooling(word_embedding_model.get_word_embedding_dimension())
    model = SentenceTransformer(modules=[word_embedding_model, pooling_model])

    # Define loss
    loss = losses.CoSENTLoss(model)

    # Define evaluator for evaluation
    evaluator = evaluation.EmbeddingSimilarityEvaluator(
        sentences1=val_data['sentence1'],
        sentences2=val_data['sentence2'],
        scores=val_data['score'],
        main_similarity=evaluation.SimilarityFunction.COSINE,
        name="sts-dev"
    )


    # Training arguments
    training_args = SentenceTransformerTrainingArguments(
        output_dir='./sbert-checkpoint', # Save checkpoints
        num_train_epochs=10,
        seed=33,
        per_device_train_batch_size=8,
        per_device_eval_batch_size=8,
        learning_rate=2e-5,
        fp16=True, # Loading model in mixed-precision
        warmup_ratio=0.1,
        evaluation_strategy="steps",
        eval_steps=2,
        save_total_limit=2,
        load_best_model_at_end=True,
        save_only_model=True,
        greater_is_better=True
    )


    # Train model
    trainer = SentenceTransformerTrainer(
        model=model,
        evaluator=evaluator,
        args=training_args,
        train_dataset=train_data,
        eval_dataset=val_data,
        loss=loss
    )
    trainer.train()

    # save the model
    model.save_pretrained("./sbert-model/")

Now. let’s understand each component inside main() function step by step,

• Define the Accelerator() to determine the number of GPUs available on the current machine.

• Load the Sentence Transformers model from the HuggingFace repository and extract the word embedding dimension using mean pooling. Add a mean pooling layer after the SBERT model as output.

• Define the Loss function, such as CoSENTLoss(), to calculate the model’s loss based on float similarity scores. Choose the appropriate loss function from SBERT’s options based on your data and labels. Refer to the Loss Overview in the Sentence Transformers documentation.

• Use the Evaluator() class provided by Sentence Transformers to calculate the evaluation loss during training and obtain specific metrics. Choose the appropriate evaluator, such as EmbeddingSimilarityEvaluator(), based on your data and use case. Refer to this table for available options.

• Specify training arguments, such as the output directory for storing checkpoints, batch size per device (CPU/GPU), number of training epochs, learning rate, float16 precision for model loading, evaluation steps, etc., using the SentenceTransformerTrainer class which is indirectly inherited from transformers TrainingArguments.

• Train the model using the SentenceTransformerTrainer class by defining the training and validation data, optionally including an evaluator, specifying training arguments, and defining the loss function. Initiate training by calling the train() method. Perform training and save the model further.

Various Methods for Training SBERT

After defining the main() function, simply call it to initiate the model training process. There are several ways to do this:

For Single GPU:

• If you are running code in Google Colab free version with T4 GPU then just create a new cell and call function: main()

• If you are running your code in Python script, then just run a python command in the terminal: python main.py.

For Multi-GPU:

HuggingFace transformer supports DistributedDataParallel (DDP) training to perform distributed parallel training on multiple GPU or in multiple machines. Read this article to understand how DDP works.

• If you are running your code in colab or any notebook which contains multi-gpu, then:

from accelerator import notebook_launcher
notebook_launcher(main, num_processes=2)

By running above code in a separate will run your code in multi-gpu.

• For Python Script:

accelerate launch –multi-gpu –num_processes=2 main.py

These are some common ways to run a script or notebook for SBERT training.

Test the Model

After training the model, we can reload it and perform inference testing. For instance, if we have a list of product names and users input search terms, our goal is to identify the most similar product names along with a score.

Having trained our embedding model on sentence similarity data using similarity scores as labels, it will now improve the embeddings.

Here is the sample list of product name which we are using for embedding data:

# List of products
products = [
    "Apple iPhone 15 (256GB) | Silver",
    "Nike Air Max 2024 | Blue/White",
    "Samsung Galaxy S24 Ultra (512GB) | Phantom Black",
    "Sony PlayStation 5 Console | Digital Edition",
    "Dell XPS 13 Laptop | Intel i7, 16GB RAM, 512GB SSD",
    "Fitbit Charge 6 | Midnight Blue",
    "Bose QuietComfort 45 Headphones | Triple Black",
    "Canon EOS R6 Camera | 20.1 MP Mirrorless",
    "Microsoft Surface Pro 9 | Intel i5, 8GB RAM, 256GB SSD",
    "Adidas Ultraboost 21 Running Shoes | Core Black",
    "Amazon Kindle Paperwhite | 32GB, Waterproof",
    "LG OLED65C1PUB 65\" 4K Smart TV",
    "Garmin Forerunner 955 Smartwatch | Slate Grey",
    "Google Nest Thermostat | Charcoal",
    "KitchenAid Stand Mixer | 5-Quart, Empire Red",
    "Dyson V11 Torque Drive Cordless Vacuum",
    "JBL Charge 5 Portable Bluetooth Speaker | Squad",
    "Panasonic Lumix GH5 Camera | 20.3 MP, 4K Video",
    "Apple MacBook Pro 14\" | M1 Pro, 16GB RAM, 1TB SSD",
    "Under Armour HeatGear Compression Shirt | Black/Red"
]

Next, load our fine-tuned SBERT model and convert product names into vector embeddings:

# Load fine-tuned model
model = SentenceTransformer('./sbert-model')

To convert product names into embeddings, we’ll utilize the GPU and convert them into tensors. You can do so using the following code:

product_data = model.encode(products, convert_to_tensor=True).to("cuda")

By converting embeddings to CUDA, we leverage GPU computational support (dtype=torch.float32); otherwise, if CPU is selected, it defaults to (dtype=float32).

This product_data serves as our vector database, now stored in memory. Alternatively, you can utilize vector databases like Qdrant, Pinecone, Chroma, etc.

Lastly, create a function that accepts a user query from the terminal or as user input and returns the top products along with their Cosine-Similarity scores.

def search():
    query = input("Enter Query:\n")
    query_embeddings = model.encode([query], convert_to_tensor=True).to("cuda")
    hits = util.semantic_search(query_embeddings, product_data,
                                score_function=util.cos_sim)
   
    for i in range(5):
        best_search_term_id, best_search_term_core = hits[0][i]['corpus_id'], hits[0][i]['score']
        print("\nTop result: ", products[best_search_term_id])
        print("Score: ", best_search_term_core)

Test run:

You can observe that our model is performing exceptionally well, with satisfactory scores. To further enhance result relevance, consider adding a threshold ratio of 0.5.

Conclusion

Using SentenceTransformer 3.0.0 makes training or fine-tuning embedding models a breeze. The new version boasts support for multi-GPU utilization via the DDP method and introduces logging and experimentation features through Weights & Biases. By encapsulating our code within a single main function and executing it with a single command, developers can streamline their workflow significantly.

The Evaluator functionality aids in evaluating models during the training phase, catering to defined tasks like Embedding Similarity Search in our scenario. Upon loading the model for inference, it delivers as anticipated, yielding a satisfactory similarity score.

This process harnesses the potential of vector embeddings to enhance search results, leveraging user queries and database interactions effectively.

Resources

Training and Finetuning Embedding Models with Sentence Transformers v3 (huggingface.co)

Training Overview — Sentence Transformers documentation (sbert.net)

Thank you for reading the article. If you found it helpful, please consider giving it an upvote and sharing it. Connect me on LinkedIn and X.

Until next time, Signing off!

Fine-Tuning Sentence Transformers for Embedding Search was originally published in Level Up Coding on Medium, where people are continuing the conversation by highlighting and responding to this story.

Introduction

Semantic caching is a method of retrieval optimization where frequent or similar queries are instantly fetched from the knowledge base by storing them in a cache. This concept is similar to caching in operating systems, where frequently accessed data is stored in a high-speed storage layer for quick retrieval.

In a Retrieval-Augmented Generation (RAG) system, users ask questions and receive results from a vector database. In a large-scale application with multiple users asking frequent questions, optimizing retrieval tasks by caching frequently accessed contexts can significantly improve the application’s performance.

In a simple RAG setup, the user’s query searches for related contexts in the database and returns them to the augmented prompt and Language Model (LLM). By introducing semantic caching, the system can quickly serve frequent queries from the cache, enhancing overall efficiency.

How Does Semantic Cache Work?

The simple retrieval process can be time-consuming because each search needs to find the context and retrieve the data, even when most queries have different semantics but the same context. This is where caching comes into play.

By storing frequently accessed contexts in cache memory, the need for repeated read-write operations is reduced, significantly improving efficiency. Caching ensures that the context is readily available for commonly asked questions, speeding up the retrieval process and enhancing overall performance.

See the diagram:

Scenarios to use a semantic cache layer in the above diagram:

In the diagram, the semantic cache is placed between the vector database and the user query. This is the appropriate placement. When a user’s query is embedded and similar contexts are found from the database using distance metrics such as Cosine, Euclidean, etc., it fetches the data and stores it in another collection for the semantic cache. The results are then passed to the augmented prompt and the Language Model (LLM) to generate an answer for the user.

Another approach is to place the semantic cache after the text is generated by the augmented prompt and LLM. This will store answers from the LLM in the semantic cache, so users will receive answers in the generated text format. However, there is a caveat: if the user includes specific instructions in their query, this scenario may not be effective. For example, the queries “What is the Indus Valley?” and “I want to know about the Indus Valley civilization in 1000 words” have the same semantic context but different requirements. Since we are saving LLM answers in the cache, this approach would not fulfill the second query accurately.

The above scenarios are based on your requirements. So, choose wisely and implement by considering the pros and cons.

Guide to Using Semantic Cache Using Qdrant, LangChain and Groq + Llama3

Groq has released an LPU that can generate output with lightning speed, offering 10x performance at 1/10th the latency. Currently, Groq supports the Llama3 and Mixtral models as LLMs. It demonstrates impressive speed when it comes to the inference of LLM output tokens.

We have already seen how a semantic cache improves RAG performance in retrieval speed. Groq will be the cherry on top, accelerating application performance even further.

Let’s see an example:

First, install qdrant-client, langchain, langchain-groq, fastembed, and datasets using pip command.

pip install qdrant-client langchain langchain-groq fastembed datasets

Then create a Qdrant Cluster on cloud and collect the URL and API key. Visit GroqCloud and create the API key to use Groq in LangChain.

Import dependencies and paste your credentials:

import numpy as np
import pandas as pd
from datasets import load_dataset
from qdrant_client import QdrantClient
from qdrant_client.http import models


import uuid
import time
from typing import List
from fastembed import TextEmbedding
from qdrant_client.http.models import PointStruct, SearchParams

from langchain_core.prompts import ChatPromptTemplate, PromptTemplate
from langchain_core.output_parsers import StrOutputParser
from langchain_core.runnables.passthrough import RunnablePassthrough
from langchain_groq import ChatGroq


qdrant_uri = "Qdrant-URI"
qdrant_api = "Qdrant-API-Key"
groq_key = "Groq-API-key"

Load the LLM model. We are using the Llama3–8B model; you can use the 70B parameter model also.

llm = ChatGroq(groq_api_key=groq_key,
               model='llama3-8b-8192',
               temperature=0.2) # set temperature by your own

Now, load the dataset.

data = load_dataset("llamafactory/PubMedQA", split="train")
data = data.to_pandas()
data.head()

Get the subset of the data to ingest in the database

MAX_ROWS = 1000
OUTPUT = "output"
subset_data = data.head(MAX_ROWS)

Create Qdrant DB Client collection and insert the document chunks into the database.

client = QdrantClient(
    qdrant_uri,
    api_key=qdrant_api
)


chunks = subset_data[OUTPUT].to_list()

# Add data to defined collection
client.add(
   collection_name='PubMedQA',
   documents=chunks
)

The above code block will add data in the collection name PudMedQA.

Now, we will see how retrieval works with and without semantic cache

1. Without Semantic Cache

class QdrantVectorStore():
    def __init__(self):
        self.encoder = TextEmbedding(model_name="BAAI/bge-small-en-v1.5")
        self.client = QdrantClient()
        self.db_collection_name= "PubMedQA"
       
        self.db_client = QdrantClient(
            qdrant_uri,
            api_key=qdrant_api
        )
       
        self.euclidean_threshold = threshold
   
    def get_embedding(self, question):
        embedding = list(self.encoder.embed(question))[0]
        return embedding
   
    def query_database(self,query_text):
        result = self.db_client.query(
            query_text=query_text,
            limit=3,
            collection_name=self.db_collection_name
        )
        return result
   
    def query(self, question):
        start_time = time.time()
       
        db_results = self.query_database(question)
       
        if db_results:
            response_text = db_results[0].document
            print('Retrieval without cache.')
            elapsed_time = time.time() - start_time
            print(f"Time taken: {elapsed_time:.3f} seconds")
            return response_text
       
        print("No answer found in Database")
        elapsed_time = time.time() - start_time
        print(f"Time taken: {elapsed_time:.3f} seconds")
        return "No answer available"

Create an object from the class.

vector_db = QdrantVectorStore()

Define augmented prompt and LLama3 model in LangChain Language Expression.

def chat_query(question):
    context = vector_db.query(question)
   
    prompt_template = """Answer the question with given context as per requirement.
                {context}
           
                Question: {question}"""
    prompt = PromptTemplate.from_template(prompt_template)


    chain = prompt | llm | StrOutputParser()
   
    result = chain.invoke({"question":question, "context":context})
    print("\n\n")
    print(result)

Ok, time to find out the performance. Here are questions I am using to compare performance with and without semantic cache.

question_1 = "Does bacterial gastroenteritis predispose people to functional gastrointestinal disorders?"
question_2 = "Can bacterial gastroenteritis lead to functional gastrointestinal disorders?"

chat_query(question_1)

Retrieval without cache.
Time taken: 1.187 seconds

 
Yes, bacterial gastroenteritis predisposes people to functional gastrointestinal disorders, specifically Irritable Bowel Syndrome (IBS) and functional diarrhea.

Let’s see similar question_2

chat_query(question_2)

Retrieval without cache.
Time taken: 0.392 seconds


Yes, according to the given context, bacterial gastroenteritis can lead to functional gastrointestinal disorders, specifically Irritable Bowel Syndrome (IBS) and functional diarrhea.

Here, we observe that even without a semantic cache, queries can perform quickly. However, let’s see what happens with a semantic cache.

2. With Semantic Cache

class SemanticCache:
    def __init__(self, threshold=0.35):
        self.encoder = TextEmbedding(model_name="BAAI/bge-small-en-v1.5")
        self.cache_client = QdrantClient(":memory:")
        self.cache_collection_name = "PubMedQA-cache"


        self.cache_client.create_collection(
            collection_name=self.cache_collection_name,
            vectors_config=models.VectorParams(
                size=384,
                distance='Euclid'
            )
        )


        # Initialize Qdrant Client for external database
        self.db_client = QdrantClient(
            qdrant_uri,
            api_key=qdrant_api
        )
       
        self.db_collection_name = "PubMedQA"


        self.euclidean_threshold = threshold


    def get_embedding(self, question):
        embedding = list(self.encoder.embed(question))[0]
        return embedding


    def search_cache(self, embedding):
        search_result = self.cache_client.search(
            collection_name=self.cache_collection_name,
            query_vector=embedding,
            limit=1
        )
        return search_result


    def add_to_cache(self, question, response_text):
        # Create a unique ID for the new point
        point_id = str(uuid.uuid4())
        vector = self.get_embedding(question)
        # Create the point with payload
        point = PointStruct(id=point_id, vector=vector, payload={"response_text": response_text})
        # Upload the point to the cache
        self.cache_client.upload_points(
            collection_name=self.cache_collection_name,
            points=[point]
        )


    def query_database(self, query_text):
        results = self.db_client.query(
            query_text=query_text,
            limit=3,
            collection_name=self.db_collection_name
        )
        return results


    def ask(self, question):
        start_time = time.time()
        vector = self.get_embedding(question)
        search_result = self.search_cache(vector)
        print(search_result)
        if search_result:
            for s in search_result:
                if s.score <= self.euclidean_threshold:
                    print('Answer recovered from Cache.')
                    print(f'Found cache with score {s.score:.3f}')
                    elapsed_time = time.time() - start_time
                    print(f"Time taken: {elapsed_time:.3f} seconds")
                    return s.payload['response_text']


        db_results = self.query_database(question)
        if db_results:
            response_text = db_results[0].document
            self.add_to_cache(question, response_text)
            print('Answer added to Cache.')
            elapsed_time = time.time() - start_time
            print(f"Time taken: {elapsed_time:.3f} seconds")
            return response_text


        # Fallback if no response is found
        print('No answer found in Cache or Database.')
        elapsed_time = time.time() - start_time
        print(f"Time taken: {elapsed_time:.3f} seconds")
        return "No answer available."

In the above code, there is another Qdrant DB collection that holds the cached data for all unique contexts based on the queries.

cache = SemanticCache()


def chat_cache(question):
    context = cache.ask(question)
   
    prompt_template = """Answer the question with given context as per requirement.
                {context}
           
                Question: {question}"""
    prompt = PromptTemplate.from_template(prompt_template)


    chain = prompt | llm | StrOutputParser()
   
    result = chain.invoke({"question":question, "context":context})
    print("\n\n")

chat_cache(question_1)

Answer added to Cache.
Time taken: 1.412 seconds

 
Yes, bacterial gastroenteritis predisposes people to functional gastrointestinal disorders, specifically Irritable Bowel Syndrome (IBS) and functional diarrhea.

The first time a query is made, or for queries where the context is not present in the cache, simple retrieval is performed. However, the retrieved context is then stored in the cache.

Let’s ask question_2

chat_cache(question_2)

Answer recovered from Cache.
Found cache with score 0.329
Time taken: 0.053 seconds

Yes, according to the given context, bacterial gastroenteritis can lead to functional gastrointestinal disorders, specifically Irritable Bowel Syndrome (IBS) and functional diarrhea. The study found that symptoms consistent with IBS and functional diarrhea occur more frequently in people after bacterial gastroenteritis compared to controls, even after excluding individuals with pre-existing functional gastrointestinal disorders.

We successfully retrieved answers from the cache using a threshold score. For the threshold, we used a Euclidean distance-based score between the user query and the provided context.

Benchmarks with and without Semantic Cache

(All time stamps in this article are based on retrieval performance, not on Groq performance.)

We have seen that on question_1 and question_2 the two approaches have different results.

Let’s compare on another set of questions:

Query 1:

'Can we predict the duration of chemotherapy-induced neutropenia in febrile neutropenic patients, focusing on regimen-specific risk factors?'

Query 2:

'Can we predict the duration of chemotherapy-induced neutropenia in febrile neutropenic patients?'

Here is the result:

Again, in both the examples, semantic cache performs way better when the same semantic query is asked a second time.

Conclusion

Semantic caching is an important technique to implement in any RAG-based system. It significantly reduces the retrieval time for frequently asked questions from the database. By placing the cache before the augmented prompt and LLM, answers can be tailored based on the given instructions in the query.

However, there are some drawbacks to consider. Semantic cache collections need to be managed differently. Additionally, as data and queries change over time, it’s important to periodically clear the semantic cache to keep it updated with only the most relevant data. Remember, the semantic cache should hold a limited amount of data; otherwise, it will take time to find answers from a large volume of data.

Finally, by implementing Groq and semantic caching, you will experience a significant increase in the speed of generating answers to user queries.

If you find this article useful, please upvote. Follow me on Medium and LinkedIn.

References

Implementing semantic cache to improve a RAG system with FAISS. — Hugging Face Open-Source AI Cookbook

Semantic Cache: Accelerating AI with Lightning-Fast Data Retrieval — Qdrant

A Guide to Using Semantic Cache to Speed Up LLM Queries with Qdrant and Groq. was originally published in Stackademic on Medium, where people are continuing the conversation by highlighting and responding to this story.

Large language models (LLM) have significantly altered the landscape of technology. They’ve revolutionized natural language processing (NLP) capabilities, which has enabled advancements in machine translation, sentiment analysis, chatbots, and more. Most LLMs are trained on huge datasets from all over the internet. They are mostly known for their capabilities in the English language. Models like ChatGPT, Llama2, and Mistral are heavily trained on a large corpus of the English language. ChatGPT understands more than 50 languages, but when we talk about an open-source model, we are looking for a specialized model that focuses on one domain or language.

How can we build our LLM application specifically for the Portuguese language? Is there any model for that? Yes, the answer is “Sabia-7B”. This LLM is specially trained for the Portuguese language. In this article, we will see how to optimize the power of Sabia-7B to create a RAG application on our data.

Sabia-7B:

Sabia: Portuguese Large Language Models is an official research paper published by Maritaca AI. Sabia is a pre-trained model on GPT-J and LLaMA models on Portuguese texts. It is built upon Poeta, a suite of 14 Portuguese datasets. The researchers stated that their model outperforms English-centric and multilingual counterparts by a significant margin; Sabia-65B performs on par with GPT-3.5-turbo.

The model evaluated the dataset comprising texts created by native Brazilian Portuguese speakers, as well as the dataset translated into Portuguese. According to the Hugging Face model card, Sabia-7B is an auto-regressive model that uses the same architecture as LLaMA-1–7B and uses the same tokenizer as LLaMA-1–7B, with a maximum sequence length of 2048 tokens.

Maritaca AI is the organization behind building LLM for the Portuguese language. You can access the official API from their website, which is paid. In this article, we will explore how to use the Hugging Face model of Sabia-7B to build a RAG application.

What is RAG:

Retrieval Augmentation Generation (RAG) is a powerful technique for enhancing the capabilities of LLM. There are two main components:

1. Retriever: This model finds relevant information from a large corpus documents based on your query. Think of it like a super librarian who scours all the books in the library to find the ones most likely to have your answer.

2. Generator: Once the retriever provides relevant context, the generator uses its language understanding and generation skills to produce a response tailored to your specific question.

Stages of RAG pipeline:

1. Document Ingestion: The relevant documents are processed and converted into a format suitable for efficient retrieval. This often involves extracting key information and creating vector representations of the documents.

2. Indexing: These vector representations are stored in a specialized database called a “vector database” for faster retrieval during the online stage.

3. Query Processing: Your query is received and encoded into a similar vector representation.

4. Retrieval: The query vector is compared to the document vectors in the database. The most relevant documents are retrieved based on their similarity scores.

5. Generation: The retrieved documents are prepended to your query as additional context. Then, the generator uses this contextualized query to generate a response that addresses your specific needs.

Now let’s build a simple chatbot application using the HuggingFace model: Sabia-7B.

Steps to Build the Chatbot Application Using Sabia-7B

Collect the Data:

Custom data is the base of a RAG application. So in this article, I have extracted the data from Wikipedia which is in Portuguese. There are two PDF files containing information about the famous Brazilian footballer Neymar Jr. and the Brazilian festival Carnival. While collecting this data, I changed Wikipedia language to Portuguese, copied the paragraphs, and pasted them into individual PDF files.

You can take any data you want, but it should be in Portuguese.

Initial Setup:

Before starting the coding part let’s install the necessary dependencies:

!pip install transformers datasets peft bitsandbytes trl langchain accelerate huggingface_hub qdrant_client pypdf sentence-transformers

Note: I am using Google Colab’s free version for this explanation. Connect T4 GPU simply paste the above command in the first cell.

Configure Vector Database:

A vector database is a type of database that specializes in storing and querying high-dimensional vector data efficiently. In the vector database, we are storing your text data in a vector form using embeddings. If you don’t know what word embeddings are, or how they work, just read this article for a better understanding: What are Vector Embeddings? Applications, Use Cases & More (datastax.com)

We are using Qdrant DB as our vector database for this article.

QdrantDB:

To begin with this stage, first set the Qdrant database for your application.

1. Login to the Qdrant.

2. Get your API key (copy it and paste it on your side first, you can’t see the API key again after copying).

3. Copy the database URL with the key and save it locally first.

Store your credentials along with dependencies. Like below:

import os
import qdrant_client
from langchain_community.vectorstores import Qdrant
from langchain.embeddings import HuggingFaceEmbeddings
from langchain_community.document_loaders import PyPDFLoader, DirectoryLoader
from langchain.text_splitter import CharacterTextSplitter
import json
import re
from pprint import pprint
import pandas as pd
import torch
from qdrant_client import QdrantClient


QDRANT_URI = "YOUR-URI"
QDRANT_API_KEY = "YOUR-API-KEY"
HF_AUTH_KEY="HUGGINGFACE-TOKEN"


device = "cuda" if torch.cuda.is_available() else "cpu"

Also, to get your Hugging Face API key: Go to your Hugging Face profile >> Edit Profile >> Access Token >> New Token >> Give a name to the token, copy your key, and simply paste it into a variable.

Select the Sabia-7B Model:

embedding_model = "sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2"
model_name = "maritaca-ai/sabia-7b"


print(f"Device: {device}")
print(f"Embedding Model Name: {embedding_model}")
print(f"LLM Name: {model_name}")

I have selected two models from HuggingFace. The first one is the embedding model which will be used to embed our textual data into a numerical representation. The second one is the large language model for our Portuguese language use case: Sabia-7B.

“sentence-transformers” provide embedding models for the English language and also multilingual languages. You can try with multiple versions of sentence-transformers.

Chunk, Embed, and Store:

We will start by chunking and breaking our text into pieces. Before that, let’s see how we can load our data from a directory and clean it using Langchain and Python Regex.

We will clean the text of PDFs, this is the simplest way you can do this:

# Function to clean the text
def clean_text(text):
  cleaned_text = text.strip()  # Remove leading/trailing whitespace
  cleaned_text = re.sub(r"\s+", " ", cleaned_text)  # Replace extra spaces with a single space
  cleaned_text = re.sub(r"[^\w\s]", "", cleaned_text)  # Remove non-alphanumeric characters
  return cleaned_text

Langchain is a go-to library to deal with LLM projects using Simple Python language.

# Load document from data directory in .pdf format
def load_documents():
    loader = DirectoryLoader('data/', glob="*.pdf", loader_cls=PyPDFLoader)
    documents = loader.load()
    for doc in documents:
      cleaned_text = clean_text(doc.page_content)
      doc.page_content = cleaned_text
    return documents

After this stage, the most critical step has to be performed which is converting numerical representations into embeddings:

Define the Qdrant Client

# Create a Qdrant Client
client = qdrant_client.QdrantClient(
    QDRANT_URI,
    api_key=QDRANT_API_KEY
)


# Create a collection
vectors_config = qdrant_client.http.models.VectorParams(
    size=384,
    distance=qdrant_client.http.models.Distance.COSINE
)


client.recreate_collection(
    collection_name="Brazilian-Portugeese",
    vectors_config=vectors_config
)

There are three variables: client variables define the QdrantClient API so we can deal with Python. We can also define the configuration of our embeddings such as size and distance metrics, vectors_config does this work for us. Using client.recreate_collection() we can define a collection and store our data.

Let’s use our embedding algorithm from Hugging Face and perform our vector database ingestion task:

# Define Embeddings using HF
embeddings = HuggingFaceEmbeddings(
    model_name=embedding_model)
# Split texts
def get_chunks(text):
    text_splitter = CharacterTextSplitter(
        separator="\n",
        chunk_size=300,
        chunk_overlap=50,
        length_function=len
    )
    chunks = text_splitter.split_documents(text)
    return chunks


cleaned_documents = load_documents()
text_chunks = get_chunks(cleaned_documents)

Simply save text chunks by performing the above function.

In the last step push your data into your QdrantDB collection

qdrant = Qdrant.from_documents(
    text_chunks,
    embeddings,
    url=QDRANT_URI,
    api_key=QDRANT_API_KEY,
    prefer_grpc=True,
    collection_name='Brazilian-Portugeese',
)

Build HF Model:

Using open weights from Hugging Face transformers we developers can easily utilize them for our specific task. In this article, we will see a simple RAG application so we will start buildingour model with quantization, as the model is complex and large. There is another approach which is Fine-Tuning that we will see in another blog. For this application, let’s see how we can do this magic in simple lines of code.

from torch import cuda, bfloat16
import transformers


bnb_config = transformers.BitsAndBytesConfig(
    load_in_4bit = True,
    bnb_4bit_quant_type='nf4',
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=bfloat16
)


model_config = transformers.AutoConfig.from_pretrained(
    model_name,
    token=HF_AUTH_KEY
)

This code snippet initializes a configuration object for running a Hugging Face Transformer model on a GPU with 4-bit weight quantization using the Bits And Bytes (BnB) technique. It imports necessary modules from PyTorch and the Hugging Face

Transformers library then configures the BnB parameters for low-memory computation while setting the compute data type to bfloat16. At last, it creates a configuration object specific to the chosen model architecture. Using this setup users are allowed to train or perform inferences with their desired model efficiently while reducing memory footprint through 4-bit quantization.

Now define the Causal Language Model to perform text generation operation:

model = transformers.AutoModelForCausalLM.from_pretrained(
    model_name,
    trust_remote_code=True,
    config=model_config,
    quantization_config=bnb_config,
    device_map='auto',
    use_auth_token=HF_AUTH_KEY
)

model.eval()

Using transformer architecture build your model using open weights based on the configuration provided.

Tokenizer is the essential part for LLMs to understand and process human-readable text. We will use a tokenizer of the same model as below:

tokenizer = transformers.AutoTokenizer.from_pretrained(
    model_name,
    use_auth_token=HF_AUTH_KEY
)

Create a transformer pipeline that generates text using a model and tokenizer combination.

generate_text = transformers.pipeline(
    model=model, tokenizer=tokenizer,
    return_full_text=True,  # langchain expects the full text
    task='text-generation',
    # we pass model parameters here too
    temperature=0.2,  # 'randomness' of outputs, 0.0 is the min and 1.0 the max
    max_new_tokens=512,  # mex number of tokens to generate in the output
    repetition_penalty=1.1  # without this output begins repeating
)

Creating a Transformer pipeline involves setting up a process for inputting text and generating output text, specifically for the purpose of text generation.

Using LangChain let’s convert the transformer pipeline into LLM using HuggingFacePipeline.

from langchain.llms import HuggingFacePipeline
llm = HuggingFacePipeline(pipeline=generate_text)

Use a vector store for retrieval generation. Let’s create a function for that:

def get_vectore_store():

    client = qdrant_client.QdrantClient(
        QDRANT_URI,
        api_key=QDRANT_API_KEY
    )

    embeddings = HuggingFaceEmbeddings(
        model_name="sentence-transformers/all-MiniLM-L6-v2"
    )

    vectore_store = Qdrant(
        client=client,
        collection_name="Brazilian-Portugeese",
        embeddings=embeddings
    )

    return vectore_store

Create RetrievalQA pipeline for RAG pipeline:

from langchain.chains import RetrievalQA

vectore_store = get_vectore_store()
rag_pipeline = RetrievalQA.from_chain_type(
    llm=llm, chain_type='stuff',
    retriever=vectore_store.as_retriever()
)

We will create rag_pipeline to generate text by passing text.

Let’s see examples in Portuguese language:

 rag_pipeline(“Carreira futebolistica de Neymar Jr”)

{'query': 'Carreira futebolística de Neymar Jr',
 'result': '\n\nNeymar da Silva Santos Júnior Mogi das Cruzes 5 de fevereiro de 1992 é um futebolista brasileiro que atua como atacante Atualmente joga pelo AlHilal e pela Seleção Brasileira É considerado o principal futebolista brasileiro da atualidade e um dos melhores do mundo11121314 sendo o maior artilheiro da Seleção Brasileira15 Revelado pelo Santos em 2009 Neymar ganhou diversos títulos com o clube incluindo dois Campeonatos Paulistas seguidos uma Copa do Brasil e uma Libertadores com esse último sendo o primeiro do Santos desde 1963 Ganhou por duas vezes seguidas o prêmio de melhor jogador sulamericano do ano em 2011 e 2012 e um prêmio Puskás de gol mais bonito do ano em 201116 Em 2013 foi vendido ao Barcelona após ser protagonista da conquista da Copa das Confederações de 2013 pela Seleção Brasileira Considerada a venda mais cara da história do futebol brasileiro Neymar na sua segunda temporada pelo clube espanhol estrelando um trio de ataque ao lado de Lionel Messi e Luis Suárez conquistou a tríplice coroa de La Liga Copa del Rey e Liga dos Campeões da UEFA terminando como um dos artilheiros da última e se consagrando como um dos melhores futebolistas do mundo sendo finalista da Bola de Ouro da FIFA por suas atuações no mesmo ano Question: Carreira futebolística de Neymar Jr\nHelpful Answer:\n\nNeymar da Silva Santos Júnior Mogi das Cruzes 5 de fevereiro de 1992 é um futebolista brasileiro que atua como atacante Atualmente joga pelo AlHilal e pela Seleção Brasileira É considerado o principal futebolista brasileiro da atualidade e um dos melhores do mundo11121314 sendo o maior artilheiro da Seleção Brasileira15 Revelado pelo Santos em 2009 Neymar ganhou diversos títulos com o clube incluindo dois Campeon'}

We will create a simple chatbot that takes input queries and generates answers.

def rag_chatbot():
  input_question = input("Pergunte qualquer coisa: ")
  print("\n")
  print("Resposta:\n")
  response = rag_pipeline(input_question)
  return response['result']

rag_chatbot()

Pergunte qualquer coisa: O que é Festival de Carnaval no Brasil?


Resposta:


'\nO Carnaval brasileiro é uma das festas mais famosas e vibrantes do mundo celebrada em todo o país com uma mistura de música dança desfiles e tradições culturais Originado de tradições europeias africanas e indígenas o Carnaval no Brasil tornouse uma manifestação única e cheia de energia que reflete a diversidade e a criatividade do povo brasileiro A história do Carnaval no Brasil remonta aos tempos coloniais quando os colonizadores portugueses trouxeram suas festas e tradições religiosas como o Entrudo que envolvia jogos de água e tinta Com o tempo essas celebrações foram influenciadas pelas tradições africanas trazidas pelos escravos como o batuque e as danças rituais No século 19 surgiram os primeiros blocos carnavalescos e cordões que reuniam pessoas para desfilarem pelas ruas ao som de músicas e instrumentos improvisados Com o passar dos anos o Carnaval foi se tornando cada vez mais organizado com a criação das escolas de samba e dos desfiles competitivos Hoje o Carnaval brasileiro é conhecido por seus desfiles extravagantes especialmente no Rio de Janeiro e em São Paulo onde as escolas de samba competem em uma verdadeira batalha pelo título de campeã do Carnaval Os desfiles apresentam carros alegóricos elaborados fantasias deslumbrantes e performances emocionantes tudo ao som de samba enredo Além dos desfiles o Carnaval também é celebrado com festas de rua bailes e blocos de Carnaval onde as pessoas se reúnem para dançar cantar e se divertir As festividades começam oficialmente na sextafeira antes da Quartafeira de Cinzas e continuam até a terçafeira seguinte conhecida como o dia do Carnaval Os desfiles são realizados em vários estados brasileiros como Rio de Janeiro São Paulo Minas Gerais e Bahia Onde o Carnaval é mais famoso e popularizado O Carnaval brasileiro é uma mistura de cultura que reflete.’

Conclusion:

This is the simplest way we can build a Portuguese language-based chatbot using an open-source LLM. You can use the Streamlit app for chatbot UI for interactive ways of chatting. Also, you can use more GPUs to train your model for higher accuracy. You can deploy your model to the endpoint on AWS so you will get faster and better inference time.

That’s it for today. Feel free to ask any questions in the comments section. If you found this blog valuable, do give an upvote and follow me on LinkedIn.

Introduction

In this article, we are going to explore how to match the voice of a speaker with an existing set of voices. You can think of it like a biometric system but using the human voice, unlike physical senses such as the thumb and the eye. To achieve this, we will use the magic of vector embeddings and open-source technologies.

This type of technology is used in Google Assistant or Siri. When you buy a new device, like an Android phone, while setting up Google on your system, it asks for your voice to capture its pattern, vocals, and so on, for security reasons. This is so that only you can access Google Assistant by saying “Ok, google”.

Before we get into the details, let’s understand first what vector embedding is and how it has been used for audio.

Vector Embeddings for Audio

Vector embeddings are a way to represent objects, such as words, sentences or, in our case, audio data, as vectors in a mathematical space. Audio data can be represented as vectors, where different aspects of the audio (features like frequency, amplitude, etc.) are mapped to specific positions in the vector.

In the context of audio data, machine learning models can be trained to learn these embeddings. The model analyzes the patterns and characteristics of the audio data to generate meaningful vector representations. Once the model is trained, it can encode audio data by transforming it into a vector representation. This vector now captures important information about the audio’s content and characteristics.

Audio content will have vectors that are close together in the embedding space. This allows for tasks like audio similarity comparison, where you can quickly identify how similar two audio clips are by measuring the distance between their respective embeddings. To generate vector embeddings, we will use an open-source tool called Resemblyzer and store those vectors in Qdrant DB.

We have a set of audio clips of some famous personalities: Cristiano Ronaldo, Donald Trump, and Homer Simpson (yes, he is famous).

Resemblyzer: An Overview

Resemblyzer allows us to derive high-level representation of voice through a deep learning model. It simplifies the life of developers by enabling them to convert audio clips into vectors with just a few lines of code, eliminating the need for neural networks. See official github repository.

Install Resemblyzer for Python (3.5+)

pip install resemblyzer

I’m using Google Colab with a free T4 GPU for this task. You can also use CPU, but it may take a long time. Click here to get audio data.

Let’s begin with the coding part:

# import necessary libraries
from resemblyzer import preprocess_wav, VoiceEncoder
from  pathlib import Path 
from tqdm import tqdm 
import numpy as np 
from Ipython,display import Audio 
from itertools import groupby
import heapq

# run sample audio
audio_sample = Audio(‘path-to-audio-folder/train/Trump.mp3’, autoplay=True)
display(audio_sample)

PLAY: Trump.mp3

Let’s fetch audio clips from the train folder.

# Get all the audio clips data file which is in ".mp3" format.
wav_fpaths = list(Path("path-to-audio-folder/train").rglob("*.mp3"))
speakers =list(map(lambda wav_fpath: wav_fpath.stem, wav_fpaths))
print(speakers)

['Ronaldo', 'Ronaldo2', 'Homer', 'Homer2', 'Trump']

Now, the important part, we will unleash the power of Resemblyzer and convert audio clips into the vector embeddings with just a few lines of code. Preprocess the waves first for all audio clips.

wavs = np.array(list(map(preprocess_wav, tqdm(wav_fpaths, “Preprocessing wavs”. len(wav_fpaths)))), dtype=object)
speaker_wavs = {speaker: wavs[list(indiices] for speaker, indices in groupby(range(len(wavs)), lambda i: speakers[i])}
print(speaker_wavs)

{'Ronaldo': array([array([ 0.00045622, -0.00088888,  0.00016845, ..., -0.00079568,
               -0.00718354, -0.01011641], dtype=float32)               ],
       dtype=object),
 'Ronaldo2': array([array([ 0.0025312 ,  0.00321749,  0.00460094, ..., -0.01093079,
               -0.01293177, -0.01618683], dtype=float32)               ],
       dtype=object),
 'Homer': array([array([0., 0., 0., ..., 0., 0., 0.], dtype=float32)], dtype=object),
 'Homer2': array([array([ 1.33051715e-14,  3.98843861e-14, -3.70518893e-15, ...,
                5.39025990e-04, -5.10490616e-04, -5.79551968e-04], dtype=float32)],
       dtype=object),
 'Trump': array([array([-0.0165875 ,  0.03297266, -0.01565401, ..., -0.03698713,
               -0.03372933, -0.02938525], dtype=float32)               ],
       dtype=object)}

In the above code, we converted these sound waves into numerical representations with a few lines of code and without using any neural network.

Now, convert these numerical representations into embeddings.

# compute the embeddings
encoder = VoiceEncoder(“cuda”)
utterance_embeds = np.array(list(map(encoder.embed_utterance, 
print(utterance_embeds)

[[0.         0.         0.0173962  ... 0.         0.04333723 0.00142971]
 [0.         0.00967959 0.00503905 ... 0.04058945 0.09630667 0.0495304 ]
 [0.15830468 0.         0.01373593 ... 0.         0.         0.        ]
 [0.18647183 0.         0.11558624 ... 0.         0.         0.        ]
 [0.         0.11265804 0.         ... 0.         0.         0.14819394]]

This vector contains a float object where each row represents each audio clip.

For this task we are using Qdrant DB as our primary vector database. So, for that, we need to convert this representation into a suitable format. Basically, we need a list of dictionaries where each dictionary contains key as id and vector as keys. Id will be an incremental numeral value.

To get similar vectors, a unique id must be assigned to the vectors.

# Create an empty list to hold the embeddings in the desired format
embeddings = []

# Iterate through each embedding in the array
for i, embedding in enumerate(utterance_embeds):
   # Create a dictionary with “id” and “vector” keys
   embedding_dict  = {“id”:i+1, “vector”:embedding.tolist()} # Start IDs from 1
   # Append the dictionary to the embeddings list
   embeddings.append(embedding_dict)

QdrantDB: An Overview

Qdrant DB is one of the most popular vector databases out there. Using Qdrant DB, web developers can store embeddings and retrieve them seamlessly. Here is the official documentation.

To start with Qdrant DB, Sign up on their Cloud Service to start with their free tier which has limits of up to 1GB per cluster. Get your API key (copy it locally and safely — you can’t see the API key again after copying).

For Python Qdrant DB has its own API qdrant_client which is very easy to use with fewer lines of code. Let’s set-up Qdrant DB.

Install qdrant_client via pip:

pip install qdrant_client

import qdrant_client
qdrant_uri = 'paste-your-db-uri' # Paste your URI
qdrant_api_key = 'paste-your-api-key' # Paste your API KEY

Let’s create a collection in the database; here the meaning of collection is the same as MongoDB collection.

# Create a collection
vectors_config = qdrant_client.http.models.VectorParams(
  size=256, # requires for embeddings from resemblyzer
  distance=qdrant_client.http.models.Distance.COSINE
)

Now, after initializing QdrantDB, we will upsert (or add) embeddings from Resemblyzer.

# Upsert embeddings
client.upsert('my-collection', embeddings)

Till here we stored our audio samples in an encoded version in Qdrant DB. Now we will test this using a new voice, which has a record in the database.

Speaker Recognition:

To recognize the user with a new voice is all about finding a similarity between the new voice and the set of voices already stored. For example, take the new voice of Cristiano Ronaldo and check if it’s recognized or not. We already have Ronaldo’s voice in the database.

I’m taking the iconic short speech by Ronaldo, which he gave after winning the UCL:

“Muchas gracias afición esto para vosotros. Siuuuuuuuuu!”

PLAY: Siuu.mp3

Convert the new voice into embeddings

test_wav = preprocess_wav(“/content/drive/MyDrive/audio_data_colab/Siuu.mp3”)
# Create a voice encoder object
test_embeddings = encoder.embed_utterance(test_wav)

# Search related embeddings
results = client.search(“my-collection”, test_embeddings)
print(results)

[ScoredPoint(id=2, version=0, score=0.6956655, payload={}, vector=None, shard_key=None),
 ScoredPoint(id=1, version=0, score=0.6705738, payload={}, vector=None, shard_key=None),
 ScoredPoint(id=5, version=0, score=0.56731033, payload={}, vector=None, shard_key=None),
 ScoredPoint(id=3, version=0, score=0.535391, payload={}, vector=None, shard_key=None),
 ScoredPoint(id=4, version=0, score=0.42906034, payload={}, vector=None, shard_key=None)]

By the above result you can see that the ids 1 and 2 are associated with the Ronaldo clip (we did this in the embedding code). The highest score is about 70%, which is fine because we have a very small amount of data. Also the length of the clips are 3–4 seconds on an average. You can add more data and try this out.

To get top 2 similar results, just run the following code (you can also do it for top 1 or top 3 and then decide based on mode in top 3 case).

# Get the top two results based on scores, handling potential ties
top_two_results = heapq.nlargest(2, results, key=lambda result: result.score)

# Extract and align IDs, considering potential ties
top_two_ids = sorted({result.id - 1 for result in top_two_results}) # Remove duplicates

# Get corresponding names, checking for valid IDs
top_two_names = []
for aligned_id in top_two_ids:
 if 0 <= aligned_id < len(speakers):
  top_two_names.append(speakers[aligned_id])

 else:
  print(f”Invalid ID {aligned_id + 1} encountered.”)

print(“Top two speakers: “, top_two_names)

Top two speakers: ['Ronaldo', 'Ronaldo2']

Yes! It’s a match. We have successfully verified the new voice with the existing set of voices.

Conclusion

In this article we implemented the audio-driven speaker recognition with just a few lines of code by using open-source technologies such as Resemblyzer and Qdrant DB. Resemblyzer is the easiest way to work on audio data and encode them into embeddings. There is no need for a neural network or transformer architecture. Qdrant DB, on other hand, provides an efficient way to store and retrieve embeddings.

Thanks for reading this article. Don’t forget to upvote, follow and subscribe my newsletter on Medium.

References

Qdrant Documentation — Qdrant

resemble-ai/Resemblyzer: A python package to analyze and compare voices with deep learning (github.com)