A lot of AI engineering comes down to decisions that do not have one clean answer. Which trade-off matters more here? Is this system actually failing because of the model, the context, the retrieval layer, or the infrastructure around it? When should you optimize, and when should you leave something alone?

This week’s issue is full of those kinds of decisions.

You’ll learn:

• Why continuous batching changes LLM serving efficiency

• Which vLLM settings actually matter and what each one controls

• How quantization, distillation, and speculative decoding change inference trade-offs

• Why a second GPU may be solving a KV-cache problem, not a model-size problem

• How to preserve conversational context for retrieval without sending the entire history every time

There’s also a community-built Qwen 3.5 implementation in C, a new poll on how people are working around coding-agent limits, and a few collaboration opportunities from the community.

Let’s get into it!

What’s AI Weekly

I create a lot of resources on becoming an AI engineer. We have courses, tutorials, videos, a book, and even a GitHub roadmap that aims to organize the entire learning path.

All of that is useful. But when I think about how I actually learned most of what I know, a huge part of it came from asking people questions. A lot of questions.

For me, the problem was always access. I had to find someone who knew more than I did, hope they had time to answer, wait for a response, and often realize that the answer created three more questions.

That is really the idea behind Towards AI Mentorship. We wanted to create a place where you can ask the questions that become too specific for a course, and where an AI can give you an answer but perhaps shouldn’t be the only opinion you trust.

Instead of having one person you can occasionally ask, you have access to 15+ senior AI engineers and mentors. You can ask questions asynchronously, join two live sessions each week, bring in your architecture or project for feedback, and get help on the career side too, including resumes, portfolios, and projects.

You can read more about why we built it in my blog post, or see everything included on the mentorship page.

AI Tip of the Day

In a multi-turn conversation, the user’s latest message is written for the conversation, not for your retrieval system. It often depends on details mentioned several turns earlier.

A simple example we use in the Context Engineering lesson of our Agent Engineering course shows why it is important to know this. A user reports a mild headache and wants to avoid medication. A few turns later, they ask: “Could stress be causing this?”

If you retrieve using only that question, the system no longer knows what “this” refers to. It also reduces the severity of the headache and hides the user’s preference for avoiding medication.

Instead, maintain a small session state with the important facts from the conversation. Keep the last two or three raw messages as well, then build the retrieval query from that state plus the newest message.

To check whether this actually improves retrieval, test it on 20 follow-up questions. Run each one twice: once with only the latest message, and once with the session state. Track:

• Whether a relevant source appears in the top five

• Whether any important user constraint is missing from the query

This gives retrieval the context it needs without sending the entire conversation back to the model every time.

— Louis-François Bouchard, Towards AI Co-founder & Head of Community

Learn AI Together Community Section!

Featured Community post from the Discord

Exquisite_peacock_20933 shared qwen35-in-c, a from-scratch C implementation of Qwen 3.5 inference. Rather than relying on PyTorch or a full serving framework, the project exposes the lower-level operations involved in loading the model, running its attention and linear-attention layers, maintaining state, and generating tokens. It is especially useful if you want to understand what an LLM inference runtime is actually doing beneath libraries such as Transformers or vLLM. The implementation is small enough to inspect rather than another abstraction to learn around. Check out the repo and support a fellow community member. If you have questions or feedback, share them in the thread.

AI poll of the week!

What stands out to me is that 47% of you hit your subscription limit at least once a day, while another 29% rarely or never do. That is quite a wide gap for the same generation of coding tools.

I don’t think this necessarily measures how much someone codes with AI. It may measure how they have designed their workflow around AI. Long-running sessions, large contexts, higher reasoning effort, parallel agents, and more autonomous tasks can consume dramatically more capacity than short, bounded tasks. Both Claude Code and Codex explicitly meter usage differently depending on context, task complexity, model choice, and agentic work.

So I’m curious whether heavy AI users eventually split into two camps: those who keep buying more capacity and those who redesign their workflows to make each token go further.

For those hitting the daily limit, what have you optimized first: model routing, shorter contexts, splitting work across agents/providers, API fallbacks, or simply paying for more capacity? And for the heavy users who rarely hit a limit, I’m especially interested in your setup. That’s the more useful workflow to learn from. Share it in the thread!

Collaboration Opportunities

The Learn AI Together Discord community is flooding with collaboration opportunities. If you are excited to dive into applied AI, want a study partner, or even want to find a partner for your passion project, join the collaboration channel! Keep an eye on this section, too — we share cool opportunities every week!

1. Siammzm is looking for a learning partner to build AI skills through studying, experimenting, discussing ideas, and building things. If you are also working towards the same goal, connect with them in the thread!

2. Likelytobelaura. is building ConspiracyBench, a benchmark to evaluate whether models believe various conspiracy theories, and is looking for someone interested in working on the benchmark and eventually fine-tuning the model. If this sounds interesting, reach out to them in the thread!

3. Prafullmishra. is starting a group to build ML projects and share resources. If you are pursuing a similar track, contact him in the thread!

Meme of the week!

Meme shared by bin4ry_d3struct0r

TAI Curated section

Article of the week

LLM Continuous Batching Explained: The Secret Behind Fast LLMs by Divy Yadav

Continuous batching is why LLM responses stay fast even under heavy concurrent load. This article builds on that idea from single-request serving and static batching, then explains how prefill, decode, PagedAttention, and chunked prefill fit into the same system. It also separates continuous batching from dynamic batching and speculative decoding, with benchmarks demonstrating that scheduling alone can yield substantial throughput gains.

Our must-read articles

1. Tuning vLLM: What Every Setting Does to the Arithmetic by Satsawat Natakarnkitkul (Net)

vLLM has dozens of settings, but only a handful usually deserve attention, and each controls a different bottleneck. This article maps six important knobs to the resource they actually change: concurrency, KV-cache capacity, work per scheduling step, prefix reuse, KV-cache precision, and eager execution. It also explains why vLLM’s unified token-budget scheduler allows chunked prefill, prefix caching, and speculative decoding to work together rather than as separate optimizations. Most importantly, it shows how to tune against your own traffic using TTFT, TPOT, throughput, and goodput instead of chasing a single benchmark number.

2. The Ultimate Guide to LLM Inference Optimization by Ashish Abraham

A large model can fit on paper and still be impractical to serve once weights, KV cache, latency, and throughput are factored in. This guide works through the model-level techniques that make inference cheaper: quantization, distillation, and speculative decoding. It shows the arithmetic behind mapping FP32 weights to lower-precision formats, explains when post-training quantization differs from quantization-aware training, and compares knowledge transfer through logits with Orca-style reasoning traces. It then shows how approaches such as EAGLE and Medusa use draft predictions to accelerate training of larger models without changing their final output distributions.

3. Polynomial Degree And Turning Points, Made Simple, Really Simple by Kamrun Nahar

You can often predict what a function will look like before plotting a single point. This article builds that intuition through real examples instead of starting with definitions: UK stopping distances reveal how linear and quadratic terms combine, the MPG illusion shows what happens when you invert a ratio, and Runge’s classic experiment explains why higher-degree polynomials can behave badly at the edges. Tax brackets, skid marks, and Anscombe’s quartet extend the same reasoning to other function families.

4. Your Second GPU Is Bought for the Cache, Not the Model by Satsawat Natakarnkitkul (Net)

Adding GPUs does not solve one scaling problem; it can solve several completely different ones. This article separates the reasons to scale into model capacity, throughput, and KV-cache capacity, then maps each to data, tensor, pipeline, or expert parallelism. It also covers the problem that another GPU alone does not fix: prefill and decode competing for the same hardware, which systems such as DistServe, Mooncake, and NVIDIA Dynamo address by separating them into different pools and moving KV cache between them.

If you want to publish with Towards AI, check our guidelines and sign up. We will publish your work to our network if it meets our editorial policies and standards.

There’s a big difference between getting an AI agent to write code and getting it to produce code you can actually ship. This week, I’m sharing the workflows we use at Towards AI to close that gap, from context and debugging to testing and review.

I also have a simple rule for testing stochastic LLM systems that catches failures a single successful run can hide. And if you use Claude Code or Codex regularly, I’m opening up my own AI engineering setup in a live workshop with Packt, including how I work across agents, manage usage limits, and keep skills and knowledge synced.

Plus, this week’s reads will help you:

• Understand what an FFT is actually doing, with working Python

• See why long context can make your KV cache larger than the model

• Reduce inference cost with prompt caching and KV offloading

• Understand how LLM text watermarking works and where it fails

• Build persistent agent memory beyond basic RAG

Let’s dive in!

What’s AI Weekly

This week, in What’s AI, I am sharing what the Towards AI engineering team uses to ship reliable code with AI agents, including three AI coding workflows, context rules, debugging habits, tests, and review guardrails. There is still a huge gap between the people who ship working products with agents and the people who ship working demos. This article shares everything you need to close that gap. By the end, you will have the best practices for your own repo, whether you are a developer learning to direct agents or a beginner upgrading from random vibe coding to something that ships. Read the full article here, or, if you prefer, watch the walkthrough on YouTube.

AI Tip of the Day

A student in our 10-Hour LLM Fundamentals Video Course asked us a useful question: How many times should you run an important test before trusting the result?

For stochastic LLM tests, our practical baseline is five runs.

Suppose a difficult test passes 90% of the time. If you run it once and it passes, you might conclude that everything is working. Run the same test five times, though, and there is about a 41% chance you will see at least one failure.

For important tests, run the same case five times and record two things:

• The pass rate across all runs

• Whether it passed all five times

Keep the prompt, model, temperature, and other randomness settings, as well as the source context, fixed. Otherwise, you are changing the test while trying to measure its consistency.

Five runs is not a statistical guarantee. It is a simple way to catch intermittent failures that a single successful run can easily hide.

— Louis-François Bouchard, Towards AI Co-founder & Head of Community

Live Workshop: Build Your Personal AI Engineering System

In other exciting news, I’m doing a 90-minute live workshop with Packt on how I’ve structured my own AI engineering setup around coding agents like Claude Code and Codex.

If you’re already using agents in your workflow, this should be especially useful. You’ll see how I:

• Keep skills, instructions, and knowledge synced across agents

• Turn repeated corrections into reusable skills

• Decide what stays in context and what gets retrieved

• Split work between Claude Code and Codex

• Work around usage and token limits

• Set up scheduled and always-on tasks

• Use an open-source vault template you can adapt

Who it’s for: AI engineers and developers already using coding agents who want a more structured system around them.

• When: Tuesday, September 8, 2026

• Time: 8:30 PM-10 PM GMT+5

• Where: Online

Register here

Learn AI Together Community Section!

Featured Community post from the Discord

Jose_sonder has built SkillNet, an open-source adaptive learning system. Users can share a topic or existing material, and it builds a structured course that presents the same knowledge differently for each learner. It can run as a shared space for an organization or class, or as an individual learning workspace. In a dynamic course, the shared knowledge and objectives remain stable, while the explanation, activities, support, and interface adapt based on the learner’s declared preferences, role, level, and progress. Check it out on GitHub and support a fellow community member. If you have any questions or feedback, share them in the thread.

AI poll of the week!

Most of you are still on free or basic plans, which makes me wonder how coding agents actually fit into your work today. Are they already handling a meaningful share of your coding and automation, or are you still bringing them in for specific tasks where they are particularly useful?

I’m also curious about the people at either end. If you’re paying for 5x or more, what makes the extra capacity worthwhile? Is it simply that you use agents much more often, or is staying within a familiar Claude Code/Codex workflow valuable enough that you would rather pay than move work elsewhere?

And the 8% running locally might be the group I’m most curious about: what are you running, on what hardware, and which parts of your workflow have you been able to move local?

There are probably some very different AI engineering setups hiding behind these percentages. Share yours in the thread, especially what you use, what you pay for, and what still runs locally.

Meme of the week!

Meme shared by bin4ry_d3struct0r

TAI Curated section

Article of the week

How The Fast Fourier Transform Actually Works by Kamrun Nahar

If you have used a Fast Fourier Transform (FFT) without really understanding what happens underneath, this article builds the intuition from the ground up. It starts with sine waves, implements the Discrete Fourier Transform in Python, and checks the result against NumPy before explaining how the FFT makes the same computation practical at scale. You’ll also see how sampling rate, aliasing, spectral leakage, and windowing affect the frequencies you recover from real signals, then apply the ideas to spectrograms, compression, MRI, and audio forensics.

Our must-read articles

1. Azure Storage for AI Inference: Prompt Caching, KV Offload, and Faster Model Loading by Dave R

Inference performance depends on more than GPU compute. This article shows how storage can reduce the amount of work your GPUs repeat. It explains Microsoft Foundry’s three levels of prompt caching, including Azure Context Cache for sharing cached prefixes across nodes, then shows how NIXL and LMCache move KV caches beyond GPU memory. You’ll also learn how Run:ai Model Streamer and AKS caching reduce model-loading time, giving you a practical map of where storage can improve latency, GPU utilization, and inference cost.

2. Your KV Cache Is Bigger Than Your Model by Satsawat Natakarnkitkul

Long context can exhaust GPU memory even when the model itself fits comfortably. This article shows you exactly why by calculating KV-cache memory from a model’s architecture, sequence length, precision, and batch size. Using gpt-oss-120b on two H100s shows that capacity drops from 504 concurrent conversations at 4K tokens to just 15 at 128K tokens. It then works through six ways to recover capacity, including grouped-query attention, sliding windows, prefix caching, paging, and FP8 quantization.

3. Watermarking Text Generation Efficiently by Enzo Lombardi

Text watermarking can work without inserting hidden characters or maintaining a list of special words. This article explains how watermarking can modify token probabilities during generation, including the green-list approach and the distortion-free method used in Google’s SynthID-Text. It then derives how a simple z-score can detect the resulting statistical pattern and tests both methods inside a real inference engine. You’ll also see where the approach breaks down: changes in paraphrasing, translation, and tokenization can weaken or remove the signal.

4. Beyond RAG: Implementing Agentic Memory Architectures for Long-Horizon AI Systems by Sourav Mukherjee

RAG can retrieve what an agent needs in the moment, but it does not automatically help the agent learn from past events. This article builds a memory architecture that separates episodic memory (what happened), semantic memory (what the agent learned), and procedural memory (how it should behave). It then adds consolidation to turn past interactions into reusable knowledge and reflection to promote successful lessons into future behavior. Comparisons with Mem0, Letta, and LangGraph help you see how these ideas translate into existing memory frameworks and what you need beyond retrieval for agents that operate across many sessions.

If you want to publish with Towards AI, check our guidelines and sign up. We will publish your work to our network if it meets our editorial policies and standards.

We recently gave our Context Engineering in 2026 workshop at the AI Engineer World’s Fair, using our production AI tutor to test what actually happens as an agent’s context grows. We spent close to $590 on the Gemini experiments alone, and some of the results contradicted the defaults we had already shipped.

Everything is open source. You can inspect the full tutor codebase and eval harness, run the live experiment demo, see every result, or go through the workshop slides and watch the recording.

If you’re managing context in an agent today, these are the findings I would start with:

1. Measure what is filling the window before trying to shrink it. In our tutor, chat history wasn’t the main problem. Old retrieval and tool outputs were. Retrieval alone could contribute up to 100k tokens, pushing some turns toward 200k. Optimizing conversation history first would have attacked the wrong part of the context.

2. Check your cache economics before summarizing anything. Our production compaction setup sent roughly 41% fewer tokens than keeping the complete history and still cost about twice as much. Keeping history allowed around 87% of the input to receive the cache discount; summarization rewrote that prefix and lost it. On our traces, keep-everything remained cheaper while cached input cost less than roughly $0.55 per million tokens.

3. Shrink before you rewrite. Capping tool outputs at a stable size cut cost per turn by 38% with no measured loss in memory. Unlike summarization, it reduced the context without continually changing the cached prefix. If that isn’t enough, move information into files or retrieval and leave pointers behind so the agent can recover it later.

4. Test the capability you cannot afford to lose. Our compacted tutor could score 97–99% on overall response quality while forgetting information from earlier in the conversation. Generic quality evals didn’t catch it. We had to plant facts early in a session and explicitly probe for them later.

5. Let the constraint choose the technique. If the window doesn’t fit, retrieve instead of stuffing everything in. If cost is the problem, inspect cached-input pricing or switch models before adding summarization. If quality is deteriorating, measure what is being lost and where. In our corrected tests, distinctive facts remained retrievable out to 800k tokens, while ambiguous facts degraded.

And keep the simple retrieval tools around. Dense retrieval dropped to 0% recall at a 400k-token corpus in one buried-fact test while BM25 remained at 100%. We also tested GraphRAG against ordinary hybrid retrieval and achieved the correct source 100% of the time with both methods, while GraphRAG consumed 61% more input tokens and cost 44% more per turn. More sophisticated did not mean better for our data.

Our production setup is now much simpler: DeepSeek V4 Flash, hybrid retrieval, full history, stable caps on tool outputs, no tool-output clearing, and summarization pushed so far out that it rarely fires.

In the full article, I go much deeper into the experiments, eval harness, local-model tests, context rot, retrieval, caching, and the things we built that ultimately weren’t worth shipping.

The principle to remember is to name the constraint before choosing the context strategy. Window, cost, and quality degradation are different problems, and the same optimization will not solve all three.

You can read the full article, watch the workshop, or go directly to the code, live experiments, and slides if you want to dig into the results yourself.

AI Tip of the Day

The tutor from our Context Engineering workshop is the same one we build in the Full Stack AI Engineering course. While updating that course, we found another retrieval issue: hybrid search can still lose an exact match if you merge results too early.

Semantic search is good at matching meaning. Keyword search is better for exact strings such as product IDs, error messages, names, and code symbols.

Say a user searches for order #8821. Keyword search may put the exact page first, while semantic search returns several broader pages about orders. If you combine both lists immediately and keep only the highest-ranked results, those broader semantic matches can push out the exact result.

Our fix is simple: keep the strongest candidates from each retriever before combining them. We take the top five keyword results and the top five semantic results, then merge and deduplicate them.

This also aligns with one of the workshop findings: in our buried-fact tests, dense retrieval eventually failed to retrieve distinctive terms, while BM25 kept finding them.

The practical takeaway: hybrid search only works if your fusion strategy preserves the different strengths of both retrievers.

If exact identifiers matter in your application, reserve part of the final context for strong keyword matches.

— Louis-François Bouchard, Towards AI Co-founder & Head of Community

TAI Curated Section

Article of the week

NVIDIA’s NOOA Proves the Harness Matters More Than the Model, and Everyone’s Watching the Wrong Number by allglenn

NVIDIA’s NOOA reached 82.2% on SWE-bench Verified with GPT-5.5 while using about half the tokens of comparable harnesses. The article explains how much of that gain comes from the surrounding system rather than the model itself. NOOA passes tool outputs by reference rather than repeatedly inserting them into the prompt, which reduces context growth and preserves prompt-cache hits. It also breaks down six harness capabilities NVIDIA considers important, including typed contracts and model-controlled context management.

Our must-read articles

1. AI Agent Architecture Demystified: From LLMs to Harnesses, Loops, and AgentOps by Kaush B

This article gives a clear structure for understanding the different layers of an agent system. It separates prompt and context engineering from the agent loop, harness, graph structure, and AgentOps, then shows how they fit together. It is especially useful if you are trying to decide whether a problem should be solved with better context, a different workflow, stronger execution controls, or more observability.

2. Agentic Analytics with Power BI and Microsoft Fabric: The AI-First Way to Build Enterprise BI by Dave R

This piece shows how coding agents can work directly with Microsoft Fabric and Power BI rather than only generating code around them. It covers agents creating lakehouses, working with semantic models, editing Power BI project files, and scaffolding applications, all while operating within existing governance and security controls. The main value is seeing how agentic workflows can fit into an enterprise data stack without bypassing the systems already in place.

3. Stop Using Long-Context Windows for AI Agents (Build This Instead) By Udaykiran Estari

This article separates long context from persistent memory and explains when each is useful. It maps memory across working state, caches, persistent stores, and model weights, then shows how systems such as Redis, vector databases, and graph stores can support retrieval across sessions. It also covers practical risks, including stale memories, conflicting information, poisoning, and privacy concerns, which become important once an agent is expected to remember beyond a single conversation.

4. Skills, Hooks and Subagents: Optimizing Claude Code Around What Stays in the Window By Utkarsh Mittal

This article looks at Claude Code’s features through the lens of context cost. It compares instruction files, skills, hooks, subagents, and agent teams, showing how the same repository and model can produce up to a sevenfold difference in cost depending on what remains loaded. The practical takeaway is how to decide what should stay permanently in context, what should load only when needed, and what should be moved into a separate agent or enforced through a hook.

If you are interested in publishing with Towards AI, check our guidelines and sign up. We will publish your work to our network if it meets our editorial policies and standards.

Coding agents can now take on enough work that the question is no longer just how much faster they make us. It’s how closely we still need to watch them.

This week, I look at where vibe coding works well, where it starts to get risky, and what I’ve learned about using coding agents without handing over the engineering judgment with the code. There’s also a small observability detail that can completely distort your reliability and cost numbers if you get it wrong.

Then, a few reads worth your time:

• See why testing an agent means evaluating the path it took, not just whether the final answer looks right.

• Give enterprise agents the right context without wiring every source into the application yourself.

• Catch a Claude Code scheduling behavior that can quietly repeat expensive work.

• Find out what each agent and workflow is actually costing you, instead of relying on one provider-level bill.

• Build and deploy an entire backend to Microsoft Fabric from TypeScript.

We also finally share Towards AI Mentorship here, plus a new community-built coding copilot that reports cutting codebase context by 70–80%.

Let’s get into it!

What’s AI Weekly

This week, in What’s AI, I dive into something that has become an integral part of software development: vibe coding. Vibe coding with AI agents can speed up software work, but what I want to talk about is how it is a game changer only IF you use it correctly. To use it right, you need good context, tests, reviews, and task boundaries. The risky version of vibe coding is shipping whatever the agent produced because it looked like it worked. Read how to do it right or watch the video version on YouTube.

AI Tip of the Day

A retry is not a new user request. Your traces should reflect that.

In the Opik observability lesson from our Agent Engineering course, we trace model calls and tool calls across an agent run. One issue that comes up quickly is how retries should be recorded.

If every retry is counted as a separate request, a single user request can appear several times in your dashboard. That inflates request volume and makes it harder to see how many attempts the agent actually needed to succeed.

Use the same request ID across every retry, and add an attempt number for each one. Keep separate trace and span IDs for the individual operations.

This lets you measure both the number of user requests and the number of attempts required to complete them.

That distinction is important to note for cost and reliability. A request that succeeds after three attempts may look successful in the dashboard, while using far more time and tokens than a request that succeeds on the first try.

— Louis-François Bouchard, Towards AI Co-founder & Head of Community

A couple of weeks ago, we opened something new at Towards AI that I haven’t had a chance to share here yet.

One thing we kept hearing from students was that learning the material wasn’t always where they got stuck. The harder questions came afterward: Is this architecture actually a good idea? Why is my agent failing with real users? Is this project strong enough for my portfolio? Why isn’t my resume getting through?

Those questions are difficult to solve with another lesson or another chatbot response because the answer depends on your specific work.

So we now have Towards AI Mentorship, where you can bring those questions directly to our team of 15 senior AI engineers. That includes async technical and career help, live sessions twice a week, resume and project reviews, monthly production blueprints from our deployment work, and workshops with engineers working in the field.

It’s $99/month, and you can use it whether you’re trying to land an AI role or already building AI systems and want experienced engineers to sanity-check the decisions you’re making.

Learn more about Towards AI Mentorship

Learn AI Together Community Section!

Featured Community post from the Discord

Ang_0007 built PariPari, a repo-aware AI copilot that uses Paritok context compression to explore massive codebases, fix bugs, and generate PRs without blowing up LLM token limits or budgets. It is built with Python and FastAPI to manage the agent loop, tool execution, and GitHub API interactions, and uses the Groq API. He reports achieving an average of 70–80% token reduction on codebase file reads using Paritok compression. Check it out and support a fellow community member. If you have any questions or feedback, share them in the thread.

AI poll of the week!

The results barely moved as more people voted: 57% still run long coding-agent tasks on their main machine. What I find more interesting is how fragmented the other 43% are. There doesn’t seem to be a clear second choice yet. Some people rely on provider-hosted environments, while others have moved to dedicated machines or their own servers.

That makes me curious about what these setups actually look like in practice.

For those running agents somewhere other than your main machine: what does your setup look like? What are you running, where does the code live, how do you connect to the agent, and do you typically have one task running or several at once? I’d love to hear what has actually worked for you.

Meme of the week!

Meme shared by supastishn

TAI Curated section

Article of the week

A Field Guide to Agentic Eval Frameworks: Langfuse, LangSmith, and What to Measure by MongoDB

This article examines why traditional unit tests miss trajectory-level failures in AI agents, using a research agent that hallucinated a published report, misused a search tool, and looped through failed warehouse queries. It defines six evaluation dimensions: task success, trajectory quality, tool correctness, safety, factual accuracy, and cost, then outlines rule-based checks, LLM-as-judge grading, and human review across component, trajectory, and end-to-end testing.

Our must-read articles

1. Microsoft’s Four IQs: How Foundry IQ, Fabric IQ, Work IQ, and Web IQ Ground Enterprise Agents by Dave R

This article breaks down Microsoft IQ, the context layer that grounds AI agents in four kinds of enterprise knowledge: unstructured documents, structured business data, human work signals from Microsoft 365, and fresh information from the web. It covers what each of the four engines (Foundry IQ, Fabric IQ, Work IQ, and Web IQ) does, how they fit together inside a single agent architecture, and where Foundry IQ unifies the others. It also walks through a worked refund processing agent so you can see how the pieces cooperate, why your agent instructions still matter after the IQs remove most of the plumbing, and how Agent 365 gives an agent its own identity and security boundary.

2. The Claude Code Idempotency Test That Prevents Runaway Agent Costs by Udaykiran Estari

A scheduling flaw in Claude Code’s ScheduleWakeup mechanism re-fires one-shot slash commands instead of resuming paused tasks, silently doubling API costs on database writes, pull requests, and expensive searches. This article shows how to choose between loops, skills, subagents, and workflows without burning budget.

3. Adding Cost Metering and LLM Spend Visibility to a Multi-Agent System by MongoDB

Multi-agent LLM systems break provider billing dashboards, which report spend by key or model but never by agent, workflow, or trace. This piece details a metering layer that captures token usage at each call, enriches it with runtime context, and prices it against a versioned rate card stored in a separate collection. Aggregation pipelines then slice costs by agent, model, or outcome, feeding Atlas Charts dashboards.

4. Rayfin: Define a Full App Backend in TypeScript and Ship It to Microsoft Fabric by Dave R

This article explains how Rayfin works, from the TypeScript you write to the services that run in Microsoft Fabric. Rayfin is an open source SDK and CLI that lets you define an application backend, including its database, access policies, APIs, and server-side logic, entirely in code, then deploy it to Fabric with a single command. It walks through the programming model, the decorators that turn classes into tables, the CLI workflow, how connectors reach existing data, and how the whole application lands as a governed Fabric artifact.

If you want to publish with Towards AI, check our guidelines and sign up. We will publish your work to our network if it meets our editorial policies and standards.

Learning AI has changed dramatically over the past year. Building a working prototype is no longer the hard part. The challenge now is understanding the decisions behind it: what to build, how to structure it, where it should run, and how to make it reliable.

This week:

• Learn why persistent memory is becoming essential as coding agents tackle longer and more complex projects.

• See what happens when the cost of observability grows faster than your application and when self-hosting starts to make sense.

• Understand why agent memory needs governance, not just a larger vector database.

• Follow the same AWS agent built four different ways to see exactly what each layer of abstraction gives you.

• And if you’re working with Spark, learn how one memory management decision can make the difference between a failed job and a successful one.

Let’s get into it.

What’s AI Weekly

This week in What’s AI, I am walking through how I’d learn AI Engineering in 2026. You can open Codex, Claude Code, or Cursor today, describe an app in English, and get a convincing result in minutes. The agent can scaffold a RAG pipeline, add API routes, write tests, and prepare a deployment. That speed is genuinely useful. It is also one of the easiest ways to fool yourself while learning AI. When the first version is easy to generate, your value moves to the decisions around it. That is my idea behind AI engineering today. And I have put together the practical learning plan. Read the full article here or watch the video on YouTube.

AI Tip of the Day

When we built the “Research and Writing Agent with MCP” lesson for our Agent Engineering course, we ran into a deceptively simple evaluation question: which articles should the judge score?

The easy option was to ask another LLM to generate a batch of articles and use them as the test set. But those articles would never pass through the system we actually wanted to evaluate. The judge might measure writing quality, while missing failures in research, context transfer, or the handoff between agents.

So every evaluation example now follows the production path. We begin with the same brief a user would submit, run the research agent, pass its saved findings into the writing workflow, and score the final article.

This turns the output into a test of the entire system. If the research is weak, an important source is lost, or the writer receives incomplete context, the failure appears in the article, where the judge can detect it.

The takeaway: generate evaluation outputs with the workflow you plan to ship. Otherwise, you may be testing the quality of a substitute model rather than the reliability of your own system.

— Louis-François Bouchard, Towards AI Co-founder & Head of Community

Learn AI Together Community Section!

Featured Community post from the Discord

Vardhan0902 has built Forefetch, a platform designed for a future where AI agents work more like teammates than assistants. Instead of tying long-running tasks to your laptop, Forefetch gives agents a persistent environment where they can keep working, maintain state across sessions, and finish jobs independently. As coding agents become capable of tackling increasingly complex projects, it’s an interesting glimpse at what an agent-first development workflow could look like. Check it out and share your feedback with the creator in the thread.

AI poll of the week!

More than half of you still run long Claude Code or Codex tasks directly on your main machine. That makes sense for day-to-day work, but as agents start running for hours instead of minutes, your setup becomes just as important as the model you’re using. I’m curious whether this is simply the most convenient option today, or whether people have found that the trade-offs aren’t big enough to justify moving elsewhere.

I’d love to hear more about your setup. Do you usually have one agent running at a time, or several in parallel? What does your workflow look like, and what made you settle on your current environment? Let’s talk in the thread!

Collaboration Opportunities

The Learn AI Together Discord community is flooded with collaboration opportunities. If you are excited to dive into applied AI, want a study partner, or even want to find a partner for your passion project, join the collaboration channel! Keep an eye on this section, too — we share cool opportunities every week!

1. Isaac_tigges built a local coding agent that brings reasoning and verification to compact open models and needs help with code quality & architecture review. If you think you can help, reach out to him in the thread!

2. Itsha123_45386 and their team are trying to develop an alternative to biometric authentication systems like Face ID and Touch ID/fingerprint sensors. They need some volunteers to help them collect data for their dataset. If you would like to contribute, connect with them in the thread!

Meme of the week!

Meme shared by jacekde

TAI Curated section

Article of the week

Persistent Memory for Claude Code on MongoDB Atlas

Long Claude Code sessions silently lose constraints when compaction drops tool outputs and summarizes earlier exchanges. This article builds a plugin that captures those statements into MongoDB Atlas before they vanish, using a PreCompact hook and two MCP tools. Retrieval runs entirely inside the database: Automated Embedding generates Voyage vectors, $rankFusion blends vector and keyword pipelines through reciprocal rank fusion, and a hosted reranker orders results.

Our must-read articles

1. I Self-Hosted Langfuse so My LLM Traces Would Stop Living On Someone Else’s Bill

Langfuse crossed 100K monthly traces and pushed managed pricing into painful territory, so the author self-hosted the open-source observability platform instead. Docker Compose spins up Postgres, ClickHouse, Redis, and MinIO in minutes, though production requires TLS, SSO, and eventually Kubernetes with externally managed dependencies. The article flags a notorious UTC timezone bug, treats Redis queue depth as the real health signal, and lays out honest cost math: self-hosting secures data control immediately, but only beats managed pricing at scale.

2. How Spark Manages Memory — The Unified Memory Model, Spills, and AQE

A daily 2.1TB aggregation job kept dying at 85% completion, and the author traced the failure to Spark’s Unified Memory Manager rather than raw heap size. This article walks through how execution memory always wins over storage in the borrowing contract, why protected cached blocks starved a skewed aggregation of room to grow, and how memoryOverhead, not executor.memory, killed the YARN container. Adaptive Query Execution’s skew-join splitting fixed the root problem, cutting cluster cost while pinpointing exactly which memory region overflowed.

3. Agent Memory Is the Real Moat

Agent memory becomes dangerous the moment it is treated as an unbounded vector store rather than a governed lifecycle. This article maps four distinct memory classes: working checkpoints, episodic outcomes, semantic facts, and procedural lessons, each with its own ownership and retention rules. Benchmarks from TraceRetain and LoCoMo show unbounded memory collapsing under noisy writes while selective retention holds steady. It closes with a practical adoption blueprint.

4. Road to Bedrock AgentCore, From a Single API Call to a Production Agent

Building the same weather-fetching agent four times across AWS’s tooling stack exposed exactly what separates each layer. This article implements the same core function on the Bedrock Converse API, Bedrock Agents, the Strands SDK, and AgentCore, keeping the logic identical to isolate each layer’s tradeoffs. Converse demands hand-written orchestration. Bedrock Agents hands the loop to AWS at the cost of visibility. Strands restores control with model portability. AgentCore deploys the same agent unchanged, adding Gateway, Memory, Identity, and Observability as managed infrastructure.

If you want to publish with Towards AI, check our guidelines and sign up. We will publish your work to our network if it meets our editorial policies and standards.

When I first moved away from academia, I had a fairly glamorous picture of engineering in my head. Three monitors. Beautiful code. Blackout curtains. Hours spent quietly building.

AI engineering cured me of that fantasy very quickly. Especially since ChatGPT and now even more powerful agentic systems to help us at each step.

Most of the job isn’t writing code. It’s making decisions. Which model fits the constraints? Do we need retrieval? How should we evaluate this? Is the problem in the data, the pipeline, the prompt, or the model? One decision after another, each one changing the one that follows.

You can get surprisingly many of them wrong and still build an impressive demo. You just can’t build something people should rely on.

We’re launching something that makes these decisions less of a solo adventure, more reliable, and honestly more fun, with the right team behind you.

It’s called Towards AI Mentorship. Our engineering team- Omar, Fabio, Samridhi, Louie, and 11 more engineers- on call for your questions.

See what’s inside →

The biggest thing: you stop making production calls alone. You post a question- architecture, deployment, eval strategy, career- and an engineer who’s shipped that system writes you back. Not when they happen to be online. As their job. That shift, knowing someone will actually answer, changes how you approach every decision.

Then there are the production blueprints, and this is the part I’m quite excited about. Every month, we hand over best practices from our deployment work, or something we’ve actually built for clients, with the judgment calls behind every decision. The first one is mine: my current setup and our team’s best practices for working with agents as engineers. The same setup we use daily. You adapt it in your repo, and it becomes a portfolio project built on real engineering thinking, not a tutorial you followed.

Live Q&A runs twice a week across two time zones, so you’ll always find a session that works.

Resume and portfolio reviews come back in 48–72 hours with written feedback from people who’ve been on the hiring side.

Python for LLMs and LLM Fundamentals are included from day one, 25% off everything else, and early access to courses we’re still building.

You know the people behind this. You’ve seen Omar run workshops. You’ve read Fabio’s newsletter. You’ve watched Samridhi present her AI scribe work. They’ve been answering questions in our channels for years; the difference now is that it’s structured and every question gets a response.

I’ll be honest, we just launched this. We’re going to learn as we go and get better every month. That’s how we built the courses, and I think they turned out all right. The people who join first will genuinely shape what this becomes.

I’d love for that to be you.

Join for $99/month →

$899/year saves 24% · 30-day money-back on yearly · cancel monthly anytime

AI engineering is slowly becoming less about writing prompts and more about building systems that don’t surprise you in production.

That’s exactly where this week’s issue goes.

• One evaluation trick that’s easy to add but surprisingly effective.

• A practical look at tracing multi-agent systems when something inevitably breaks.

• A guide to choosing the right LangGraph pattern before reaching for full autonomy.

• Why GRPO is changing how many people think about RL training.

• And a story that explains why millions of data points can still give you the wrong answer.

Let’s get into it.

What’s AI Weekly

This week in What’s AI, I break down the agent pipeline I use to research, write, illustrate, package, translate, and publish content, while keeping the most important 10% human.

Nova handles research, Brown coordinates the writing workflow, parallel workers produce the visuals, and a separate loop develops titles and translations. Together, they have reduced two or three days of production work to roughly five hours of my time.

The useful part is not simply what I automated. It is how the workflow preserves my voice, where I still intervene, and why removing the final human decisions would make the output worse. Read the full breakdown, or watch the video to see the pipeline and diagrams in action.

AI Tip of the Day

When we designed the evaluation lessons for our 10-Hour LLM Fundamentals course, we looked for checks that improve reliability without forcing you to rebuild your pipeline. This is one of the simplest.

When an LLM judge compares answer A with answer B, run the evaluation twice and reverse their order on the second call.

If the same answer wins twice, keep the result. If the winner changes, mark the comparison as undecided or send it for human review.

This exposes position bias: the judge may prefer an answer partly because it appears first or second, not because it is better. Track how often your results flip. Frequent reversals can reveal an unclear rubric, answers that are too similar, or a judge that is not reliable enough for the task.

— Louis-François Bouchard, Towards AI Co-founder & Head of Community

Learn AI Together Community Section!

Featured Community post from the Discord

Mikieldus has built Hoshi, an agent workspace designed around a simple idea: an agent should do more than talk. Each user gets an isolated cloud computer with its own filesystem, shell, running processes, and credentials, so the agent can work directly with code and complete tasks. Teams can also publish their existing setup as a boilerplate, while interactive forms, charts, and choices replace long text transcripts when decisions are needed. Check it out here, give it a try, and share your feedback with the creator in the thread.

AI poll of the week!

Kimi may be leading some benchmark tables, but it has not yet displaced the models people already rely on. Only 10% of respondents regularly use it, while Claude and GPT account for nearly two-thirds of the selections.

It again reinforces that better scores may earn attention, but reliability, integrations, familiarity, and developer experience determine whether a model becomes part of someone’s daily workflow.

What would it actually take for you to switch your primary model? A clear performance advantage, lower cost, better tooling, stronger coding ability, or something else? Share what matters most in your day-to-day workflow.

Collaboration Opportunities

The Learn AI Together Discord community is flooded with collaboration opportunities. If you are excited to dive into applied AI, want a study partner, or even want to find a partner for your passion project, join the collaboration channel! Keep an eye on this section, too — we share cool opportunities every week!

1. Jadexrose is recruiting for AI engineering roles at startups and is looking for someone to discuss AI engineering concepts, do some mock interviews, and build stuff. If this sounds like it would help you as well, connect with them in the thread!

2. Divyanshijoshi is currently looking for people to work together on AI research, especially in interdisciplinary fields with a focus on machine unlearning, LLM bias mitigation, and studying bugs in software engineering. If this sounds interesting, reach out to her in the thread!

3. Mikieldus is looking for a team to develop his project. If you want to work on projects and need more details, write to him in the thread!

Meme of the week!

Meme shared by bin4ry_d3struct0r

TAI Curated section

Article of the week

When Your Agents Go Dark: Observability in Multi-Agent Systems with OpenTelemetry by MongoDB

When a multi-agent system fails, ordinary logs may show the error without revealing which agent, tool call, or handoff caused it. This article shows how to trace that path with OpenTelemetry. You will see how to instrument an agent handoff, inspect the resulting trace, and capture failures that application logs often miss. It also explains when full distributed tracing is worthwhile and when a simpler setup will do.

Our must-read articles

1. Building Intelligent Feedback Systems: A Deep Dive into Conditional Agentic Workflows with LangGraph by Sandip Palit

This piece walks you through building a conditional agentic workflow for a customer review triage system using LangGraph, LangChain, Groq, and Pydantic. The author uses llama-3.3–70b-versatile at zero temperature to extract sentiment, then routes reviews through a StateGraph: positive feedback triggers a quick thank-you node, while negative reviews are passed through structured diagnosis for issue type, tone, and urgency before generating tailored resolutions.

2. GRPO from Scratch: Group Relative Policy Optimization in Python by Armin Norouzi, Ph.D

GRPO removes PPO’s separate value model and estimates advantage by comparing outputs within a group, reducing memory use while simplifying the training loop. This article implements the algorithm from scratch, making the reward normalization, clipping, and stability trade-offs easier to understand before you use a production library.

3. Every LangGraph Pattern You’ll Actually Use: Explained Properly by Bessie Delight Kekeli

Not every workflow needs an autonomous agent. This guide compares six LangGraph patterns, from simple chains and routing to evaluator loops and orchestrator-worker systems, then helps you decide which structure fits the problem. It also grounds every pattern in working code, clarifying how structured output and tool binding turn a plain LLM into something genuinely capable. It is especially useful when you understand LangGraph’s components but are unsure how to assemble them.

4. How A Tiny Spoonful Can Know The Whole Pot by Kamrun Nahar

A large dataset can still give you the wrong answer when the sample is biased. Through famous historical examples and a simple Python simulation, this article shows why representativeness matters more than raw sample size and gives you a practical checklist for questioning the data behind a statistic. It closes with a five-step field kit and a flowchart for spotting tilted data before trusting statistics.

If you want to publish with Towards AI, check our guidelines and sign up. We will publish your work to our network if it meets our editorial policies and standards.

The AI discourse has a new buzzword this week, and for once, the underlying idea is worth your time, even if Anthropic already wrote about it in 2024. We’re also tackling a tradeoff most teams don’t measure: forcing structured outputs can make your model’s answers worse, and a 100% parse rate won’t tell you.

We also cover:

• How a journalist used real data and real statistics to convince the world that chocolate helps you lose weight, and what that teaches about why most people misread p-values.

• A multi-agent support triage system you can run locally on Llama3 or swap to GPT-4o-mini with one function change.

• DPO from scratch in pure NumPy: no reward model, no RLHF pipeline, just the math and a working implementation.

• An orchestrator pattern that decomposes tasks, runs workers in parallel, and self-corrects with a free demo mode so you can test every path without API costs.

• A full self-hosted LLMOps stack with Langfuse: prompt versioning, tracing, evals, and a CI gate.

Let’s get into it!

What’s AI Weekly

I made a video on loop engineering a few weeks ago, and apparently it died before some of you even got a chance to watch it. We now have Graph engineering, basically, this week’s name for connecting several agent loops into one orchestrated system. I am glad we are talking about it, but Anthropic’s Building Effective Agents post from 2024 covers each of these concepts already. We are talking about it now because drawing the graph forces you to think about the process and set it up once and for good. Graph engineering is this week’s name for taking orchestration a bit more seriously now that we have access to models like 5.6 and Fable. So today, in What’s AI, I am diving into what graph engineering actually is and which part deserves your attention.

Read the full article here, or if you prefer watching, check the video on YouTube.

AI Tip of the Day

Structured Outputs make responses easier to plug into software, but they also change how the model generates its answer.

When we first wrote the lesson ‘Structuring Your Data’ for our Full Stack AI Engineering course, the main problem was getting models to return JSON your application could reliably parse. As structured outputs became standard across newer reasoning models and smaller models used in production, another problem became harder to ignore: the schema can be perfectly valid while the answer inside it gets worse.

This is especially important on tasks that already stretch the model’s reasoning. Requiring it to solve the problem within a rigid schema can reduce accuracy, particularly when the schema is complex, or the model has little capacity to spare.

Before adding a schema, run the task in free form and record the quality of the answers. Then introduce the smallest schema your application needs and repeat the same evaluation.

Measure answer quality and schema validity separately. A 100% parse rate only tells you that your software can read the response. It does not tell you whether the response is correct.

When structure reduces accuracy, simplify the schema. For tasks where the final formatting is deterministic, let the model work through the problem freely and package its answer into the required structure in code.

— Louis-François Bouchard, Towards AI Co-founder & Head of Community

Learn AI Together Community Section!

Featured Community post from the Discord

Dr_zoe88 just built review-toolkit, a small toolkit that gives Claude Code two “reviewer” agents. One reviews your plan before you write any code, and one reviews your code before it merges. The repo ships with a test suite of planted flaws, and both reviews are allowed to say: “nothing to fix”. Try it out on GitHub and support a fellow community member. Share your feedback, questions, and if it catches something for you in the thread!

AI poll of the week!

Anthropic leads at 21%, but 18% of you don’t want to work at any of these companies; you want to build your own. Nearly 1 in 5 picked founder over every major lab on the list. NVIDIA takes third at 16%, which makes sense if you’re thinking long-term; betting on the infrastructure layer is a different kind of bet than betting on any one model provider.

Also, 5% of you said Towards AI; we see you, and we appreciate it. As we mentioned last week, the deployment pods do hire from this community first.

For those who picked founder: are you already building something, or is that the plan once you feel ready? And for those who picked a lab, what’s the pull? The research, the product, or just wanting to be where the biggest models get made? Let me know in the thread!

Collaboration Opportunities

The Learn AI Together Discord community is flooded with collaboration opportunities. If you are excited to dive into applied AI, want a study partner, or even want to find a partner for your passion project, join the collaboration channel! Keep an eye on this section, too — we share cool opportunities every week!

1. Jadexrose is recruiting for AI engineering roles at startups and is looking for someone to discuss AI engineering concepts, do some mock interviews, and build stuff. If this sounds like it would help you as well, connect with them in the thread!

2. Divyanshijoshi is currently looking for people to work together on AI research, especially in interdisciplinary fields with a focus on machine unlearning, LLM bias mitigation, and studying bugs in software engineering. If this sounds interesting, reach out to her in the thread!

3. Mikieldus is looking for a team to develop his project. If you want to work on projects and need more details, write to him in the thread!

Meme of the week!

Meme shared by bigbuxchungus

TAI Curated section

Article of the week

Hypothesis Testing Is Just a Courtroom With Worse Snacks by Kamrun Nahar

A journalist named John Bohannon convinced the world that chocolate helps you lose weight. He didn’t fake a single number. The study was real, the p-value was real, and that’s exactly the problem. This article uses that story to explain why p-values fool people, traces hypothesis testing back to Fisher’s tea-tasting experiment, and walks through null hypotheses, significance levels, Type I and II errors, power, and confidence intervals with runnable Python code throughout.

Our must-read articles

1. Building a Multi-Agent Support Ticket Triage With LangGraph by Sarah Lea

If you’ve been looking for a clean starting point for multi-agent systems, this is it. A classifier assigns category and priority, a router directs traffic, a responder drafts replies, and urgent tickets escalate to a human. One factory function lets the same graph run locally on Llama3 or through GPT-4o-mini, no architecture changes needed.

2. DPO Fine-Tuning from First Principles in Python by Armin Norouzi, Ph.D

Most DPO tutorials start at the loss function. This one starts at why the loss function works. The author derives everything from Bradley-Terry preferences through the reparametrization that eliminates the reward model entirely, then implements it in pure NumPy with analytical gradients. A beta sweep at the end shows how temperature controls KL drift in practice.

3. I Built a Team of AI Agents That Manage Themselves — Here’s the Orchestrator Pattern Behind It by Sai Insights

An orchestrator decomposes a question into subtasks, runs specialist workers in parallel over a shared blackboard, and a critic flags low-confidence answers for capped retries. What makes this useful: a free demo mode verifies every code path without API costs, and a live Claude mode lets you run it for real. The full execution logs are included so you can see exactly what happens at each step.

4. Langfuse from Zero to Production: Tracing, Prompt Versioning, and Evals in One Stack by Harish Ramkumar

If you want LLMOps without handing your data to a third party, this is the walkthrough. The author sets up Postgres, ClickHouse, Redis, and MinIO via Docker Compose, instruments a RAG pipeline with nested traces, adds versioned prompts with one-click rollback, and wires up LLM-as-judge scoring. A CI quality gate at the end turns every prompt change into a testable, reversible release.

If you want to publish with Towards AI, check our guidelines and sign up. We will publish your work to our network if it meets our editorial policies and standards.

This week is a mix of practical builds and a milestone for us. We also have a big announcement from Towards AI; more on that below.

Inside the issue:

• How Claude Code’s /goal and /loop commands turn one-shot prompting into autonomous loop engineering.

• Three context failure modes that show up when you manage them carelessly.

• A beginner-friendly path to your first LLM app on Amazon Bedrock, from IAM setup to RAG pipelines with guardrails, all for under a dollar.

• 83 vLLM configurations benchmarked on an L40S: open-source results showing how each optimization actually performs in practice.

Let’s get into it!

What’s AI Weekly

This week, I don’t have a video for you, but something even bigger.

It’s a big week for me at Towards AI: we just launched Towards AI Deployment, a dedicated enterprise division.

Here’s the part I’m proudest of: several of the AI engineers in those pods started right here, as community members and students. They trained with us, and now they’re deploying production AI inside client organizations. And everything they learn doing that work flows back here: the failure cases become AI Tips of the Day, the evaluation methods become course lessons. The context engineering patterns we shared at our World’s Fair workshop came from exactly this kind of work.

Two of our co-founders dive deeper into how we went from teaching machine learning to half a million practitioners to deploying AI directly inside businesses, and why we’ve made PE-backed companies and investors our focus. Watch the full video here.

And, if you’ve been building in public here, keep going. This community is the first place we look when the pods hire.

AI Tip of the Day

Agent evals should intentionally break the tools.

A clean staging run shows how an agent behaves when all dependencies cooperate. Production is where a tool times out after completing an action, or sends back a payload the agent cannot parse.

That is when agents repeat work, invent a successful result, or continue without the information they need.

Put these failures into the eval harness. Force a 429 response at a known step. In another test, make a required dependency unavailable. Record each tool call and assert what the agent should do next.

A read may be safe to retry automatically. A write should first check whether the original action already happened. If the task cannot continue safely, the correct result is a clear stop or escalation.

Measure recovery success separately from normal task completion. Otherwise, a high pass rate on easy paths can hide weak recovery logic.

To learn more about agent evaluation, tool use, and guardrails, check out our Agent Engineering: Building Multi-Agent Systems Course.

— Louis-François Bouchard, Towards AI Co-founder & Head of Community

Learn AI Together Community Section!

Featured Community post from the Discord

Vivek_m4 has built his first open-source project, Spider Canvas, an AI-powered rendering runtime that lets developers build, preview, and iterate on React, HTML, CSS, JavaScript, WebGL, and three.js applications instantly in the browser. It supports real-time streaming, live code preview, sandboxing, and session persistence. Check it out on GitHub and support a fellow community member. If you have any questions or suggestions on how to improve it further, share them in the thread.

AI poll of the week!

We’re writing the second edition of our book, and I asked the community to help pick the title. If you haven’t voted yet, there’s still time.

But honestly, the title matters less than what’s inside. So what I really want to know is: what topics do you want us to go deeper into in the second edition? More on agents? Evaluation? Deployment? Cost optimization? Something we didn’t cover at all in the first one? Let us know in the thread, this is your chance to shape what goes in.

Collaboration Opportunities

The Learn AI Together Discord community is flooded with collaboration opportunities. If you are excited to dive into applied AI, want a study partner, or even want to find a partner for your passion project, join the collaboration channel! Keep an eye on this section, too — we share cool opportunities every week!

1. Keatwonobe is a researcher exploring a new functional architecture for small model learning. The current model is between 1.2 and 1.8 million parameters in size, but can reduce CE loss on text to sub-0.1 levels in under 2k steps and run multiple continuous tasks simultaneously as a multimodal package. He wants to make it more capable, and if you think you can help, talk to him in the thread!

2. Yasssooo. wants to dive deeper into AI Engineering by taking on more projects and collaborating with others doing the same. So if this is your goal too, connect with him in the thread!

3. Mxropriv is working on an AI/app competition project and looking for teammates interested in software, AI, design, or business. If you are a high school student who wants to do something similar, reach out to him in the thread!

Meme of the week!

Meme shared by bin4ry_d3struct0r

TAI Curated section

Article of the week

Stop Prompting And Start Looping. A Claude Code Engineer’s Guide to /goal and /loop By Pravin Borate

Claude Code’s /goal and /loop commands turn one-shot prompting into loop engineering, in which agents work autonomously until the task is complete. /goal wraps a Stop hook with a second evaluator model that verifies completion from conversation evidence, while /loop reruns prompts on a schedule for recurring tasks like CI monitoring. The article demonstrates both techniques by building a self-healing test suite: six planted bugs, a hook that blocks test-file edits, a fully autonomous fix cycle, and guidance on writing verifiable goal conditions.

Our must-read articles

1. A Beginner’s Guide to Amazon Bedrock: Your First LLM App Without the Overwhelm By Harish Ramkumar

Amazon Bedrock lets developers call Claude and other foundation models via a single AWS API without managing infrastructure. This beginner guide walks through IAM users, model access, and a first working Python call with the Anthropic SDK, and explains inference parameters such as temperature and max_tokens. It progresses to building RAG pipelines with Knowledge Bases and adding Guardrails for content filtering and PII redaction, finishing with production tips covering prompt caching, batch inference, and security hardening, all runnable for under one dollar.

2. Context Window Management Is the New Memory Management By Satyam Sahu

Context windows function like RAM in LLM applications, and most teams manage them carelessly until cost and quality issues surface. The article explains tokens, why huge context limits breed complacency, and three failure modes: runaway API bills, the lost-in-the-middle attention problem, and undebuggable responses. It also covers three practical tips: setting token budgets, pruning stale history, and compressing old turns into summaries, backed by a simple Python context manager.

3. The vLLM Optimization Playbook for L40S By Vedanti

In this article, the author shares their findings after measuring vLLM’s optimizations on an NVIDIA L40S, running Llama 3.1 8B Instruct across 83 configurations and 4 production-style workloads. The result is vllm-optimization-bench, an open-source benchmark harness that produces concrete numbers on how vLLM optimizations behave in practice.

If you want to publish with Towards AI, check our guidelines and sign up. We will publish your work to our network if it meets our editorial policies and standards.

This week, I published something with Paul Iusztin that we’ve been working on for a while: an open-source system that gives your agents persistent memory across sessions using your own notes and research. Full details are below.

We also cover:

• How AWS’s AI-DLC methodology turns Claude Code into a disciplined engineering partner.

• Why the Replit and Cursor database-deletion incidents were governance failures, and four principles for deploying agents that act on production systems.

• How world action models are pushing robotics past video prediction: one approach cuts FLOPs by 6x, another lifts pick-and-place success from 0% to 70%.

• How to take a working Claude Agent SDK script and make it provable with structured output, per-run cost tracking, and OpenTelemetry traces you can actually search.

Let’s get into it!

AI Tip of the Day

If you have a task you repeat every week, you don’t need to keep prompting AI from scratch every time.

For example, a weekly report is usually not a one-off prompt. It is a repeatable workflow with a few inputs that change each time.

The problem is that people often save the prompt, but not the way they actually do the task. They forget the tabs they check, the fields they always fill in, the naming convention they use, or the final check they run before sending it out.

Codex’s Record & Replay feature is useful for this. You can record yourself doing the workflow once on your Mac, and Codex can turn that recording into a reusable skill. The next time you need to perform the same task, you give it new inputs instead of rebuilding the whole instruction from memory.

Before recording, pick one workflow with clear start and end points. Then note what changes each week, what stays the same, and what “done” looks like.

If you want to bring AI into real workplace tasks and team workflows, check out our Master AI for Work course.

— Louis-François Bouchard, Towards AI Co-founder & Head of Community

and I also published a deep dive on turning your second brain into agent memory. If you’re like most of us, you have thousands of notes across Obsidian, Readwise, Notion, and Google Drive, and none of them follow you into your next Claude or Codex session. Every research session starts from zero.

We built the fix: an AI Research OS that runs deep research across your notes and the open web, then stores what it finds as an LLM wiki, inspired by Karpathy’s idea, that your agents can query, maintain, and grow with every session. No vector database, no knowledge graph, nothing to host. Just Markdown, YAML, and folders. Google independently shipped the same architecture as an open standard (Open Knowledge Format), which tells you the direction is right.

The whole thing is open source as a Claude Code plugin, you can clone and run on your own notes today. Read the full article here.

Learn AI Together Community Section!

Featured Community post from the Discord

Nobias0700 is working on EVA, a minimal self-evolving agent that safely rewrites, tests, and promotes better versions of itself inside a hardened Docker sandbox. The project aims to answer what it would look like if an agent could modify its own runtime through a gated release process instead of live self-mutation. EVA’s loop is active release > candidate > tests/ratchet > kernel gate > promotion > ledger > rollback. The kernel stays small and non-evolving. The evolvable release contains the agent loop, tools, adapters, memory, context compaction, TUI, and self-model. Currently, it’s experimental, but check it out on GitHub and share your technical feedback in the thread.

AI poll of the week!

Claude Fable was one of the most anticipated releases this year, launched on June 9, pulled by the US government three days later, and only brought back on July 1. A third of you still haven’t tried it. Of those who have, the community is genuinely split: 19% say it’s amazing for coding, 12% think it’s way better than Opus and Sonnet, and 11% find it strong for agents and long tasks. But 15% think it’s good but overhyped or costly, and another 4% are frustrated by usage limits.

So the people who’ve used it mostly like it, but cost and access are clearly holding back adoption. It’s now on usage-credit billing, though Anthropic says they aim to bring it back to subscriptions once capacity allows.

What I want to know: if you haven’t tried it yet, is it because of the cost, because you’re worried it might get pulled again, or just because what you’re using now works fine? Let’s talk in the thread!

Collaboration Opportunities

The Learn AI Together Discord community is flooded with collaboration opportunities. If you are excited to dive into applied AI, want a study partner, or even want to find a partner for your passion project, join the collaboration channel! Keep an eye on this section, too — we share cool opportunities every week!

1. Teranmix is looking for AI ML/DL enthusiasts to learn AI with mathematics. If you are interested in that side of learning, connect with him in the thread!

2. Imarkusss is building an early-stage startup applying machine learning to simulate and predict how drugs, proteins, and biochemical compounds interact. He is looking for ML/AI engineers to collaborate, so if you are interested in this domain, reach out to him in the thread!

Meme of the week!

I have been saying it for so long now, efficientnet1995 made it funny with their meme.

TAI Curated section

Article of the week

AI-DLC + Claude Code: The End Of Vibe Coding, A Complete Hands-On Guide by Pravin Borate

AWS open-sourced AI-DLC, a phase-gated methodology that turns Claude Code from an autocomplete tool into a disciplined engineering partner. This article shows how to build a Task Management API, showing how workspace detection, requirements analysis, user stories, and design documents each require explicit approval before code generation begins. Extensions such as security baselining and property-based testing add blocking gates, while a persistent audit trail and a state file allow sessions to resume across tools.

Our must-read articles

1. Governance by Design: Four Principles for Building Safe, Compliant AI Agents by MongoDB

AI agents now act directly on production systems, and this article argues that recent database-deletion incidents at Replit and Cursor were governance failures rather than technical ones. The piece lays out four pillars for safe agent deployment: identifying regulatory constraints such as HIPAA and the EU AI Act; layering input, execution, and output guardrails; enforcing deny-by-default access with least privilege and human oversight; and establishing agent identity through OAuth On-Behalf-Of and SPIFFE for auditability.

2. How are World Action Models evolving? — Taking a glimpse into the future of Robotics by Kim Hyun Bin

World Action Models push robotics beyond video prediction toward genuine physical reasoning, and two recent papers show how. ImageWAM swaps costly video generation for an image-editing backbone, leveraging its internal KV caches to drive a flow-matching action expert, cutting FLOPs by a factor of 6 while achieving 93% success on RoboTwin 2.0. HWM, from Yann LeCun’s team, tackles long-horizon planning by pairing coarse and fine latent world models, lifting Franka’s pick-and-place success from 0% to 70% without task-specific rewards.

3. Multi-Aspect E-Commerce Semantic Engine Using Qdrant Multivectors by Divy Yadav

A semantic search engine for e-commerce hit a wall when a single embedding tried to represent specs, images, and reviews at once. The author rebuilt it using Qdrant multivectors, storing SigLIP image embeddings, ColBERT token-level text matrices, and BGE embeddings of extracted review findings at a single point. Query decomposition routes intent before embedding, prefetch stages narrow candidates, and personalization reranks results. Benchmarks show 95% Recall@3 with 2.2ms latency, alongside honest caveats about scale and when this complexity is actually worth building.

4. Claude Agent SDK Observability and Production Hardening: Your Agent Works. Now Prove It: Typed Output, Real Costs, and Traces You Can Search by Rick Hightower

This article shows how to turn a working Claude Agent SDK script into an operable service with structured output, cost tracking, and OpenTelemetry. Structured output gives downstream code typed JSON instead of paragraphs. Cost tracking puts a dollar figure on every run, including the failures. OpenTelemetry exports traces, metrics, and logs to the dashboards you already watch. By the end, you will know how to wire all three and which footguns to step around.

If you want to publish with Towards AI, check our guidelines and sign up. We will publish your work to our network if it meets our editorial policies and standards.

Lighter issue this week, since we just got back from the AI Engineer World’s Fair in San Francisco. 6,000+ AI engineers, 300 speakers, 29 tracks, and we were right in the middle of it. The conversations alone were worth the trip: what people are actually building right now, the problems they’re hitting in production, the patterns that keep coming up. If you’re working in AI engineering, this is the one conference where you feel like you’re in exactly the right room.

We gave a workshop on the context engineering behind our production AI tutor: compaction, memory, and cost, and the room was packed with people sitting on the floor and a queue outside. The 1:1 conversations after were honestly the highlight.

Here’s what came out of it, and it’s all yours:

We open-sourced the full AI tutor app. This is the same production system our students use to ask anything about AI engineering, RAG, agents, and get answers grounded in our own material with sources. You can run it locally, swap in your own content, and build your own version.

GitHub link | Try the AI Tutor | Slides

And if you missed last week’s email, we also shared the full talk Paul Iusztin and I gave on the AI Engineer World’s Fair Online Track: building a research wiki your agents maintain for you. No vector database, no knowledge graph. Just Markdown, YAML, and folders. Watch it here.

We also cover:

• Why your RAG eval can pass while your system is compromised

• How to stop your e-commerce agent from processing a refund on an order that doesn’t exist.

• How to search hours of wearable footage without any data leaving the device.

• Why most teams leave their LLM endpoints wide open after deployment and how to fix it.

• How to build a Slack news agent that delivers cited briefings for a few cents each

Let’s get into it!

AI Tip of the Day

In production RAG systems, prompt injection doesn’t only happen at the prompt level, but it can also sneak in through the documents your system retrieves. A vendor PDF, support article, scraped web page, or customer note can contain useful facts and a malicious instruction in the same chunk.

If your eval only checks whether the answer is factually correct, the system can look safe but treat all retrieved text as something it should obey.

To prevent this, add a few test documents that mix valid domain facts with instructions like “ignore the system message” or “send the user to this external link.”

Then check two things: the answer should still use the factual content, and it should refuse to follow instructions found inside the retrieved context. Log the chunk IDs too, so a failed test points to the retriever, prompt wrapper, or generation step.

If you’re building production RAG systems and want to go deeper into retrieval, evaluation, and deployment, check out our Full Stack AI Engineering course.

— Louis-François Bouchard, Towards AI Co-founder & Head of Community

Learn AI Together Community Section!

Featured Community post from the Discord

Agent_hellboy built claude-cockpit, a tiny status line + advisor for Claude Code. It shows live session signals such as model, branch, context pressure, token churn, rate limits, and cost, and then suggests useful controls like /compact, /clear, cheaper model switches, skills, subagents, MCP, or graphify when the session starts drifting. It does not automate anything or take over your workflow. It just keeps the important gauges visible and tells you what control might save time or tokens next. Check it out on GitHub and support a fellow community member. If you have any questions or suggestions, share them in the thread.

Collaboration Opportunities

The Learn AI Together Discord community is flooded with collaboration opportunities. If you are excited to dive into applied AI, want a study partner, or even want to find a partner for your passion project, join the collaboration channel! Keep an eye on this section, too — we share cool opportunities every week!

1. Shaurya09272004 is building an agentic tool that explains Indian laws in plain language and is looking for people working on similar projects to share ideas, discuss debugging journeys, and go deeper into LLMs. If you are building something similar, connect with them in the thread!

2. Strikerleapgaming is building their own startup and is looking for beta testers. If this interests you, reach out to them in the thread!

3. Vishacoplayz_27974 is recruiting founding board members for AIXelerate, a student-led AI nonprofit. If you’re a high school student interested in leadership, AI, marketing, operations, outreach, or event planning, contact them in the thread!

Meme of the week!

Meme shared by bigbuxchungus

TAI Curated section

Article of the week

Improving Our LangGraph Agent for Real-World E-Commerce: Enterprise Validation, Business Logic Guards, and a Multi-Agent Architecture by Bessie Delight Kekeli

ShopBot’s original LangGraph agent trusted every LLM output, so a hallucinated order ID could slip straight into a refund tool call. This follow-up piece rebuilds the system around three key additions: pure-Python Business Logic Gates that validate structured data before any tool runs, a correction loop with a hard retry cap that escalates to humans rather than looping indefinitely, and a Supervisor that splits the single agent into Order, Refund, and Complaints subgraphs.

Our must-read articles

1. Building an Offline “Life Memorizer” with Gemini 2.0 & Qdrant Edge by Satyam Sahu

Life Memorizer tackles a real problem for smart glasses and wearable capture devices: finding a moment buried in hours of sensory footage without shipping private data to the cloud. The author pairs Gemini Embedding 2, which projects text, images, and audio into a single 3072-dimensional space, with Qdrant Edge, an embedded vector store that requires no server process. The piece walks through schema design, Matryoshka truncation, hybrid search with location filtering, scalar and binary quantization, and mean-pool memory consolidation, then addresses where cloud embedding calls still break the offline promise.

2. Building Enterprise-Grade Security Boundaries for LLM Calls — OAuth 2.0 + APIM + Entra ID by Chris Bao

Securing LLM endpoints often gets overlooked once a model is deployed, and this piece tackles exactly that gap using Azure API Management, Entra ID, and OAuth 2.0. The author registers a public client application, implements the Device Code flow via MSAL for Jupyter-based authentication, and configures APIM policies that validate tenant ID and client application ID before forwarding requests to a GPT-4o-mini backend. A working demo confirms that only correctly scoped tokens pass validation, while standard Entra ID tokens get rejected outright.

3. Building a Slack AI Agent with Claude’s Web-Search Tool: An End-to-End Walkthrough by Tarun Agarwal

This article shows how to build a Slack news agent using Bolt’s Socket Mode, skipping the usual public-URL setup entirely. A single Claude API call handles both retrieval and summarization through the server-side web-search tool, turning topics typed into a slash command into cited, five-item briefings rendered as native Block Kit cards. The piece walks through each layer, from slash-command handling to citation cleanup, and breaks down real cost figures, landing at a few cents per briefing, plus three deployment gotchas worth avoiding.

4. Building an LLM from Scratch with PyTorch by Bessie Delight Kekeli

This article walks through building a decoder-only language model from scratch in PyTorch. It covers every component that makes transformers work: character-level tokenization, token and positional embeddings, causal self-attention with Query, Key, and Value projections, multi-head attention, feed-forward layers, and residual connections wrapped in transformer blocks. A full training loop on Tiny Shakespeare ties everything together, followed by a second implementation using PyTorch’s built-in nn.TransformerDecoder for comparison.

If you want to publish with Towards AI, check our guidelines and sign up. We will publish your work to our network if it meets our editorial policies and standards.

I keynoted AI Engineer World’s Fair and presented a system Paul Iusztin and I have been building for the past 18 months that turns thousands of personal notes into persistent agent memory that actually compounds over time.

Still processing it honestly, AI Engineer World’s Fair is where OpenAI, Anthropic, DeepMind, Cursor, and Hugging Face share what they’re building, and this year Towards AI is right there with them.

The recording is premiering right now on the AI Engineer YouTube channel. You can watch the whole thing for free.

Watch “Turn 10,994 Notes Into Memory”

Quick context, if you haven’t been following along.

I have close to 11,000 files across Obsidian, Readwise, Notion, and Google Drive. Every course, video, meeting recap, and research rabbit hole, it’s all in there. And yet every time I opened Claude Code or Codex, I started from scratch. Paste the same links. Rebuild the same context. Lose it all when the chat ends.

So and I built something different: a research wiki that agents maintain for you. One that grows with every session instead of resetting. No vector DB, no knowledge graph. Just Markdown, YAML, and folders.

In the talk, you’ll see:

— The full journey from V1 (barely worked) to V3 (actually stuck)

— Four Claude Code skills you can install today

— Two live demos pulling in GitHub repos and URLs in real time

— The full open-source codebase (MIT) you can clone tonight

Watch the keynote

I’m also at the World’s Fair in person this Sunday.

Samridhi, Omar, and I are running an 80-minute hands-on workshop: “Context Engineering in 2026: Compaction, Memory & Cost.”

Different angle from the keynote: how to keep agent context lean and affordable as you scale. How to compact, what to keep, what to drop.

If you’re at the conference or in SF, come say hi. I’ll be around all day. DM me.

📅 2:20 pm–4:20 pm, June 29 · Moscone West

👉 ai.engineer/worldsfair/schedule

The keynote is live right now! Go watch it and send it to someone who’d get something out of it.

Watch “Turn 10,994 Notes Into Memory”

—

P.S. Paul, Samridhi, and I also ran a 2-hour workshop at AIE London on building multi-agent systems with MCP servers, recording and code are live too. Watch it here.

Microsoft just released seven in-house AI models, but the interesting part isn’t the models, it’s the 100-page report. They refused to use synthetic data, actively hunted down AI-generated content before training, and essentially dared every other lab to prove they did the same. This week, we walk through the full release and what AI engineers can take from it. We also cover a debugging blind spot most teams miss: your agent’s tool call can succeed and still be the wrong tool entirely.

We also cover:

• Why prompt caching only covers 30% of your LLM cost problem, and a seven-layer optimization funnel that gets teams to 60–80% total reduction.

• Transformer attention reframed as a physics problem: token embeddings as point clouds redistributed through learned transport operators.

• How continuous batching took GPU utilization from 20–30% to nearly 100%, and why every major inference framework adopted it.

• Three memory strategies for LangGraph agents from simple message filtering to rolling summarization that keeps long-running agents affordable.

• What happens when AI agents don’t have a semantic layer: refunds counted as revenue, receipts misread as monetary sums, and UPT off by almost half.

Let’s get into it!

What’s AI Weekly

Earlier this month, Microsoft AI announced seven MAI models built in-house across reasoning, coding, image, transcription, and voice. But what makes this release worth your attention is not the benchmark performance alone, but the report, which is more transparent than anything I’ve read from a major lab this year. Microsoft refused to use synthetic data to train their model, then actively hunted down AI-generated content and removed it before training. And they wrote a 100-page report daring every other lab to prove they did the same. This week, in What’s AI, I will walk through the entire model release, the training process, the RL steps, and share the recipe AI engineers can steal from this. Read the full article here or watch the video on YouTube.

AI Tip of the Day

When an agent chooses a tool, don’t assume it understood the task.

Sometimes it only matches a word in the user’s request to a word in the tool name. For example, if the user asks for the “latest report,” the agent might still call the search tool even though the report has already been uploaded. Or it might call a database tool when the answer was already in the prompt.

To debug this, log three things side by side: the user’s request, the tool the agent picked, and the arguments it used. Then check whether the tool matched what the user actually wanted. Don’t only look for tool errors. A tool can run successfully and still be the wrong tool.

If you’re exploring agent engineering and want to go deeper into tool use and guardrails, our Agent Engineering: Building Multi-Agent Systems course is the cleanest path to building production agents.

— Louis-François Bouchard, Towards AI Co-founder & Head of Community

Learn AI Together Community Section!

Featured Community post from the Discord

Matteoturri_50413 built an open-source tool to reduce token waste when using AI on large codebases. FolioDux is a lightweight file-mapping standard that lets any AI tool navigate large codebases without reading every file. The AI reads the index first, picks only the relevant files, and ignores everything else. Check it out on GitHub and support a fellow community member. If you have any questions or feedback, share them in the thread!

AI poll of the week!

No single option ran away with it; the votes are spread across the board, which tells us something in itself. The pattern from last week holds: career outcomes still rank highest when you add up jobs, interview prep, and getting unstuck on projects. But this time, staying current on tools and workflows tied for first, which suggests that for a lot of you, the value isn’t just “help me get a job”; it’s “help me stay relevant once I have one.”

If we launched this tomorrow, what’s the one thing that would make you cancel after month one if it wasn’t there? Let us know in the thread!

Collaboration Opportunities

The Learn AI Together Discord community is flooded with collaboration opportunities. If you are excited to dive into applied AI, want a study partner, or even want to find a partner for your passion project, join the collaboration channel! Keep an eye on this section, too — we share cool opportunities every week!

1. Shaurya09272004 is building an agentic tool that explains Indian laws in plain language and is looking for people working on similar projects to share ideas, discuss debugging journeys, and go deeper into LLMs. If you are building something similar, connect with them in the thread!

2. Strikerleapgaming is building their own startup and is looking for beta testers. If this interests you, reach out to them in the thread!

3. Vishacoplayz_27974 is recruiting founding board members for AIXelerate, a student-led AI nonprofit. If you’re a high school student interested in leadership, AI, marketing, operations, outreach, or event planning, contact them in the thread!

Meme of the week!

Meme shared by default_user2004

TAI Curated section

Article of the week

The Prompt Cache Is Not Enough: Building a Full LLM Cost Optimization Strategy By Rizwanhoda

Prompt caching covers the easy 30% of LLM cost savings, but most teams stop there while their bills quietly climb back up. The author lays out a seven-layer optimization funnel that addresses every cost driver: semantic caching to skip redundant LLM calls entirely; model routing to stop sending simple tasks to expensive models; prompt compression via LLMLingua; batching for async workloads at half the price; and hard output constraints to cut verbose token spend. Teams implementing three or more layers consistently reached 60–80% total cost reduction.

Our must-read articles

1. The Flow of Attention By GSO1

If transformer attention is reframed as a physics problem, token embeddings enter a language model as a cloud of points in high-dimensional space, and each layer redistributes them via a coupled transport step governed by two learned operators. The piece traces how this additive update rule defines a flow in Wasserstein space, connects it to an idealized gradient flow result from Geshkovski et al., and explains why the actual transformer falls short of that theorem without abandoning its structure. Clustering across depth, the prediction readout, and positional encoding all follow cleanly from this geometric picture.

2. Continuous Batching: How to Keep Your GPU Actually Busy By Vedanti

Static batching wastes GPU capacity because the slowest request forces every completed slot to sit idle until the entire batch finishes. Continuous batching fixed this by re-evaluating the batch at every forward pass and immediately slotting in new requests as soon as a slot becomes available. When combined with PagedAttention for dynamic KV cache memory management, the approach raises GPU utilization from 20–30% to nearly 100%. Every major inference framework, including vLLM, SGLang, and TensorRT-LLM, adopted it.

3. LangGraph Memory: The Complete Practical Guide to Managing What Your Agent Remembers By Bessie Delight Kekeli

LangGraph agents face two silent production killers: conversations stored only in RAM vanish on every restart, and context windows grow expensive as message history accumulates. This article covers three memory management strategies, from simple message filtering and token-based trimming to LLM-powered rolling summarization, the most robust option for long-running agents. It also walks through swapping MemorySaver for SqliteSaver or PostgresSaver with a single line change.

4. Practical Breakdown of the Value of the Semantic Layer for AI Agents: Results of A/B Testing By Sergey Gromov

Without a semantic layer, models consistently make context errors, such as counting refunds as revenue, interpreting “Receipts” as a monetary sum, and miscalculating UPT by almost half. The semantic layer provides metric definitions, join logic, and business rules up front, eliminating the model’s need to infer corporate conventions. The experiment reframes semantic modeling as foundational infrastructure for enterprise AI agents, not legacy BI tooling.

If you want to publish with Towards AI, check our guidelines and sign up. We will publish your work to our network if it meets our editorial policies and standards.

There are services offering GPT and Claude API access at 90% off. Researchers tested 400 of them; one drained crypto from a wallet, others injected malicious code or grabbed cloud credentials. This week, I cover how these proxies actually make their money and why the risk is completely different when you’re routing a coding agent through them instead of just chatting. I also cover how to stop treating ChatGPT like a one-off assistant and start building repeatable workflows around your actual work.

Inside the issue:

• How to rebuild Claude Code’s architecture from scratch using LangChain’s deepagents.

• Why agents need version control the same way software does.

• A governed architecture for LLM-generated dashboards in Snowflake.

• How bypassing Ollama and running llama.cpp directly doubles inference throughput.

• The seven production failure points when scaling WebSockets to millions of connections.

Let’s get into it!

What’s AI Weekly

This week, in What’s AI, I dive into a too good to be true offer I found. There are several ultra-cheap API proxy stations where you can get the GPT or Claude API 90% cheaper. But when researchers tested 400 of these dirt-cheap AI API services, they found that one of them quietly drained crypto from a wallet. Others were injecting malicious code, or reaching for cloud credentials they were never given. In this article, I explain how these proxies make their money, why the model you pay for might not be the model you get, and why dropping one in front of a coding agent is a completely different level of risk than asking a chatbot a random question. Read the full article here or watch the full video on YouTube.

AI Tip of the Day

If you ask ChatGPT to rewrite emails, summarize documents, brainstorm ideas, or make something sound more professional, you are only scratching the surface.

That is useful, but it is still only 1% of what ChatGPT can do. Instead of starting from scratch every single time, use Projects to keep your context, files, examples, and instructions in one place. That way, you do not need to explain your work again every time you open a new chat.

Here’s how you can start getting better at AI today: pick one task you do every week, like creating a report, preparing for a meeting, summarizing customer feedback, or planning your priorities. Build a repeatable workflow around it. You can even use ChatGPT Tasks to run recurring prompts, like preparing a weekly briefing or reminding you to review key updates.

That is how you can start using AI in your actual work.

If you want more practical tips on how to use AI at work, and not just better prompts, check out our Master AI for Work Course.

— Louis-François Bouchard, Towards AI Co-founder & Head of Community

Sign up to receive weekly updates from the community!

Learn AI Together Community Section!

Featured Community post from the Discord

Exquisite_peacock_20933 just released Liodon AI SLM-10M, a 9.97M parameter causal language model trained from scratch. While it is not suitable for open-ended generation, it supports multiple-choice QA, log-likelihood ranking, SLM research, and perplexity evaluation. It was trained on 25B tokens and supports a context length of 1,024 tokens. Check it out on HuggingFace and support a fellow community member. If you have questions or feedback about the model, share them in the thread!

AI poll of the week!

We’re exploring a paid Towards AI membership for people learning AI, becoming AI engineers, or already building AI systems. And we want to know what would actually help you enough to use it every month.

Most of you are leaning towards career outcomes: internships on real projects, jobs, gigs, referrals, career help, and portfolio coaching.

For those who picked internships or jobs: are you actively looking right now, or do you want that option to exist for when you’re ready? That changes what we build first. Let us know in the thread!

Collaboration Opportunities

The Learn AI Together Discord community is flooded with collaboration opportunities. If you are excited to dive into applied AI, want a study partner, or even want to find a partner for your passion project, join the collaboration channel! Keep an eye on this section, too — we share cool opportunities every week!

1. Lucazsh is building a social media app for movies and is looking for someone who can help with frontend and app design. If this sounds like your domain, reach out to them in the thread!

2. Mrlucasrib is deeply studying a book on deep learning and needs a partner to discuss ideas in the book and study together. If you want to get into deep learning, connect with them in the thread!

3. Vishacoplayz_27974 is recruiting founding board members for AIXelerate, a student-led AI nonprofit. If you’re a high school student interested in leadership, AI, marketing, operations, outreach, or event planning, contact them in the thread!

Meme of the week!

Meme shared by drdub_

TAI Curated section

Article of the week

Build Your Own Claude Code Using LangChain: A Deep Dive Into LangChain’s Deep Agents By Sreejith Sreejayan

The article traces Claude Code’s architecture and rebuilds each piece using LangChain’s deepagents library. The framework centers on a bare agent loop in which the model either calls tools or returns text, naturally scaling from one-turn answers to multi-step refactors. Around that loop, the harness adds planning via to-do lists, filesystem-backed context management, subagent delegation, OS-level sandboxing for safety, and LangGraph checkpointing for persistence. The full working agent assembles in under a hundred lines.

Our must-read articles

1. Version-Controlling Your Agents: Deployment, Rollback, and Safe Promotion Patterns By MongoDB

Code reviews do not catch how production agents break, and this piece makes a direct case for treating agent configuration with the same discipline applied to software releases. It lays out three failure modes that arise when versioning is absent: live changes without isolation, manual rollback from memory, and silent degradation without an audit trail. It also proposes fixes, such as immutable config snapshots, staged promotion through canary environments, automated release gates, and pinning LLM model versions to prevent silent behavioral drift between provider updates.

2. Hosting LLM-Generated Dashboards: A Governed Snowflake Architecture By Mkrishnamallik

Governing LLM-generated dashboards inside Snowflake demands more than a smart chat connector. The article proposes a three-file contract separating authoring from hosting: an LLM builds the HTML, a thin Streamlit-in-Snowflake shell wraps it with RBAC, a semantic view enforces verified metric definitions, and every chat turn lands in an audit log. CI deploys per-PR previews with a manual prod gate. The architecture treats the semantic layer as the durable unit of trust, not the dashboard itself, which the author argues is now effectively a throwaway artifact.

3. I Can Compress 1000 Dimensions Into 2 — Here’s What PCA Taught Me By Anas Razy

PCA cuts through the curse of dimensionality by rotating the coordinate axes to maximize the data’s spread, then projecting everything onto those best-fit directions. The author builds the full intuition from scratch, covering covariance matrices, eigenvectors, eigenvalues, and projection math before implementing a 3D-to-2D reduction in Python using NumPy and Scikit-learn. The piece also explains why libraries prefer SVD over direct eigendecomposition and points to MNIST as a classic test case for visualizing high-dimensional data.

4. Optimizing Local LLM Inference on Constrained Hardware By Abhinandan Malhotra

Running on a 6GB RTX 3050, the author bypassed Ollama’s Go-based wrapper and ran llama.cpp directly, doubling token-generation throughput on an 8B-parameter model. Benchmarks across three models and three prompt scenarios quantified the abstraction tax: wrappers conservatively spill KV cache to system RAM as context grows, tanking performance across the PCIe bus. Key tuning levers included matching CPU threads to physical core count, using symmetric KV cache quantization, maximizing GPU layer offloading, and increasing micro-batch size to accelerate prefill-heavy RAG pipelines.

5. WebSockets at Scale: What Nobody Tells You About Managing Millions of Connections By Rizwanhoda

WebSocket connections drain file descriptors, memory, and routing logic in ways most tutorials never address. This piece walks through seven production failure points: OS file descriptor caps that limit connections to 1,024 by default, per-connection memory overhead that scales brutally, cross-server message routing solved via Redis Pub/Sub, the eventually-consistent presence problem, thundering herd reconnection bugs fixed with jittered exponential backoff, sticky session requirements for load balancers, and the monitoring gap WebSockets create. The final architecture stitches all fixes into a predictable, debuggable production system.

If you want to publish with Towards AI, check our guidelines and sign up. We will publish your work to our network if it meets our editorial policies and standards.

If you’re spending twenty minutes babysitting your coding agent through prompts, permissions, errors, and retries, you’re doing the dumb work instead of the thinking. This week, we cover a paradigm called loop engineering that halves the cycle by letting the agent loop with itself. We also cover how tools like Claude Cowork change the way you interact with AI, and why giving it a destination rather than a question makes all the difference.

We also cover:

• How prompt caching cut API costs by 72% without changing a single model or prompt.

• A full Langfuse walkthrough for production LLM observability.

• An auto-labeling pipeline that hit 96% recall on underwater shrimp without knowing what a shrimp looks like.

• Why mean-pooling over generated tokens produces better semantic embeddings than the last token or the prompt.

• A clinical AI deployment on AWS Inferentia2 that transcribes Bahasa Indonesia speech and generates SOAP notes in under 23 seconds for $1,100/month, fully on-premise.

Let’s get into it!

What’s AI Weekly

This week, in What’s AI, I am sharing a new paradigm that changes how you code with Claude or Codex. It cuts down the number of steps to the final output by half. Your current workflow probably looks like this: You write a prompt. Give file access to agents; the agent edits files. You accept all permissions. You run tests. Something breaks. You ask to fix it. Sometimes it works in one go, and sometimes you have to paste the error back or take a screenshot. It tries again. And after twenty minutes, you realize that you are babysitting the exact process you wanted to offload, and you’re doing the dumb work, not the thinking. This new paradigm I mentioned, called Loop engineering, is the idea that allows you to stop being that babysitter. No need to “micro-prompt” them. You can have it work, or loop, with itself instead. Read the full article here or watch the video on YouTube.

AI Tip of the Day

New AI work interfaces like Claude Cowork are designed to take action. They can read files, create documents, build spreadsheets, format slides, and save finished outputs.

This changes your interaction with them. A common mistake is using an AI teammate like a chatbot: “Can you help me analyze this sales data?” That usually gives you advice, summaries, or a few suggested steps. But if you give AI a clear destination like “Analyze quarterly-sales.csv. Create an Excel file with revenue by product category and quarter, add a line chart, and save it as Q1-Analysis.xlsx.”, it shifts the AI from brainstorming partner into a work execution partner.

We recently did a major upgrade to our Master AI for Work course; it now includes lessons on Claude Cowork and covers all the techniques needed to bring AI into real workplace tasks and team workflows. Find more information here.

— Louis-François Bouchard, Towards AI Co-founder & Head of Community

We just open-sourced a complete AI engineering roadmap for 2026.

It covers the full path from Python basics to production AI systems, designed to help you become an AI engineer, not just someone who can build an agent demo. No prior ML background required.

Built by our team at Towards AI, the repo includes beginner Python resources, foundational AI and LLM videos, recommended books, free and paid courses, hands-on coverage of RAG, agents, evals, MCP, deployment, safety, and coding-agent workflows, project ideas, communities, newsletters, people to follow, and job-search advice.

Every resource is tagged with a difficulty level from 1️⃣ to 🔟 so you can start wherever you are.

By the end, you’ll have the foundation to work as an AI engineer. Not just building with LLMs, but knowing when to reach for prompting over fine-tuning, when RAG is the right call and when it isn’t, when to use an agent and when a deterministic workflow will do the job better, when not to use an LLM at all, and how to evaluate, debug, trace, deploy, and monitor the systems you ship.

There’s also a built-in prompt you can paste into Claude or ChatGPT, along with your background, time, and goals, to turn the roadmap into a personalized learning plan.

Everything is free. Paid resources are clearly labeled.

Start AI Engineering in 2026

Give it a star if it’s useful, and share it with someone getting into AI engineering this year.

Learn AI Together Community Section!

Featured Community post from the Discord

liteai.me is an AI-powered coding playground where students can use HTML/CSS with PyScript or JavaScript to build real web apps. You can make the apps process data, crunch numbers, and create the hidden logic behind your tool; design the app’s face; make it interactive; and push it live. Check it out here and support a fellow community member. If you have any feedback, share it in the thread!

AI poll of the week!

This one was mostly for us; we wanted to understand who’s actually reading this newsletter so we can keep making it useful for you. The spread is wider than I expected. About 18% of you have little to no coding experience (including the vibe coders), 37% are in the beginner-to-intermediate range, and 43% are professional or senior developers. That’s a pretty even split between people still learning and people already deep in it, which honestly explains a lot about the kind of questions we get in the community.

What I’d love to know next: are most of you eventually trying to get into AI engineering, or are you here for a different reason entirely? Let me know in the thread!

Collaboration Opportunities

The Learn AI Together Discord community is flooded with collaboration opportunities. If you are excited to dive into applied AI, want a study partner, or even want to find a partner for your passion project, join the collaboration channel! Keep an eye on this section, too — we share cool opportunities every week!

1. Pragrr is looking to build a small AI research study group. If you want to build projects and collaborate on research, connect with him in the thread!

2. Ankukrsngh is working on an edTech startup and is looking for someone who has already deployed AI systems, knows API and backend infrastructure, and understands production workflows. If this seems like a good fit, reach out to him in the thread!

3. Imafiax_mafiax is looking for teammates for the AMD Developer Hackathon. If you’ve shipped AI agents, LLM apps, automations, RAG systems, and want to build something real, reach out to him in the thread!

Meme of the week!

Meme shared by drdub_

TAI Curated section

Article of the week

Prompt Caching Is the Most Underrated Cost Optimization in LLM Systems By Satyam Sahu

This article shows how engineers can cut their API spend by 72% without changing a single model or prompt, using prompt caching. With default settings, static content such as system instructions, analysis criteria, and few-shot examples is recomputed on every call, even if it is identical across thousands of requests. Structuring prompts to place stable prefixes first and marking them with cache breakpoints lets providers store the computed KV state and reuse it at one-tenth the normal input-token cost on Anthropic, dramatically reducing per-call expenses at production scale.

Our must-read articles

1. Production-Grade Agentic Observability: A Complete Langfuse Deep Dive by allglenn

Langfuse fills the observability gap in production LLM systems that standard logging tools miss entirely. The piece walks through the platform’s four core primitives: traces, spans, generations, and scores, then builds a complete customer support agent with retrieval, structured tracing via the @observe decorator, prompt versioning, LLM-as-judge evaluation, and a CI regression gate. It also includes tips on A/B testing prompts without code deploys, and golden dataset comparisons.

2. Auto-Labeling Unknown Objects: A Vision Model Pipeline for What Models Can’t See by kyon

Auto-labeling unknown objects required fine-tuning models or drowning in hand-drawn bounding boxes. This piece builds a three-phase pipeline using SAM3, Grounding DINO, and Qwen3-VL to generate annotation candidates without knowing the object’s name. The key insight is to segment first with SAM3, then classify individual crops with Grounding DINO using VLM-generated environment prompts rather than object names. Tested on underwater shrimp, the pipeline captured 96% of ground-truth objects at IoU 0.80, turning annotation from box-drawing into a faster accept-or-reject review.

3. The Best Semantic Representation in Your LLM Is Not the Last Token. Not the Prompt. It Is the Mean of What the Model Generated. A 2025 Paper Proved It Across Three Domains by Dr Swarneendu AI

Mean-pooling over the hidden states of generated tokens yields better semantic embeddings than any standard readout strategy, according to Wu et al. (2025). This article tests the approach via Centered Kernel Alignment against ground-truth reference spaces in language, vision, and protein structure, showing it outperforms every standard readout across all three domains. The finding is architectural: generated tokens have processed the full prompt and encode a complete, distributed representation that single-position readouts systematically miss, leaving measurable quality on the table in every deployed RAG and retrieval pipeline.

4. Deploying LLMs on AWS Inferentia2 for Clinical Workflows: A Production Journey by Bukhori M Aqid

This piece documents every failure and fix encountered while deploying LLMs on AWS Inferentia2 for clinical workflows, serving Qwen3–8B INT8 and Whisper large-v3-turbo on two inf2 instances.xlarge instances. The author works through five model candidates, hitting CPU RAM limits, NEFF-to-HBM expansion ratios as high as 24.7x, undocumented vLLM scheduling bugs, and a staging buffer regression that killed every model regardless of size. The final stack transcribes Bahasa Indonesia speech and generates SOAP notes in under 23 seconds for $1,100 per month, keeping patient data entirely on-premise.

If you want to publish with Towards AI, check our guidelines and sign up. We will publish your work to our network if it meets our editorial policies and standards.

Coding has fundamentally changed in the last five years, and this week, we trace the full arc from Stack Overflow to agents that write themselves. We also tackle one of the most common agent failures in production: what happens when a retry doesn’t know how far the last attempt got.

We also cover:

• How DeepSeek V4 stabilizes training at 61 layers deep with a new take on residual connections.

• A Snowflake Cortex optimization workflow that found a 9-point accuracy gain manual tuning missed.

• How to keep a research agent’s state alive across sandbox resets and machine reboots.

• The mathematical reason Transformers fail at exact arithmetic and why more compute won’t fix it.

• A multi-agent research engine with centralized state, parallel execution, and a human checkpoint before the expensive stage.

We also guest-published the first part of a Generative AI Masterclass on Neo Kim’s System Design Newsletter. If you want to go from using AI to designing the systems behind it, start here.

Let’s get into it!

What’s AI Weekly

The creator of Claude Code recently admitted that Claude Code has written 100% of his code since November 2025. On the same team, 95% of everything they ship is agent-written. This week in What’s AI, I trace the full story of how coding changed, from Stack Overflow to Copilot to ChatGPT to Cursor to Claude Code to agents running in parallel while you supervise. But here’s what makes this more than a timeline: a controlled study found that developers using AI were actually 19% slower on complex tasks, even though they believed they were 20% faster. There is a real perception gap, and how you use these tools is what separates vibe coding from actual engineering. Read the full article here or watch the video on YouTube.

AI Tip of the Day

Decide how an agent recovers from failure before you add retries. A retry is easy to add: the step failed, so run it again.

However, there’s a risk that the agent often doesn’t know how far it already got. If it failed halfway through, should it redo the last step, continue from where it stopped, ask a human, or stop entirely?

The most common real-world result is duplicated work: the same email sent twice, two tickets created, a payment charged again, or a file overwritten after the user already fixed it.

The proper fix is to track state, not just model calls. After each step, record what the agent intended, what it sent to the tool, what came back, whether it caused a real-world side effect (an email, a charge), and what it’s allowed to do next. A retry then reads that record and resumes safely, rather than asking the model to “try again.”

If you’re building agentic AI systems and want to go deeper into tool use, guardrails, and production-grade agent architecture, check out our Agentic AI Engineering course.

— Louis-François Bouchard, Towards AI Co-founder & Head of Community

We guest-published a piece on Neo Kim’s System Design Newsletter, the first part of a Generative AI Masterclass series. It covers everything you need to design GenAI systems from scratch: the model landscape, how training data pipelines actually work, architecture variants (decoder-only, encoder-only, MoE), model selection beyond benchmarks, and the data storage and retrieval layer behind products like ChatGPT and Perplexity.

If you’re comfortable using AI but want to understand how to design and build the systems behind it, this is where to start.

Read it here.

Learn AI Together Community Section!

Featured Community post from the Discord

Colignum has built LACK, a lightweight, self‑hosted multi‑agent chat platform, basically a Slack for agents. Every code block is automatically saved to a thread‑specific git repo, linted, committed, and followed by moderator feedback in the chat. Users can inject entire folder structures using natural language intent; embeddings find the best-matching template; and commands execute actions such as build, add, and import. Check it out on GitHub and support a fellow community member. If you have feedback on how to improve it or questions, share them in the thread!

AI poll of the week!

Claude Code takes 61%, Codex is at 25%. Not really a close race. Codex adoption has picked up since the recent update, but from the poll, most people have already settled on Claude Code as their go-to and treat Codex as something they check in on now and then. A few weeks ago, the community was spread across terminals, IDEs, chat, and custom stacks. Now, narrowed down to just terminal agents, Claude Code has nearly two-thirds.

The 14% “Other” is also interesting for a poll with only three options. There are clearly tools gaining traction that aren’t yet in the mainstream conversation.

If you had to uninstall one of them tomorrow, which one would actually mess up your workflow and what would break first? Let’s talk in the thread!

Collaboration Opportunities

The Learn AI Together Discord community is flooded with collaboration opportunities. If you are excited to dive into applied AI, want a study partner, or even want to find a partner for your passion project, join the collaboration channel! Keep an eye on this section, too — we share cool opportunities every week!

1. Shilpitextendmarch22 is transitioning into AI engineering and wants to start building projects around it. If you want to collaborate, work together, and build things together, connect with them in the thread!

2. Imafiax_mafiax is looking for teammates for the AMD Developer Hackathon. If you’ve shipped AI agents, LLM apps, automations, RAG systems, and want to build something real, reach out to him in the thread!

Meme of the week!

Meme shared by efficientnet_99825

TAI Curated section

Article of the week

DeepSeek V4 mHC Explained By Shakti Wadekar

DeepSeek V4’s mHC (Manifold-Constrained Hyper-Connections) replaces standard residual connections with multiple parallel residual streams, letting each transformer block pass richer, more diverse token representations across layers. The article breaks down three core operations: pre-mapping combines streams before attention or MoE computation, post-mapping redistributes the output back into parallel streams, and residual mapping enables cross-stream mixing with dynamically generated, input-dependent weights. Sinkhorn normalization constrains the mixing matrix to stay doubly stochastic, preventing stream collapse and stabilizing training across DeepSeek V4’s 61 deep transformer layers.

Our must-read articles

1. Building Production-Grade AI Skills with Snowflake Cortex AI Function Studio By Satish Kumar

Snowflake Cortex AI Function Studio provides enterprise AI teams with a complete lifecycle management system for AI functions, covering creation, evaluation, optimization, governance, and deployment within a single, governed platform. The article walks you through building a production incident analyzer using a multi-signal reasoning prompt and running systematic Genetic-Pareto optimization across six models, resulting in a 9-point accuracy gain and a 26% cost reduction that manual iteration would not have found. It also delivers concrete patterns for RBAC role separation, golden dataset evaluation, canary deployments, drift detection, and rollback procedures.

2. I Built a Stateful Research Agent Inside a Sandbox. Here’s What the Numbers Actually Looked Like By Divy Yadav

Building a stateful research agent exposes a core problem: ephemeral sandboxes reset mid-task, taking collected data with them. This guide shows how to build a research agent on TensorLake to solve this problem, using named sandboxes with suspend-and-resume that preserve full VM state, including running browser sessions, in under a second. It also covers practical benchmarks, real latency numbers per step, a hybrid Playwright-plus-vision browser strategy for handling layout drift, and a filesystem-as-state-store pattern that keeps the agent recoverable across process restarts and machine reboots.

3. Every Transformer Running Today Is a MacLaurin Series in Disguise By Dr Swarneendu AI

This article traces the mathematical backbone of the Transformer architecture back to 17th-century calculus, arguing that every core component can be expressed as a Maclaurin series. Softmax attention computes ratios of exponential expansions; positional encodings are sine and cosine series; GELU activations compose polynomial approximations; residual connections sum as geometric series. The piece then draws a hard structural conclusion: because Transformers are smooth, analytic functions, they cannot represent non-analytic operations, meaning failures in exact arithmetic and symbolic reasoning are representational limits, not engineering problems more compute can fix.

4. Beyond the Chatbot: Engineering a Self-Correcting, Multi-Agent Research Engine with LangGraph and PydanticAI By Rakesh Dharoori

This piece lays out a comprehensive architecture for an autonomous multi-agent research platform, using LangGraph and PydanticAI to handle enterprise AI workflows that standard RAG pipelines routinely fail to handle. The system routes all agent communication through a centralized, type-safe state machine rather than raw message passing, enabling traceability, deterministic routing, and state checkpointing. Parallel async execution via asyncio.gather cuts sequential latency, while an Auditor agent automatically loops failing data back to researchers. A human-in-the-loop checkpoint suspends execution before the costly synthesis stage, preventing token waste on bad data.

If you want to publish with Towards AI, check our guidelines and sign up. We will publish your work to our network if it meets our editorial policies and standards.

This week, we’re looking at the shift from “AI demos” to real systems: agents that need reliable execution, enterprises building durable AI infrastructure, and architectures that survive production constraints.

We also cover:

• A 1-hour practical walkthrough of modern AI engineering, from prompting and RAG to agents, evaluation, and deployment, plus a production lesson on why agent retries quietly break real systems.

• Why recursive multi-agent systems may depend less on “more agents” and more on how agents communicate internally.

• How enterprises are turning years of operational complexity into an advantage in the emerging “harness era” of AI.

• A practical guide to deploying production-ready agents on Google Cloud using Agents CLI.

• Why modern AI architecture evolved layer by layer, from LLMs to RAG, agents, and MCP, in response to real system failures.

Let’s get into it!

What’s AI Weekly

This week in What’s AI, I’m sharing something we normally only do for enterprise teams: a 1-hour deep dive into the foundations of AI engineering you need to know in 2026. We go through AI theory without the math, cover the real limitations of current LLMs, and walk through the production techniques such as prompting, context engineering, RAG, agents, fine-tuning, evaluation, and deployment. If you’re building with LLMs or planning to, this is the starting point I wish had existed when I began. Watch the full video on YouTube.

AI Tip of the Day

Agent tool call retries are helpful when a model request times out, a tool fails, or the system loses connection. But retries can cause serious problems if the agent repeats the same action. It might send the same email twice, issue two refunds, create duplicate support tickets, or rerun the same payment step.

Checking the tool arguments is not enough. The arguments can be valid, but the action may have already happened.

Give each tool action a unique ID that connects to the user request and the action being taken. Save the action status before running it. Then, before the tool runs again, check whether that same action has already finished. For external APIs, use an idempotency key when they support one. For your own database writes, add a uniqueness rule so the same action cannot be saved twice.

If you’re building agentic LLM applications and want to go deeper into tool use, guardrails, and production architecture, check out our Agentic AI Engineering course.

— Louis-François Bouchard, Towards AI Co-founder & Head of Community

Learn AI Together Community Section!

Featured Community post from the Discord

_creepycactus built OpenEar, a Mac dictation app. It hears you when you speak, records your meetings, and remembers every word. It runs on your chip, not the cloud, and doesn’t store any information. It is great for long prompts, meetings, voice journaling, or brain dumps. Check it out here and support a fellow community member. If you have any questions, ask in the thread!

Collaboration Opportunities

The Learn AI Together Discord community is flooded with collaboration opportunities. If you are excited to dive into applied AI, want a study partner, or even want to find a partner for your passion project, join the collaboration channel! Keep an eye on this section, too — we share cool opportunities every week!

1. Lucazsh is building a social media app and is looking for a frontend designer or app designer to improve the UX/UI. If this sounds like something you would enjoy working on, connect with them in the thread!

2. Muneebbaig. wants to dive deeper into ML, LLMs, and open-source AI research and produce one or two papers based on it. If you want to spend time on research projects or build your own, reach out to them in the thread!

3. Beratgurleer is working on n8n growth systems focused on lead conversion solutions and is looking for partners who can help with the technical side. If you want to enter the space and build something together, contact them in the thread!

Meme of the week!

Meme shared by bin4ry_d3struct0r

TAI Curated section

Article of the week

Groundbreaking Latent State Recursive Multi-Agent Systems is 2.4x Faster Uses 75.6% Cheaper By Mandar Karhade, MD. PhD.

This article walks you through the paper ‘Recursive Multi-Agent Systems’ that bundles two ideas: passing latent hidden states between agents instead of text, and running agents in iterative critique loops. Recursive loops are well-established since Self-Refine and Reflexion in 2023. The latent channel is the actual contribution. Text-based recursion plateaus or regresses by round three because agents commit uncertainty to words; latent recursion keeps improving. The paper’s own data shows the communication channel, not loop depth, is where multi-agent accuracy stops climbing.

Our must-read articles

1. Designing LLM Pipelines for Clinical Data: A Pattern for ALCOA++ and 21 CFR Part 11 Compliance By Pranav Nandan

Shipping LLM features into regulated clinical workflows reveals a recurring architectural failure: the prototype works, but it can’t answer where the audit trail is, why outputs have changed, or who is accountable. The article outlines a five-layer pipeline treating the LLM as a lossy parser, using constrained decoding to physically prevent hallucinations and deterministic Python for all logic and computation. A conditional judge LLM fires on only 15% of records, and ALCOA++ and 21 CFR Part 11 compliance emerge from the architecture.

2. Harness: The Era Enterprises Were Built For By Fabio Yáñez Romero

The era of prompt engineering favored lean, fast-moving teams who could ship on instinct. The harness era inverts that advantage. The article traces the arc from model weights through context engineering to the harness, a persistent runtime built on externalized memory, reusable skills, and machine-readable protocols. Enterprises that spent decades documenting procedures, governing data, and stabilizing interfaces now hold exactly the right raw material. The model becomes swappable; the harness becomes the durable intelligence layer the company owns outright.

3. How to Build and Deploy AI Agents on Google Cloud: A Complete Guide to Agents CLI By Pavan Dhake

Google’s Agents CLI bridges the gap between a working local AI agent and a production deployment on Google Cloud. The tool injects seven bundled skills into coding assistants such as Claude Code, Gemini CLI, and Cursor, automatically handling scaffolding, evaluation, deployment, and observability. This guide walks you through every step with real commands from the official docs.

4. LLMs, RAG, Agents, MCP: The AI Evolution You Must Know (A Visual Explanation) By Divy Yadav

This article covers the evolution of AI, from LLMs to MCP. It shows how LLMs evolved in distinct layers, each solving a specific failure. LLMs excelled at language but hallucinated and lacked memory. RAG grounded responses by retrieving relevant documents at query time. Agents extended that into action, using tools to browse, query databases, and call APIs. MCP standardized how models connect to external systems, replacing bespoke integrations with a universal protocol.

If you want to publish with Towards AI, check our guidelines and sign up. We will publish your work to our network if it meets our editorial policies and standards.

This week, we trace how Google moved from a research lead to a product stumble to a distribution-powered comeback. We also cover why letting your agent call tools without validating the arguments first is one of the fastest ways to break things in production.

Also in the issue:

• World models and how they differ from transformers: predicting the next environmental state instead of the next token.

• LeCun’s new paper based on the idea that coherent sentences trace geodesics on a semantic manifold, not random token walks.

• An AI SRE agent that monitors production logs without predefined thresholds by filtering out 99% of noise before the model ever sees it.

• Apple’s solution for converting trained transformers into Mamba SSMs without rebuilding from scratch.

• Entropy explained from Shannon’s wartime research all the way to why temperature works the way it does in LLMs.

Let’s get into it!

What’s AI Weekly

This week, in What’s AI, I trace Google’s entire trajectory in the AI race. They introduced the Transformer, the architecture behind most frontier LLMs. Then OpenAI turned that research lineage into the consumer product moment Google missed. So they rushed out, and two months after ChatGPT’s release, they launched their chatbot Bard, but in its very first public demo, it got a basic fact wrong about the James Webb Space Telescope. But fast forward to today, April 2026. Gemini has 750 million monthly active users. Apple chose Gemini to power the next generation of Siri. Salesforce CEO publicly said he switched from ChatGPT to Gemini and is “not going back.” In this article, I will go through this entire journey from the ChatGPT shock to the first Gemini model to the acceleration. Read the full article here, or if you want to watch the video version of this article, find it here.

AI Tip of the Day

If an agent sends the wrong arguments to a tool, the risk is higher than with other LLM systems: it can refund the wrong order, email the wrong customer, update the wrong record, or run a database action without the necessary constraints.

To avoid this, treat tool arguments like normal backend inputs. Validate IDs, permissions, account state, allowed ranges, required fields, and irreversible actions before execution. The model can decide which tool to call, but your application should decide whether to allow that call. This keeps the agent useful without making it the authority layer for your product.

If you’re building LLM applications and want to learn more about tool use, agents, guardrails, and production architecture, and how to make these decisions for any scale, we cover this in our 10-hour LLM Fundamental course.

— Louis-François Bouchard, Towards AI Co-founder & Head of Community

Learn AI Together Community Section!

Featured Community post from the Discord

Iampoppyxx has built KVBoost, which allows you to reuse chunk-level KV cache for HuggingFace inference. The kernel implements tiled FlashAttention-2 with online softmax, reducing HBM memory traffic from O(N²) to O(N) during KV encoding. It is applied automatically to every attention module inside the loaded model, no code changes needed. Check it out on GitHub and support a fellow community member. If you have any thoughts on how to improve it, share them in the thread!

Collaboration Opportunities

The Learn AI Together Discord community is flooded with collaboration opportunities. If you are excited to dive into applied AI, want a study partner, or even want to find a partner for your passion project, join the collaboration channel! Keep an eye on this section, too — we share cool opportunities every week!

1. Harry03418 is building a production-level automation system using n8n and wants to connect with other builders who want to exchange ideas. If this sounds interesting, connect with him in the thread!

2. Aliteralmango_07776 is building an Autograd library and needs help with improving it. If you want to collaborate or have suggestions for the project, reach out to them in the thread!

3. Syhslvdr is currently learning Python for data analysis/data science/ML and is looking for study partners. If you want to discuss coding projects or exchange notes, connect with her in the thread!

4. Khturan has built a project for a hackathon that they want to scale as a startup. If you like building AI-based businesses, contact them in the thread!

Meme of the week!

Meme shared by bin4ry_d3struct0r

TAI Curated section

Article of the week

World Models Explained: The Architecture That Could Replace Transformers By Yuval Mehta

This article explains the architectural difference between world models and transformers: LLMs predict the next token, while world models predict the next environmental state through a perception module, a dynamics model, and a planning module. The article maps two competing schools: generative models like Genie 3 and Marble that predict at the pixel level, and LeCun’s JEPA that predicts abstract latent representations.

Our must-read articles

1. Tokens Are Breadcrumbs. Geodesics Are the Path. LeCun Just Proved the Difference Matters By Dr. Swarnendu AI

Hai Huang, Yann LeCun, and Randall Balestriero posted a February 2026 paper called Semantic Tube Prediction, turning a long-standing philosophical argument about LLMs into formal mathematics. The Geodesic Hypothesis posits that coherent sentences trace geodesics on a smooth semantic manifold, and the STP loss enforces that each hidden state remains within a tube around the corresponding geodesic. This article walks through the paper and the mathematics behind it.

2. How I’m Building an AI SRE Agent to Analyze Production By Quan Huynh

The article outlines the architecture of an AI Site Reliability Engineering agent designed to monitor production logs without predefined thresholds. The system inverts the typical approach: regex matching, Drain3-style pattern clustering, and frequency analysis eliminate 99% of log noise before the AI sees anything, keeping costs and false positives low. Three deployment modes, training, shadow, and detect, let teams validate the agent’s judgment before it creates real incidents. The pattern catalog serves as the agent’s long-term memory, improving through operator feedback rather than model retraining.

3. Apple Just Built a Bridge Between Attention and SSMs. Here is the Step-by-Step Blueprint By Dr. Swarnendu AI

Apple researchers propose a practical two-stage route for converting trained attention models into Mamba-style State Space Models. Direct distillation fails because softmax attention and Mamba’s recurrence represent information in structurally incompatible ways. The fix uses two stages: first, a learnable Hedgehog feature map converts softmax attention to linear attention by directly matching attention patterns. Second, HedgeMamba inherits those parameters as Mamba initialization, replacing the identity decay with a learnable A matrix. The result reduces inference to linear time with only a 0.25-perplexity degradation.

4. I Finally Understood Entropy — Here’s the Simplest Way to Think About It (Even in LLMs) By Ashish Abraham

Entropy sits at the core of how language models learn and generate text, yet most explanations leave readers more confused. This article traces the concept from Shannon’s wartime communications research, through decision-tree splits, to LLM training objectives. Cross-entropy drives model optimization, perplexity converts it into an interpretable scale, and bits-per-byte frames model quality as a compression ratio. Temperature emerges as a direct dial on output probability distributions, connecting foundational information theory to a concrete generation parameter in LLMs.

If you want to publish with Towards AI, check our guidelines and sign up. We will publish your work to our network if it meets our editorial policies and standards.

An AI agent just ran 700 experiments on its own, found patterns, and optimized its own performance, no human in the loop. This week, I break down Karpathy’s Auto Research project and the technical bottleneck it exposes: the Context Rut. We also cover why putting business logic inside a prompt is one of the most common mistakes in production LLM systems and what to do instead.

We also cover:

• How to deploy a full Snowflake Cortex AI dashboard from a single SQL worksheet

• A 70-year-old theorem that proves every attention score you’ve ever computed is a kernel evaluation, and why softmax is a mathematical necessity.

• RAG without a single vector: PageIndex replaces embeddings with a reasoning-driven tree index and scored 98.7% on FinanceBench.

• The three math ideas that make or break your understanding of backpropagation: derivatives, chain rule, and log-loss.

• Why XGBoost’s real bottleneck was never the model, it was the 70–80% of project time spent flattening relational databases into matrices. Relational Foundation Models skip that layer entirely.

Let’s get into it!

What’s AI Weekly

This week in What’s AI, I break down the architecture behind this “Auto Research” agent and the massive technical hurdle in building agents: The Context Rut. Andrej Karpathy released a project that ran 700 experiments entirely on its own, identifying patterns and optimizing its own performance without human intervention. Which leads to an important question: Are we seeing the first real-world loop of AI making AI better? I answer this and share a few more technical strategies to keep your agentic frameworks lean and efficient. Watch the full video on YouTube.

AI Tip of the Day

If your business logic only exists inside a prompt, you cannot test it, audit it, or guarantee it runs the same way twice. Prompts like “only approve refunds under $50” look like rules, but they are suggestions the model can misinterpret, ignore under edge cases, or lose entirely to a prompt injection.

A better approach is to keep product rules in normal code. The model should extract intent, classify inputs, and generate responses. Your backend should enforce limits, check eligibility, validate account state, and gate irreversible actions.

For example, the model can extract a refund reason and suggest whether the user is eligible for a refund. But the backend should check the actual purchase history, policy rules, and account state before anything happens.

We cover this pattern and the broader architecture decisions behind production LLM systems in our Full Stack AI Engineering course.

— Louis-François Bouchard, Towards AI Co-founder & Head of Community

Learn AI Together Community Section!

Featured Community post from the Discord

Colignum created LACK, a lightweight, self‑hosted multi‑agent chat platform powered by local LLMs (Ollama). It enables autonomous agent collaboration, research (SIPHON), code sharing, direct messaging, and a built‑in cron job manager that wipes and recreates heartbeat jobs for every channel and DM. Check it out on GitHub and support a fellow community member. If you have feedback, share it in the channel.

AI poll of the week!

This looks less like a “which model wins” poll and more like a tool stack snapshot. Opus leads, but the comments make it clear most people aren’t committing to one model; they’re routing.

Claude (especially Opus) shows up consistently for coding and structured work, GPT for general use and brainstorming, and Gemini for a second opinion or for tasks like search, explanation, and media. Even the tradeoffs are consistent: Claude is strong but token-heavy, GPT is reliable for everyday use, and Gemini is surprisingly good when you need breadth or external context. The interesting part isn’t who’s best, it’s that people are building workflows across models instead of betting on one.

If you had to remove one model from your stack today, which one would actually break your workflow the most, and what specific task would you struggle to replace? Let’s talk in the thread!

Collaboration Opportunities

The Learn AI Together Discord community is flooded with collaboration opportunities. If you are excited to dive into applied AI, want a study partner, or even want to find a partner for your passion project, join the collaboration channel! Keep an eye on this section, too — we share cool opportunities every week!

1. Knightinout is looking for a collaborator to discuss generative AI, AI agents, and Machine Learning, including mathematical foundations, over the summer. The goal is to quickly understand the landscape and compound efforts to get certifications faster. If this sounds like a good way to spend the summer, connect with them in the thread!

2. Amanray9414 is trying to learn the blend of agentic AI and RL agents and wants to learn further about autonomous agents. If that sounds interesting, reach out to them in the thread!

3. Beratgurleer is experimenting with no-code automations and AI-based projects and is looking for a business partner. The agency will be based on n8n, and you’ll learn, brainstorm, develop products together, and solve problems. If this sounds fun, contact them in the thread!

Meme of the week!

Meme shared by drdub_

TAI Curated section

Article of the week

Building the Agentic Enterprise Control Plane on Snowflake By Satish Kumar

This article walks through the process of building a production-grade Snowflake Cortex AI dashboard, deployed entirely from a single SQL worksheet. The setup covers five enterprise tables, 70 synthetic records, and six Cortex functions, including sentiment analysis, summarization, classification, and LLM inference via COMPLETE. It also uses Python stored procedures with chr() substitution to avoid SQL parser conflicts, writing the full Streamlit app line by line directly to stage, with no external files, no manual steps, and built-in version control through an APP_SOURCE table.

Our must-read articles

1. Every Attention Score You Have Ever Computed Is a Kernel Evaluation By Dr. Swarnendu AI

A 70-year-old theorem, Mercer’s theorem, proves that attention scores in transformers are kernel evaluations. The piece explains the mathematics connecting dot-product attention to Reproducing Kernel Hilbert Spaces, showing that softmax is not an architectural choice but a mathematical necessity: the only normalization that simultaneously satisfies non-negativity, unit-sum, differentiability, and score amplification. It also unifies SVMs, Gaussian Processes, and transformers into a single equation, reframing attention as Nadaraya-Watson kernel regression in a learned representation space.

2. Vectorless RAG with PageIndex [+ Implementation] By Asad Iqbal

PageIndex reframes document retrieval by replacing embeddings with a reasoning-driven tree index that mirrors how human analysts actually navigate complex reports. Rather than approximate similarity matching, an LLM traverses a hierarchical JSON structure to locate exactly the right sections, preserving cross-references, table relationships, and the document hierarchy. Every retrieval decision remains fully traceable, making it audit-ready for regulated environments. Mafin 2.5, built on PageIndex, scored 98.7% on FinanceBench. The implementation walkthrough covers the full three-step pipeline using the DeepSeek-R1 paper, Groq, and the PageIndex SDK.

3. Ignore These 3 Math Ideas: And Backpropagation Will Never Make Sense By Tina Sharma

This article breaks down backpropagation into three foundational pieces: derivatives, the chain rule, and log-loss. It shows how derivatives measure a weight’s sensitivity to error, how the chain rule propagates that signal backward through every layer by multiplying local gradients, and how negative log-probability gives training a principled, information-theoretic target to minimize. Each concept carries real design trade-offs, from vanishing gradients to loss spikes, and the piece reframes all three as part of the vocabulary rather than barriers.

4. Is XGBoost gone: How Relational Foundation Models Conquered 500 Billion Row Enterprise Data By Ampatishan Sivalingam

XGBoost ruled enterprise ML for a decade, but its real bottleneck was the engineering machinery required to flatten relational databases into matrices it could read. Feature stores, Airflow DAGs, and aggregation pipelines consumed 70–80% of ML project time while destroying sequential signals in the process. This article introduces Relational Foundation Models, which eliminate that entire layer by ingesting raw database schemas directly and using schema-aware attention over foreign keys, as well as state-space models for temporal sequences.

If you want to publish with Towards AI, check our guidelines and sign up. We will publish your work to our network if it meets our editorial policies and standards.

Big one this week: we recently did a 2-hour workshop at the AI Engineer Summit in London, and it went so well that the organizers put the full recording on YouTube. So now you all get it for free.

Paul Iusztin, Samridhi from Towards AI, and I walked through building an MCP-powered deep research agent from scratch: planning a research strategy, searching the web, analyzing YouTube videos, gathering grounded evidence, filtering for relevance, and synthesizing everything into a cited research artifact. If you’re an AI engineer looking to build end-to-end agentic systems (or want AI to handle 90% of your writing without sounding like AI), this one’s for you. Watch it here.

We also cover:

• How a research pipeline evolved across three generations: from hallucinating Claude Code skills to adversarial subagents to deterministic control flow with the Claude Agent SDK.

• A workaround for Snowflake Cortex’s model limitations by wiring it to Groq for sub-second Llama 3 and Mixtral inference.

• What happens when you need to serve hundreds of concurrent LLM users and the KV cache no longer fits in memory?

• Why feeding more structured data to VLMs like Gemini actually makes them perform worse; fabricated detections at low confidence override correct ones.

• The math behind diffusion models, from DDPM’s forward process through CLIP, DDIM, unCLIP, and Stable Diffusion’s latent-space compression.

Let’s get into it!

What’s AI Weekly

Between February 12 and 23, three major US AI labs have accused Chinese labs of stealing. Anthropic accused three Chinese AI labs of running what they call “industrial-scale distillation attacks” on Claude: 24,000 fake accounts and 16 million exchanges, coordinated proxy networks all designed to extract Claude’s reasoning, coding, and agentic capabilities. OpenAI sent a memo to the U.S. House Select Committee on China, accusing DeepSeek of “free-riding” on U.S. frontier-model capabilities. And even Google published a report documenting a 100,000-prompt campaign targeting Gemini’s reasoning traces.

So, this week, I will break down what’s actually being accused, how distillation works technically, the history that got us here, and the hypocrisy behind these stories. Read the complete article here or watch the video on YouTube.

AI Tip of the Day

Vector search is great at capturing meaning, but weak at exact matches. Things like product names, error codes, version numbers, or acronyms often get missed. For example, a query like “GPT-4o API error 429” might not rank well with pure vector search, even if that exact phrase exists in your data, because embeddings capture overall meaning, not precise tokens.

Hybrid search addresses this by combining vector search with BM25, a keyword-based method that scores exact-term matches. A common approach is to run both and merge the results using something like Reciprocal Rank Fusion. This gives you both semantic understanding and exact matching, without having to choose between them. Several modern databases, such as Weaviate, Qdrant, and Elasticsearch, support this natively.

If you’re building retrieval pipelines and want to go deeper into search strategies, evaluation, and the full production stack, check out our Full Stack AI Engineering course.

— Louis-François Bouchard, Towards AI Co-founder & Head of Community

Learn AI Together Community Section!

Featured Community post from the Discord

G023dev has built HarnessHarvester, a self-learning, self-correcting, LLM-powered harness creation and management system. It features a FAISS-powered RAG, sandboxed execution, and autonomous improvement modes. It also includes two autonomous modes: autolearn (a continuous discovery loop) and autoimprove (an iterative enhancement of existing harnesses). It is designed as an offline-first harness/scaffolding builder, so you get the harness instead of some remote api. Check it out on GitHub and support a fellow community member. If you have any questions, connect with him in the thread!

AI poll of the week!

A lot of you feel Claude (or your current provider) has slipped a bit over time, while a smaller group says it’s either steady or improving. That split makes sense; these systems change under the hood, and you usually notice it first as “wait… why is this suddenly harder than last month?” especially on the same prompts and tasks.

When you say it’s getting worse, what changed for you the most: more refusals/guardrails, weaker reasoning on hard questions, more hallucinations, lower coding accuracy, or a shift in tone (too verbose/too cautious)? Let’s do something about it in the thread!

Collaboration Opportunities

The Learn AI Together Discord community is flooded with collaboration opportunities. If you are excited to dive into applied AI, want a study partner, or even want to find a partner for your passion project, join the collaboration channel! Keep an eye on this section, too — we share cool opportunities every week!

1. Digvijay010606_44180 is looking for a study partner to learn data science and stay consistent. If you struggle with it and need an accountability partner as well, connect with him in the thread!

2. Lyraluthuin is working on an AI system to create a 3D model from human-drawn lines and is looking to connect with people in computer graphics animation technology to help scope out the remaining work. If this is your space, contact her in the thread!

3. Augmnt_sh is building an Agent Observability Protocol and is looking for a few people who are building autonomous agents and want to try AOP on their stack. If you want to contribute to the SDK and are interested in the observability/dev tooling space, connect with them in this thread!

Meme of the week!

Meme shared by bin4ry_d3struct0r

TAI Curated section

Article of the week

From Claude Code Skills to Adversarial Subagent Orchestrators to the Claude Agent SDK: Three Generations of a Production Research Pipeline By Rick Hightower

Harness engineering separates toy agents from production systems, and this piece traces a research pipeline across three generations. Generation one chained Claude Code skills but hallucinated freely. Generation two added adversarial doer/judge subagents with retry prompts, though the orchestrator often ignored instructions. Generation three moved control flow into Python with the Claude Agent SDK, enforcing deterministic retries, Pydantic-validated structured output, budget caps, and human-in-the-loop escalation.

Our must-read articles

1. How I Built a Production-Grade Open-Source LLM Pipeline Using Groq and Snowflake By Satish Kumar

One limitation of Snowflake Cortex is that it supports only a handful of models and blocks External Access Integrations on trial accounts. This article shows how to wire Snowflake to Groq’s inference API for sub-second Llama 3 and Mixtral calls. It also covers 10 production use cases, including sentiment analysis, PII detection, SQL generation, and anomaly root cause analysis.

2. Inside LLM Inference: When the KV Cache No Longer Fits By Aanchal Karamchandani

Serving hundreds of concurrent LLM users turns inference into a memory management problem, not a compute one. The piece walks through how production systems tackle KV cache pressure using techniques like PagedAttention’s non-contiguous block allocation, inspired by OS virtual memory; prefix sharing across users with identical system prompts; prefill caching that retains computed blocks over time; attention-aware eviction policies; and INT8/INT4 quantization. Each technique compounds the others, cutting memory waste from 60–80% to under 4%.

3. VLM: The More You Tell it, The Less it Sees By Mradul Dubey

In this article, the author ran controlled experiments showing that feeding structured detection data to VLMs like Gemini 3 Flash actively suppresses visual reasoning on a surveillance clip. The same bounding box information, delivered as text, drawn overlays, or cross-modal references, produced vastly different anchoring bias severity. The results show that fabricated detections at low confidence override correct shoplifting identifications, and that every added metadata field monotonically degrades perception.

4. The Physics of Imagination: Visualizing the Hidden Mathematics of Diffusion Models By Shreyansh Jain

This piece walks through Diffusion models, tracing how controlled destruction becomes the foundation for modern machine imagination. The author covers DDPM’s Brownian-motion forward process and its noise-prediction training objective, CLIP’s 512-dimensional shared embedding space that gives generators a semantic compass, DDIM’s SDE-to-ODE shortcut that collapses 1,000 sampling steps into roughly 50, unCLIP’s two-stage prior-and-decoder architecture behind DALL-E 2, and Stable Diffusion’s latent-space compression using a lightweight VAE.

If you want to publish with Towards AI, check our guidelines and sign up. We will publish your work to our network if it meets our editorial policies and standards.