Here are the features that actually made me reconsider past decisions, and where I’d still think twice.

S3 Vectors — rethinking the vector store for the right use case

Every RAG system I’ve built had the same component: a separate vector store. OpenSearch, Pinecone, pgvector — each with its own ops overhead and cost model.

S3 Vectors went generally available at re:Invent 2025, letting you store and query vector embeddings directly in S3, at significantly lower cost than dedicated vector databases. The architectural impact though is hitting projects now.

For projects where retrieval latency isn’t a hard constraint and the dataset isn’t updated in real time, this simplifies the architecture meaningfully. One less service to operate, and your embeddings live where your source documents already live.

⚠️ Reality check: If your use case requires frequent embedding updates or strict p99 latency SLAs, a dedicated vector store still wins. S3 Vectors makes most sense for stable datasets with mode rate query load — think document archives, not live feeds.

Bedrock Managed Knowledge Base — good for early stages, with caveats

Data ingestion pipelines are often the unglamorous bottleneck in RAG projects: chunking strategies, format normalization, inconsistent document quality. For first project stages and MVPs, the new Bedrock Managed Knowledge Base can eliminate weeks of that work.

It provides native data connectors, Smart Parsing for automatic multi-format preparation, and an Agentic Retriever for multi-step queries.

The Agentic Retriever is architecturally interesting — it can decompose a complex question and retrieve across multiple steps rather than a single lookup. That’s a meaningfully different capability.

⚠️ Reality check: Managed means less control. If your data has unusual structure, domain-specific chunking requirements, or your retrieval strategy is where you differentiate — custom pipelines still make sense. I’d use this to move fast in early stages, then evaluate whether the limitations matter before scaling.

OpenSearch Serverless — autoscaling

OpenSearch Serverless was rebuilt from the ground up recently, with instant autoscaling and up to 60% cost savings for dynamic workloads.

The original version had a known frustration: you paid for provisioned capacity even when traffic was low, which made it expensive for workloads that weren’t constant. The rebuild changes that calculus for search-heavy AI applications with unpredictable traffic patterns — think internal knowledge bases or tools used by specific teams at specific times.

⚠️ Reality check: “Up to 60% savings” depends heavily on your traffic pattern. For steady, predictable load, provisioned OpenSearch may still be cheaper. Worth running numbers on your actual usage profile before migrating.

Thank you for reading, let’s chat 💬

💬 Have you tried S3 Vectors in any of your projects yet?
💬 Which of these would change your architecture the most?
💬 And what AWS features are you still waiting for that would actually make a difference?

I love hearing from readers 🫶🏻 Please feel free to drop comments, questions, and opinions below👇🏻

Leave a comment

API design is one of the hardest things to change after launch. Your clients are already integrated, your contracts are set, and your state model is baked into production. This is not like refactoring a function.

These 4 decisions are the ones you need to make deliberately, before you write the first endpoint. Think of this as a pre-flight checklist — not a tutorial, but a map of what to consider and why the order matters.

TL;DR

• Stateless API is the right default for anything that needs to scale — once you're stateless, workflow length determines whether you need streaming, async jobs, or both.

• Streaming vs async jobs is a response delivery decision — stream when a user is watching, use async jobs when they’re not or when the workflow runs too long

• Rate limiting for AI APIs means tokens, not just requests — RPM without TPM is incomplete

• Pydantic (or Zod in TypeScript) serves three contracts: incoming requests, tool definitions, and structured model output

Why these decisions are not independent

Stateless API is the foundational decision — it cascades into everything else. Once you commit to stateless design, your next question is workflow length: is a user watching and waiting, or is the work happening in the background? That single question determines whether streaming is enough or async jobs are required. From there, two more decisions apply regardless of which path you took — rate limiting and schema contracts. Get the foundation wrong and the rest won't save you.

1️⃣ Stateless API

Not every AI API needs to be stateless. A local tool, a single-user assistant, or a quick prototype can hold conversation history in memory and work perfectly well. The trade-off becomes visible the moment you scale — multiple instances, concurrent users, pod restarts.

For any AI API that needs to scale beyond a single instance, stateless is the right default. Store conversation history in Redis or Postgres, pass a thread_id with every request, and let the backend load-call-save on each turn. The API itself becomes a stateless endpoint: any instance can handle any request, no sticky sessions, no lost conversations on pod restart, and full replay for debugging.

Once you’re stateless, the next question is workflow length — that determines whether streaming is enough or async jobs are required.

2️⃣ Streaming vs async jobs

Once you’re stateless, you need to decide how the client receives results — and that depends entirely on whether a user is watching.

If they are, stream. Server-Sent Events keep the HTTP connection open and push each token as it’s generated. Perceived latency drops dramatically even if total generation time stays the same. SSE is the industry default for AI APIs — it’s one-directional, stateless over HTTP, and maps naturally onto how LLMs generate output. WebSockets are valid when you need bidirectional communication — for example, when a user can interrupt or redirect the agent mid-stream — but add handshake overhead and connection state you’ll need to manage.

If they’re not watching, or the workflow runs too long for a stable connection, return a task_id immediately and let the client poll or receive a webhook on completion. This lets you queue, retry, and prioritize work independently. A failed job replays without the user resubmitting.

Two downstream effects of streaming worth knowing: your rate limiting math changes (a cancelled stream still consumed input tokens), and your frontend now handles a stream, not a JSON object.

3️⃣ Rate limiting: RPM + TPM

Standard rate limiting counts requests per minute. For AI APIs this is incomplete.

Ten requests per minute sounds reasonable. If each carries a 50,000-token conversation history, those 10 requests exhaust your upstream budget in seconds. Token-per-minute (TPM) limits are how you protect against this.

• RPM (requests per minute) — protects against request floods and runaway clients

• TPM (tokens per minute) — protects against cost spikes from large-context requests and agentic loops that accumulate history

The interaction with streaming: a cancelled stream still consumed your input tokens. Estimate token cost before calling the model — if the client exceeds their TPM budget, reject with 429 before inference starts. For most production APIs this is cheaper than paying for the inference first — though for low-traffic APIs the estimation overhead may not be worth it.

4️⃣ Pydantic schemas

Pydantic does three jobs in an AI API. Most engineers know the first: validating incoming requests. In AI APIs it takes on two more — defining tool contracts for your agent and enforcing structured output from the model.

Two AI-specific places this matters. Tool definitions — when your agent calls tools, you describe each tool’s parameters as a schema. A loose schema produces unpredictable tool calls. A tight one with field descriptions and constraints produces reliable ones. Structured output — when you need the model to return JSON, define the expected shape as a Pydantic model. Most providers support constrained decoding, meaning the model can only generate tokens that produce valid JSON matching your schema. No more “parse the free-text response and hope for the best.”

Treat these schemas with the same care you’d give a public API contract. They are the interface definition between your deterministic code and your non-deterministic model.

In TypeScript the same principle applies with Zod.

Thank you for reading, let’s chat 💬

💬 Which of these did you get wrong first time and had to redo?
💬 Are you running inference inside your web tier right now — how is it holding up?
💬 What would you add as a 7th decision?

I love hearing from readers 🫶🏻 Please feel free to drop comments, questions, and opinions below👇🏻

Leave a comment

Moving from prototype to production forces a brutal architectural shift. You have to choose between raw LLM autonomy and rigid, deterministic control. But treating ReAct loops and State Machines as mutually exclusive binaries is a design trap.

TL;DR

• ReAct loops excel at dynamic discovery. Use them when the sequence of tool calls cannot be predicted at design time (e.g., open-ended research or diagnostics), and you can tolerate high latency and variable token costs.

• State machines excel at deterministic compliance. Use them when workflows require strict validation, predictable state transitions, and zero LLM autonomy over the execution path (e.g., transactional operations or financial checkouts).

• The Production meta-pattern: Never deploy an unconstrained ReAct loop. Instead, encapsulate the autonomous loop inside an fixed execution graph that acts as a programmatic escape hatch when the agent loses convergence.

What are they?

When you build an agentic workflow, you are choosing who owns the state router: the model or your code.

With ReAct, you hand control of tool selection to the model at runtime. This is the critical nuance most posts miss: the tools themselves are fully deterministic. A query_database tool always executes the same SQL. What’s autonomous is which tool the model picks next, and when it decides to stop. The workflow operates as an open-ended loop — Thought → Action → Observation — persisting until the model decides it has a final answer.

Here’s what “persisting” actually means in practice: every iteration appends to the context window. Thought, tool call, raw output, next thought — all of it accumulates. On a well-behaved three-step query, this is fine. On a diverging agent that keeps re-querying slightly different APIs hoping for a better result, your context window fills, your costs compound, and the model’s attention degrades. Context window accumulation isn’t a footnote — it’s the primary failure mode of unconstrained ReAct loops in production.

A lightweight mitigation that teams often overlook: enforce structured outputs (JSON schema) on each ReAct step. This forces the model to commit to a specific action format rather than free-text reasoning, which dramatically reduces both token waste and loop divergence. It doesn’t replace circuit breakers — it complements them.

With State Machines, instead of giving the LLM full control, you structure the workflow as a directed graph of nodes (LangGraph, Step Functions, Prefect). Technically, a State Machine is a subset: it has a fixed set of states and strict transition rules. LangGraph is actually a DAG-based orchestration engine, which is more flexible. The distinction matters because DAGs allow conditional branching and parallel execution that a true FSM cannot. The LLM lives inside individual nodes — parsing input, generating a plan, classifying intent — but deterministic code controls which node fires next. Crucially, in plan-and-execute style, the model reasons once upfront — generating the full execution plan as structured output — then steps out. Deterministic code owns the rest.

One assumption that bites teams: rollbacks are not automatic. “Easy session rollbacks” only exist if you’ve explicitly implemented state checkpointing — LangGraph Persistence, Step Functions execution history, or your own snapshot mechanism. Design for this upfront, not as an afterthought when your billing pipeline fails mid-transaction.

What to choose for your workflow?

Choosing an architectural pattern for your production workloads isn't about finding the "smartest" agent; it's about managing predictability, state persistence, and your cloud budget. Both ReAct and State Machines are fully viable for production, but only if deployed within their specific operational boundaries.

1️⃣ The pure ReAct loop / Autonomous Agent

Built for non-deterministic environments where the exact sequence of steps cannot be predicted at design time: automated log diagnostics, security threat hunting, open-ended research assistants.

• Pros: Exceptional at zero-shot tool selection and open-ended discovery across complex, unpredictable reasoning paths.

• Cons: Compounding context window growth, unpredictable latency, non-deterministic paths. Without hard guardrails, susceptible to infinite loops that exhaust both your context budget and your API budget simultaneously.

2️⃣ The pure State machine / Workflows

Built for regulated, transactional systems where data integrity is non-negotiable: e-commerce checkout, banking compliance, CRM ingestion pipelines, multi-step form validation.

• Pros: 100% deterministic routing, predictable token spending, native checkpointing (when configured), and clean rollback paths if a third-party API fails mid-transaction.

• Cons: Rigid by design. Unmapped edge cases hit dead ends. Requires significant upfront engineering to map every valid transition.

3️⃣ Dual-Strategy Escapement / Guarded Agent

This encapsulates a ReAct loop inside an Orchestration Graph that acts as a programmatic circuit breaker. The inner ReAct node handles loose, non-deterministic reasoning. The outer graph monitors hard metrics — iteration count, token consumption, semantic similarity between consecutive outputs (a strong signal the agent is spinning) — and forcibly intercepts execution if the agent fails to converge.

When the circuit trips, the system serializes current context and hands off to an immutable fallback route. The agent doesn’t just stop — it lands somewhere safe.

• Pros: The best of both worlds. Dynamic tool selection with guaranteed infrastructure reliability. You define exactly what “out of bounds” looks like in code, not in a prompt.

• Cons: Two distinct execution layers to design and maintain. State serialization across paradigm shifts is non-trivial. You need telemetry to know when and why the escape hatch is firing.

4️⃣ Upstream dynamic routing

A lightweight classifier at the entry point evaluates the raw query once and routes it to the right execution engine — ReAct for exploratory intent, Orchestration Graph for structured transactions.

The key architectural decision here is what kind of classifier. An LLM classifier (Haiku-tier) adds roughly 80–150ms latency and a small token cost, but handles nuanced or ambiguous queries gracefully. A rule-based classifier — regex patterns, embedding similarity against intent clusters, keyword matching — adds near-zero latency and zero variable cost, but requires manual maintenance as query patterns evolve. In practice, most production systems combine both: rules handle the 80% of clear-cut cases, LLM handles the ambiguous tail.

Either way, this introduces a single point of failure. A misclassified unstructured query routed to a rigid graph hits a dead end. Invest in classifier monitoring before you deploy this pattern at scale.

• Pros: Massive cost and performance optimization. Simple queries completely bypass the token-heavy ReAct loop.

• Cons: Permanent latency floor at the routing layer (rule-based: ~5ms, LLM-based: ~80–150ms). Misclassification sends the payload to the wrong engine.

⚠️ The circuit breaker

Every pattern above assumes your autonomous loop can eventually resolve itself — programmatically. That assumption breaks in high-stakes domains.

Human-in-the-loop is not a UX feature. It’s an architectural circuit breaker. When an agent has consumed N iterations without convergence, or when it’s about to execute an irreversible action (send an email, place an order, delete a record), the correct escape hatch is often a human approval step — not a fallback route.

Step Functions natively supports this with .waitForTaskToken. LangGraph Persistence lets you pause a graph and resume it after external input. The pattern is well-understood. What’s less understood is where to place the interrupt: the answer is at any node that crosses an irreversibility boundary, not just at max-iteration limits.

If you’re building anything that touches real-world side effects, design the HITL interrupt points before you design the agent’s tool set.

Thank you for reading, let’s chat 💬

💬 Where do you draw the line between "let the agent retry" and "escalate to a human"?
💬 Have you implemented structured outputs on ReAct steps? Did it reduce loop divergence?
💬 What's your circuit breaker metric — iteration count, token budget, or something else?

I love hearing from readers 🫶🏻 Please feel free to drop comments, questions, and opinions below👇🏻

Leave a comment

TL;DR

A useful mental model: queues primarily solve workload buffering and backpressure, while event buses primarily solve service decoupling and event distribution.

Use SQS when:

• You need reliable, point-to-point processing where consumers explicitly pull messages at their own pace (pull-based buffering).

• Strict worker isolation matters — you want dedicated queues so one failing consumer never stalls or risks impacting others.

• You are dealing with highly uniform workloads or heavy processing pipelines that require precise chunking/batching strategies.

Use EventBridge when:

• Multiple independent services need to react to the same architectural event.

• You want loose coupling — the producing system fires an event globally and doesn’t know or care who listens.

• You need complex routing and filtering rules evaluated directly on the infrastructure layer before hits reach your code.

• You’re building native cross-account workflows or SaaS integrations (e.g., Zendesk, Stripe, PagerDuty).

• Advanced features like a schema registry, native event archiving, or automated event replay are required.

What are they?

SQS is a message queue. Producers place messages into a queue, and consumer workers pull them for processing. While multiple consumers can process messages from the same queue, each individual message is delivered to and processed by only one consumer at a time. Messages persist safely in the queue until the consumer deletes them. The consumer is in control.

EventBridge is an event bus. One event fires, and a central router pushes copies of it to multiple matched targets simultaneously. The system relies on a publish/subscribe, push-based delivery model.

The wrong structural choice won’t break your system immediately. It just changes how hard your application is to maintain, scale, and reason about over time.

⚠️ FIFO Event Buses supported (introduced in 2023); suitable for ordering guarantees, though FIFO SQS remains the more common choice for ordered processing workloads

Why choose when you can use both?

A common misconception is that architects must choose either SQS or EventBridge. In practice, many production AWS systems use both together. EventBridge handles event distribution, filtering, cross-account routing, and SaaS integrations, while SQS provides consumer isolation, buffering, retries, and backpressure protection. A common pattern is Producer → EventBridge → SQS → Consumer, allowing teams to benefit from both loose coupling and resilient workload processing.

Thank you for reading, let’s chat 💬

💬 Which do you reach for first — SQS or EventBridge?
💬 Have you ever had to migrate from one to the other?
💬 What's your most common pattern?

I love hearing from readers 🫶🏻 Please feel free to drop comments, questions, and opinions below👇🏻

Leave a comment

⚠️ The Trap: Relevance-only thinking

When designing RAG systems, the natural instinct is to optimize for answer quality. Better chunking, better embeddings, better retrieval — all measured by how relevant the results are. Cost enters the conversation later, usually when it’s already compounding.

Every decision in your retrieval layer is also a cost decision:

• How you chunk documents determines how many vectors you store and search across at runtime

• The embedding model and size you choose determines your index footprint

• How you design your metadata fields determines whether query-time filtering narrows the search space effectively or adds overhead without reducing it

None of these feels like financial choices when you’re making them. They all are.

⚠️ The Trap: Treating all documents as equally Hot

OpenSearch offers storage tiers, and most RAG implementations ignore them entirely. This is an architectural oversight, not an engineering one.

The decision worth making explicitly: not all knowledge is equally hot. Documents ingested last week and queried daily have different economics than documents from two years ago queried occasionally. Treating them identically — keeping everything on high-performance storage — is a choice, and it’s usually the wrong one.

One important constraint: ANN indexes often can't be queried efficiently from cold storage without latency or rehydration cost. The lifecycle model applies to content that can tolerate slower retrieval — archival context, rarely accessed knowledge bases. For latency-sensitive RAG, "cold" may mean a separate index with a different serving strategy, not just a cheaper tier.

The architect’s job is to define the access pattern taxonomy upfront: what’s hot, what’s warm, what’s cold — and design an index lifecycle policy that matches the retrieval SLA to the storage cost. This is a business decision dressed as a technical one.

⚠️ The Trap: Searching blindly

How retrieval queries are structured has a bigger cost impact than most architects realize. The core tradeoff is simple: the narrower the search space before similarity ranking, the cheaper each query is.

If your RAG system serves multiple use cases, tenant segments, or vast document categories, scanning the entire global index for every query means you traverse the full vector graph regardless of relevance. You pay the full compute cost of a wide search, only to discard most results afterward. The alternative is to pre-filter for the subset of data that matters. At low volume this is invisible. At scale, it’s a budget killer.

The goal is to design a data model that enforces search space reduction upfront.

The architect-level question isn’t “how do we filter?” — it’s how do we model our data so filtering is always possible and always cheap? Are we relying on native metadata pre-filtering within a single index, or do we need strict physical partitioning across separate indices and shards?

⚠️ The Trap: Assuming Vector Search Costs Scale Linearly

Traditional database costs are predictable — more rows, more storage, roughly linear. Vector search doesn’t behave that way. The index that makes approximate nearest neighbor search fast is memory-resident and grows non-linearly with your corpus. This is the architectural surprise most teams hit too late.

The right mental model: you’re not paying per query, you’re paying for the index you’ve designed. A corpus of 10M small chunks is fundamentally more expensive to serve than 2M well-designed chunks with equivalent recall — not just in storage, but in the compute needed to search it. Well-designed here means: chunks sized so that recall doesn't require retrieving excessive top-k to compensate for poor granularity. Chunking is a multi-objective decision — too small inflates index size, too large degrades recall and forces broader retrieval to compensate.

Architect’s question: At 10x your current document volume, does your retrieval cost grow linearly or does it compound? If you don’t know, your index design hasn’t been stress-tested economically.

✅ The Mindset shift: Retrieval as an economic system

The pattern I see repeatedly: teams make retrieval decisions to hit a relevance benchmark, then treat cost as something to address later. By the time cost becomes visible, the index design is load-bearing — hard to change without rebuilding the pipeline.

The better approach is to treat retrieval design as an optimization across two axes simultaneously: recall quality and cost per query at scale. These aren’t in conflict if you design for both from the start. They become very much in conflict if you don’t.

Four questions worth answering before you finalize any RAG architecture:

1. What is our acceptable recall threshold — and have we tested whether a leaner design meets it?

2. What does our query volume look like at 10x, and what does that do to our index cost?

3. Do we have a document lifecycle model, or does everything live on hot storage forever?

4. Can our data model support pre-filtering, or are we searching blind every time?

Retrieval is not infrastructure you configure and forget. It’s a system with its own cost surface — and the architect’s job is to make that surface visible before the bill does.

Thank you for reading, let’s chat 💬

💬 What's your biggest OpenSearch cost surprise so far?
💬 Did you design your RAG retrieval layer with cost in mind from the start — or did it come later?
💬 How do you balance recall quality vs. retrieval cost in your architecture reviews?

I love hearing from readers 🫶🏻 Please feel free to drop comments, questions, and opinions below👇🏻

Leave a comment

That familiarity is the trap. LLMs carry a fundamentally different operational contract. The assumptions baked into your infrastructure — about latency bounds, output determinism, and failure modes — don’t hold. Not because the API is poorly designed, but because the underlying system behaves differently at a fundamental level.

Latency is not a number, it’s a distribution

Most internal services operate within predictable bounds — tens or low hundreds of milliseconds. LLM inference doesn’t work this way. Response time is a function of output token count, model size, and provider load at the moment of the request. A call that completes in 2 seconds normally can stretch to 15 or more under pressure — with no signal in advance telling you which scenario you’re in. Call LLMs synchronously from request threads and you’re one traffic spike away from a thread pool entirely blocked on AI inference, while unrelated parts of your system time out waiting for workers. This is upstream resource exhaustion — a cascading failure pattern where one slow dependency poisons the entire request pool.

✅ What to do: Decouple submission from completion. Async job queues let you accept the request immediately and process in the background. Where you can’t go async, use streaming — it doesn’t reduce total latency, but delivers the first token fast and keeps the user engaged. Note the distinction between TTFT (Time to First Token) and total latency — in production you often optimize them independently, since TTFT drives perceived responsiveness even when total generation time remains high.

The output is a promise, not a contract

A typed API returns a known shape. An LLM returns text that looks like what you asked for — until it doesn’t. The failure modes are subtle: a field is missing, a key gets renamed, the model wraps JSON in extra prose, a value changes shape. These aren’t edge cases — they’re the expected behavior of a probabilistic system at scale. The gap between “the LLM gave me a JSON string” and “this data is ready for my schema” is where production bugs hide — the kind that pass every test written during the PoC.

✅ What to do: Treat the LLM as an untrusted component at your system boundary. Validate every response against a schema before it touches your business logic. Use constrained decoding where available — provider-native JSON modes or libraries like Outlines and Guidance. They enforce schema at the token generation level, not after the fact. The model physically cannot produce output that violates the schema. A degraded fallback beats silent data corruption.

Naive retries will hurt you

Exponential backoff works reliably for transient failures in databases and downstream HTTP calls. With LLMs, the same logic applied carelessly becomes expensive and misleading. Retrying isn’t free — it costs tokens, time, and money. A 2,000-token request retried three times has consumed 8,000 tokens on a single user action, with output tokens costing 3-5x more than input. Worse, blind retries mask the real problem: if output consistently fails validation, the issue is in your prompt or schema design. Retrying won’t fix it, it’ll just obscure the signal. If a model is returning malformed output because your prompt is ambiguous, sending the same prompt again will produce the same malformed output — just slower and more expensively.

✅ What to do: Classify failures before retrying. A 429 rate limit warrants backoff with jitter. A 500 means the provider is down — back off and consider a fallback model. A content filter violation means retrying the same prompt is guaranteed to fail and waste money. Each error class needs its own handling path. Set token budgets per request path and alert when they’re regularly exceeded.

The model under you can change without warning

A traditional dependency ships breaking changes with a version bump and a migration guide. LLM providers don’t always extend the same courtesy. A prompt that works perfectly today can quietly degrade after a model update — your error rate stays flat, your latency looks normal, but the model stopped following an instruction it previously respected. Downstream, a dashboard shows wrong numbers that nobody notices for a week. The problem compounds when you’ve tuned prompts against a specific model version and have no record of what changed or when.

✅ What to do: Version your prompts and schemas, and log both alongside every model response. When validation error rates rise, you need to correlate against what was active at the time. Pin to specific model versions where your provider allows it, and treat behavior regression as a first-class incident — the same way you’d treat a latency spike or an elevated error rate.

Thank you for reading, let’s chat 💬

💬 Have you been burned by any of these assumptions in production?
💬 What was the first thing that broke?
💬 What’s the weirdest “production-only” LLM failure you’ve seen?

I love hearing from readers 🫶🏻 Please feel free to drop comments, questions, and opinions below👇🏻

Leave a comment

Here’s what happened at each stage — and what I’d do differently.

Phase 1: What are we actually building?

The request sounded simple: ingest documents, make them searchable via chat. So it’s tempting to jump straight into service selection. I knew the stack. Why waste time on discovery sessions?

Because “documents” means different things to different people, one of the important things I learnt from my mentor.

So I scheduled a discovery session and started asking questions:

1. What documents? The documents include HR policies, legal contracts, and employee performance reviews. That’s PII. That’s compliance. Nobody mentioned it in the initial brief because to the stakeholder it was obvious — of course, HR docs are sensitive. For me, it wasn’t, because I hadn’t asked yet.

2. What formats? PDFs, mostly. Some are scanned. Some are 200 pages.

3. What does “real-time” mean? Turns out they mean answers within 30 seconds, not sub-second latency. That’s a very different architecture constraint.

4. Who uploads documents? Not a developer — it’s an HR coordinator who has never used AWS in their life.

Phase 2: Now you design. Not guess

I sat down with my Discovery outputs. It felt like I finally had all the answers — the design should be straightforward now.

Some decisions were obvious once I knew the context:

• 200-page documents meant synchronous Lambda processing wouldn’t work — I needed an async pipeline with Step Functions to handle long-running jobs without hitting the 15-minute timeout wall

• PII meant access controls, audit logging, and a way to scan and redact sensitive content before it gets indexed

But some choices were not obvious at all. Take the embedding model — I had two options, and neither was clearly right:

1. Managed embedding service (Bedrock) — simple, no infrastructure overhead, fast to set up. But data leaves your VPC. I imagined presenting this to the stakeholder six months in, when the compliance team asks why employee performance reviews are being sent to an external API. Nobody would have flagged it because nobody had the conversation.

2. Self-hosted model on ECS — stays in your network, full control. But the team needs to maintain it. And when the model gets updated, every chunk in the vector DB becomes stale and needs reprocessing. If that’s not explicitly planned in the design, it will surprise someone in production.

Neither option was wrong. What would have been wrong is choosing one without showing the stakeholder both, and what each one actually costs them.

Phase 3: Someone has to actually build this

Design was approved. I felt good about the architecture. Then I handed it over to the engineering team.

Day one, they had questions:

• Why Textract and not a custom parser — it’s cheaper?

• Why Step Functions and not just a longer Lambda timeout?

These were reasonable questions. But I realized I had made a mistake — I had been so focused on getting the design right that I hadn’t brought the team along. They were hearing the reasoning for the first time while they were already supposed to be building. Some decisions got quietly reconsidered. Some got implemented differently because nobody had explained the why to the people actually writing the code.

The system that got shipped had started to drift from the system I had designed.

These three lessons came from one project. But the same patterns kept showing up across others — different stacks, different stakeholders, same gaps. Over time, I started writing down what I was learning.

I called it D3 for Discover → Design → Deploy.

Thank you for reading, let’s chat 💬

💬 What's the first thing you do when you get a new system design task?
💬 Have you ever discovered a critical requirement after the architecture was locked?
💬 Discover first or design first — how does your team actually work?

I love hearing from readers 🫶🏻 Please feel free to drop comments, questions, and opinions below👇🏻

Leave a comment

I spent a long time trying to figure out why this keeps happening. The technical skill was clearly there. The AWS services were solid choices. The diagrams looked clean. But the systems still failed to meet the business need. I found one thing they all had in common: the design happened in a vacuum. No structured discovery. No explicit assumptions. No trade-off assessment. Just engineers solving the problem they imagined, not the problem that existed.

TL;DR: Brilliant engineers can build wrong systems when the design process skips the hard questions. Technical skill doesn’t protect you from building the right solution to the wrong problem.

❌ Picking the storage layer before understanding access patterns

A team built a multi-tenant analytics platform on DynamoDB. Solid choice on the surface — they needed scale, low latency, and serverless. The schema looked thoughtful: single-table design, composite keys, GSIs for the main queries. But six months after launch, their customers were asking for ad-hoc filtering, date range queries across tenant data, and CSV exports. None of that was in scope during design. DynamoDB doesn’t do ad-hoc well — you either know your access patterns upfront or you pay for it later.

They ended up bolting on Athena over S3 exports, which solved it — but added operational complexity, export lag, and cost that wasn’t in the original TCO.

The access patterns were knowable. Nobody asked. The storage choice was made before the stakeholders were interviewed properly.

Tip: Before picking a NoSQL store, define your Core access patterns. If you expect more than ~30% of your queries to be unknown or ad-hoc in the first year, stick to Relational or build a robust export strategy from Day 1.

❌ Designing for scale you’ll never reach

A startup building a B2B SaaS product designed their event pipeline for 10 million events per day from day one. Kinesis Data Streams, multiple Lambda consumers, DynamoDB for state — everything sized for a load they had no evidence was coming. Infrastructure costs were $4,000/month before a single paying customer.

Eighteen months later, they had 40 customers and 200K events per day. The pipeline was 98% idle. They were also hitting a wall nobody anticipated: their customers wanted synchronous webhooks with retry guarantees, not async event streams. The architecture was built for the wrong scale and the wrong integration model.

Meanwhile another team — same domain, similar product — started with SQS and a single Lambda consumer. When load grew, they scaled with a few config changes. When customers asked for webhooks, they added an outbound processor on the same queue. Their infrastructure was $300/month at the same customer count.

The first team had a technical debate about throughput. Nobody had a structured conversation about what load was actually expected, what customers actually needed at the integration layer, and what the cost envelope looked like.

Tip: Separate the scale conversation from the architecture conversation. Validate your load assumptions before you commit to the infrastructure that supports them. If you can’t get real data, define a range and design for the lower end first, but ensure a clear promotion path to scale up without a total rewrite.

❌ Skipping the non-functional requirements until it’s too late

A team migrated a healthcare scheduling system to AWS. The migration was technically “perfect”—clean IaC, multi-AZ RDS, and a well-structured API Gateway + Lambda stack.

Three weeks post-launch, Compliance flagged a HIPAA requirement: every read of patient data had to be logged with the specific user’s identity and the data accessed. Not just “who logged in,” but “who saw what.”

The system had no mechanism for this. The Lambda functions were calling RDS directly with no shared data-access layer or middleware. Adding this wasn’t a “config change”—it meant instrumenting every function, restructuring the data access logic, and building a separate, immutable audit store. It cost them four weeks of rework on a “finished” system.

HIPAA wasn’t a surprise; it was a known constraint. But it was never surfaced during design because nobody ran a structured session to ask about audit obligations before the architecture was locked.

Tip: Non-functional requirements — compliance, availability SLAs, data retention, audit obligations — must be collected before Design starts, not discovered after launch. They change the architecture, not just the implementation.

None of these failures required better engineers. They required a better process — one that forces the hard questions before any service is chosen: What problem are we actually solving? Who are the real users and what do they need? What are the constraints we can't violate? What assumptions are we making, and what happens if they're wrong?

The architecture isn’t designed in a vacuum. It’s designed around a clear problem, with explicit trade-offs, and documented assumptions that anyone can review. I’ve been working on formalizing this kind of structured approach for architecture delivery — from first stakeholder conversation to handover. More on that soon.

Thank you for reading, let’s chat 💬

💬 Have you ever shipped something technically solid that still missed the mark?
💬 Discovery first or design first — how does your team actually work?
💬 What question do you always ask before picking a service?

I love hearing from readers 🫶🏻 Please feel free to drop comments, questions, and opinions below👇🏻

Leave a comment

TL;DR: SQL handles a significant subset of document use cases well. The gap between relational and document stores narrowed significantly. The choice between relational and document stores is now less about data shape and more about scale, write throughput, and schema change velocity.

When SQL started winning

The workloads where relational databases caught up share an equally clear pattern.

• The data model is stable, but the access pattern is not. This often happens at a specific product lifecycle moment: the schema has settled, the team understands the domain, and operational requirements like transactions and integrity start mattering more than flexibility did early on.

• Consistency actually matters, but was being sacrificed for convenience. A significant amount of NoSQL adoption was driven by developer experience — joins felt painful, ORMs added friction, schema migrations felt risky. Not by genuine technical constraints. Those are solvable problems that don’t require giving up ACID guarantees.

• The system needs both integrity and flexibility. Analytics, reporting, referential constraints, and complex queries sitting alongside document-style application workloads. Previously that required two separate systems. Storing data relationally while accessing it as JSON collapses it into one — and the tooling ecosystem around SQL (BI tools, analytics, observability) is still heavily SQL-first. That operational reality matters.

• The scale is real but not extreme. Hundreds of millions of records, moderate write throughput, occasional spikes — this describes a large share of production systems. Not Netflix, not Uber. The operational complexity of running a dedicated document store at this scale is rarely justified by what you get back.

PostgreSQL’s JSONB has been production-ready for years. Some teams are re-evaluating earlier NoSQL choices — particularly SaaS companies that adopted MongoDB before their consistency and integrity requirements solidified. Not because document stores were wrong, but because the original constraints that justified them no longer dominate.

When document stores still win

The workloads where document stores genuinely outperform relational databases share a few structural characteristics.

• Schema evolves faster than you can handle. When your data model changes multiple times per week across hundreds of millions of records — new attributes, different shapes per entity type, heterogeneous structures within the same collection — relational schema management becomes an operational bottleneck.

• Write throughput exceeds what relational can absorb. At very high write throughput, coordination overhead — locking, indexing, transactions — can become a bottleneck. Document stores and wide-column databases are designed to avoid it. This is workload-specific, not an inherent weakness of relational databases, but at the extreme end the gap is real.

• Consistency can be relaxed by design. Some systems explicitly choose eventual consistency to gain availability and partition tolerance. That’s not a limitation or workaround — it’s an architectural decision. Relational databases with ACID guarantees aren’t the right tool when you’ve deliberately chosen a different consistency model.

• Data is truly heterogeneous at scale. Not “our objects have nested arrays” — that’s JSON-in-SQL territory. But genuinely sparse, unpredictable attribute sets across billions of records, where a fixed schema adds no value and adds real storage overhead.

These patterns rarely appear in isolation. The strongest cases for document stores — Netflix, Uber, Amazon internal systems — sit at the intersection of several simultaneously. A single pattern alone is worth examining carefully before it justifies the operational overhead.

One axis worth considering: running MongoDB or Cassandra clusters in production is operationally heavier than managed relational databases. Backup strategies, consistency guarantees, debugging query performance on deeply nested JSON — these are real costs that rarely appear in architecture discussions but show up in engineering hours.

The actual decision space

The database choice is rarely wrong at the start — it becomes wrong as the system evolves. Early-stage products have uncertain schemas, unclear access patterns, and low operational maturity. Document stores are genuinely lower friction here. The problem is that early decisions get locked in. By the time transactional integrity, complex queries, and referential constraints become real requirements, migration is expensive.

Migration is hardest when your data model and access patterns diverge from your storage model — regardless of which direction you’re moving. This is worth factoring in at the start, not after the system has grown around its database.

The constraints worth evaluating honestly:

• Scale, write throughput, schema velocity at the extreme end? — Document stores were designed for this, and the gap is unlikely to close soon.

• Consistency and integrity requirements that grow over time? — They almost always do. Relational databases age better here.

• Complex queries on nested structures? — JSON in SQL is not equivalent to native document indexing. But the deeper issue is that complexity doesn't disappear — it shifts. With document stores, complexity lives in the schema design. With JSON in relational databases, it moves into the query layer. Schema is explicit and auditable. Query logic that encodes structural assumptions is harder to catch and harder to maintain.

• Early stage, schema still uncertain? — Both are viable. The question is which direction is cheaper to reverse if requirements change.

Thank you for reading, let’s chat 💬

💬 Have you ever migrated away from a document store?
💬 Has your database choice ever become wrong as the product evolved?
💬 What pushed you to a document store when SQL wasn't enough?

I love hearing from readers 🫶🏻 Please feel free to drop comments, questions, and opinions below👇🏻

Leave a comment

TL;DR: Every architecture is a product of its context. There are no bad architectures, only expensive ones. (inspired by Mark Richards and Neal Ford)

The Ideal Architecture

A production-grade GenAI chatbot should be highly available, scalable, and resilient. Responses should stream token by token. It should handle multiple models and scale efficiently under load.

On AWS this means:

• Frontend & Delivery — CloudFront and S3 serve the UI globally with edge caching. An ALB acts as the entry point, supporting HTTP Chunked Transfer Encoding to deliver a "typewriter" streaming experience to the client with minimal TTFT (Time to First Token).

• Backend — ECS serves as the "brain," managing prompt templates and RAG logic in a container environment. It retrieves conversation state from ElastiCache and invokes Bedrock using streaming APIs, bypassing API Gateway timeouts for long-running reasoning tasks.

• RAG Pipeline — S3 stores source documents. When new documents arrive, Lambda generates embeddings and stores them in OpenSearch for semantic search (or Aurora (pgvector) if the team prefers SQL). SQS decoupling ensures ingestion doesn't bottleneck during bulk uploads. At query time, the most relevant context is retrieved and passed to the model.

• Observability — CloudWatch and X-Ray provide logs, metrics, and distributed tracing across all components.

• Resilience — Multi-AZ database, ECS auto-scaling, and dead letter queues ensure the system handles failures gracefully.

• Security — Bedrock Guardrails provide native PII redaction and content filtering.

Robust. But also complex, expensive, and weeks of work to do right.

My Architecture

This was a presale demo tool — deployed into a client’s AWS account for hours or a couple of days, then torn down. The goal: show that RAG works, that clients could connect their own knowledge base and get relevant answers, using either third-party models via API or their own models hosted on SageMaker endpoints. Their data, their models, their account — that’s a trust argument in a sales conversation, not just a technical detail.

The entire infrastructure was provisioned with Terraform, which made it repeatable and deployable in any AWS account without manual steps.

The system had several EC2 instances running containers: UI, backend, and database connector. RDS with pgvector served as the vector store.

Now let me explain what this architecture deliberately doesn’t have — and why:

• No auto-scaling (Scalability) — we knew the load: one client, a few people in a room. Scaling adds complexity and cost with zero benefit for a two-hour demo.

• No Multi-AZ Failover (Reliability) — if something breaks during a demo we fix it manually. The cost of Multi-AZ for an ephemeral environment makes no sense.

• No streaming — in a presale, a correct and relevant answer matters more than how it appears on screen. Streaming is a UX improvement for production. Implementing streaming adds significant development overhead and introduces potential points of failure in the frontend-backend handshake.

• No dedicated vector database — OpenSearch is overkill for a handful of demo documents. We used RDS with pgvector because it was faster to develop with a team that already knew SQL, easier to manage, and significantly cheaper for a short-lived environment.

• No Aurora Serverless — it was explicitly rejected because it scales to zero which means cold start. For a two-hour demo that’s unacceptable.

• No centralized logging — setting up CloudWatch Logs Insights or X-Ray for a single-node, two-hour demo is overkill. We relied on standard Docker logs for real-time debugging, prioritizing speed of deployment over long-term operational monitoring.

The requirements were clear: build fast, design for portability and ephemerality. Every decision flowed from that. Not an ideal architecture — but a perfect one for this context.

Thank you for reading, let’s chat 💬

💬 Would you like to see more posts like this?
💬 What constraints shaped your last architecture decision?
💬 What was the hardest trade-off you made on a tight deadline?

I love hearing from readers 🫶🏻 Please feel free to drop comments, questions, and opinions below👇🏻

Leave a comment

TL;DR: Pipes gives you point-to-point integrations between AWS services without glue code. It wins on operational simplicity, and often on cost. The catch: once your transformation logic gets complex, you’re partially back to Lambda anyway.

✅ Replace your passthrough Lambdas

The classic glue pattern: a Lambda reads from a source, does minimal transformation, writes to a target. You’re paying for cold starts, managing deployments, writing error handling for a function with zero business logic, and getting paged when it breaks at 2am. Pipes replaces this with a fully managed pipeline: Source → optional filter → optional enrichment → target. No function to deploy, no CloudWatch logs to sift through for something that does nothing interesting. If your Lambda’s entire job is reading from one AWS service and writing to another — that’s a Pipe.

Tip: Audit your Lambda inventory for functions under 30 lines that only call AWS SDK methods. Those are Pipe candidates.

✅ Filter at the source, not inside your function

With SQS→Lambda, every message triggers your function — including ones you immediately discard. Pipes filtering happens before invocation. Messages that don’t match never reach your target. For a Kinesis stream processing 10M events/day where 85% are irrelevant: that’s 8.5M Lambda invocations eliminated. At standard pricing (128MB, 200ms average): ~$34/month saved on invocations alone, before compute time.

Pipes charges $0.40 per 1M events — but filtering is free. Enrichment and target delivery are what's charged. If 85% of your Kinesis stream is filtered out, you only pay for the 15% that reaches the target.

Tip: Define filter criteria at the Pipe source first. Even a single attribute match can cut downstream costs significantly.

✅ Enrichment without a full Lambda

Pipes has a built-in enrichment step: call a Lambda, API Gateway, or Step Functions to augment the event before it reaches the target. The difference from the old pattern is intent — enrichment is a deliberate step in a managed pipeline, not glue holding two services together.

This matters when you need some logic — a database lookup, an external API call — but don’t want a standalone function managing the entire routing lifecycle.

Tip: Use enrichment for data augmentation, not transformation logic. If your enrichment function is doing routing decisions, reconsider the design.

⚠️Production gotchas:

• No managed re-drive — unlike SQS, you need to build your own replay mechanism for failed events

• DLQ retention mismatch — DynamoDB Streams keep records for 24hrs; if events sit in DLQ longer, replay is impossible

• Batch retry behavior — one failed record retries the entire batch; without idempotency on the target, you’ll process records multiple times

• PrivateLink required for VPC sources — easy to miss until your Pipe silently can’t reach the source

• Throughput limits aren’t obvious — Pipes adds its own concurrency limits on top of source limits; high-volume Kinesis streams can hit Pipes limits first

Thank you for reading, let’s chat 💬

💬 Have you replaced Lambda glue code with EventBridge Pipes? Was it worth it?
💬 Hit the VTL wall and ended up adding Lambda enrichment anyway?
💬 Any production gotchas I missed?

I love hearing from readers 🫶🏻 Please feel free to drop comments, questions, and opinions below👇🏻

Leave a comment

❌ Paying for speed when data sits idle

Express makes sense when data is accessed constantly — multiple times per hour. If your data sits cold between jobs, you’re paying 5x more for nothing.

Tip: Use Express for hot data with frequent access. Move cold data back to Standard with lifecycle policies.

❌ Skipping the TCO math

For workloads running twice daily, monthly savings on requests and compute ($388) exceeded the storage premium ($174). Express won — but only because they ran the numbers.

Tip: Express beats Standard when (reduced compute cost + reduced request cost) > storage premium. Calculate before migrating.

❌ Ignoring object size impacts

AWS’s “50% cheaper requests” claim only applies to objects under 512KB. Above that, you pay additional per-GB charges.

When object sizes exceed 645KB, Express APIs become more expensive per byte than Standard. The sweet spot is frequent access to many small objects, not large files.

Tip: Express wins with small objects (<512KB) accessed frequently. Large files accessed infrequently? Standard is cheaper.

❌ Storing critical data in a single AZ

Express stores data in a single Availability Zone. If that AZ fails, your data is gone until it recovers.

The right approach is using Express as a cache layer with the source of truth in Standard. Analytics jobs can pull hot datasets into Express at job start, run queries, then delete the copies. If the AZ fails mid-job, reload from Standard.

Tip: Express works for caching or ephemeral data where AZ failure is tolerable. For anything you can’t afford to lose, keep the source in Standard.

❌ Underestimating the migration work

Express uses directory buckets with session-based authentication instead of standard IAM. Your existing S3 code won’t work without changes—you need CreateSession calls before every operation.

Directory buckets also don’t support versioning, object lock, or replication. One team needed versioning for regulatory requirements and couldn’t use Express at all.

Tip: Budget time for code changes and verify your required S3 features are supported before committing.

❌ Using Express for continuous high-throughput workloads

Jack Vanlightly’s analysis shows Express isn’t cheap enough to replace replicated cache implementations for transactional workloads. Systems like Neon and Kora use replicated write-ahead logs because replication costs scale with throughput, not request count. Express charges per request.

Express works for bursty analytics jobs with high-volume access patterns, but continuous high-throughput transactional systems still need traditional replication.

Tip: Express fits batch-oriented workloads. For 24/7 high-throughput transactional systems, replicated caches remain more cost-effective.

✅ When Express actually makes sense

Express delivers ROI when it removes a performance bottleneck causing you to over-provision elsewhere:

• I/O latency blocks expensive compute: ML training with frequent checkpointing, distributed processing where workers wait on S3, or batch jobs where compute sits idle during data access.

• User experience demands sub-second response: Interactive dashboards, real-time analytics, or applications with high-concurrency S3 reads where Standard’s 150-250ms latency creates noticeable delays.

• Faster S3 enables downstream optimization: When S3 latency bottlenecks your pipeline, forcing you to over-provision other infrastructure. Pinterest’s story perfectly illustrates this — faster S3 let them reduce MemQ brokers, cutting costs by 40% across the stack.

Thank you for reading, let’s chat 💬

💬 Have you tried S3 Express One Zone? What workloads did you use it for?
💬 Did the TCO math work out in your favor, or did storage costs surprise you?
💬 Have you hit unexpected limitations with directory buckets or single-AZ durability?

I love hearing from readers 🫶🏻 Please feel free to drop comments, questions, and opinions below👇🏻

Leave a comment

AWS’s recent documentation emphasizes “consider your operational needs” and warns about added complexity. It’s less “always single-table” and more “use it when access patterns align.”

❌ Lost operational flexibility

Single-table design locks you into one configuration for all data types. Every entity shares the same capacity mode, backup schedule, table class, and TTL settings.

I worked with a team that had user profiles needing on-demand capacity for spikes, plus historical analytics that could use provisioned capacity. Single table forced on-demand for everything. The monthly bill was 3x higher than needed. Another team needed hourly backups for transactions, but only daily for audit logs. A single table meant backing up everything hourly or risking data loss. Storage costs doubled.

Tip: Separate tables let you use on-demand for hot data and Standard-IA class for cold data at 50% savings.

❌ The observability nightmare

You can’t CloudWatch alarm on specific entity types — when read capacity spikes, you’re guessing which entity caused it.

S3 exports also become painful. One team wanted to analyze user behavior, but had to export everything — users, orders, inventory, and sessions. Each export took 45 minutes and cost $8. Table scans are even worse; migrating one entity type means scanning and paying for millions of irrelevant items.

Tip: Separate tables give entity-level metrics and exports. Alarm on user table throttles specifically and export only what you need.

❌ Development complexity tax

New developers take weeks to understand single-table schemas. Every entity needs composite keys: User-{id}, Order-{id}, Inventory-{sku}. You end up with GSI1PK and GSI2SK everywhere.

One team spent two weeks adding a “query by order status” feature. They needed a new GSI and had to migrate all items to include the new key, coordinating deployment across services. With separate tables, adding a GSI is one simple schema change on one table. No migration, no coordination.

❌ When parallel requests beat complex queries

Single-table design is optimized for fetching related data in one query. But modern frontends fire parallel requests — and two simple queries might be faster than one complex query with heavy filtering.

A team I worked with had a dashboard showing user details plus recent orders. The single-table query was complex: fetch user by PK, query GSI1 for orders, then filter by date range. Total latency: 120ms. We tested splitting it into two separate tables with parallel requests. User fetch: 40ms. Orders fetch: 45ms. Total latency: 50ms because the frontend made both calls simultaneously. Your mileage will vary — it depends on backend orchestration and whether you need atomic reads.

Tip: Network latency is low. Two parallel 50ms queries often beat one 120ms query — especially since you can cache them independently.

❌ The 10GB partition limit

If you choose to use Local Secondary Indexes, DynamoDB limits item collections to 10GB per partition key. Not all single-table designs use LSIs, but when they do, the concentrated data makes this limit easier to hit.

One team stored user data with LSIs for sorting. When active users exceeded 10GB of data (including all related items), writes started failing. They had to partition users across multiple “user shard” keys — adding complexity that multiple tables would have avoided entirely.

❌ GraphQL’s contradiction

STD forces you into “lookahead” resolvers to fetch nested data in one query. I saw a team turn a simple field addition into a week-long project because the lookahead logic and query builders were so tightly coupled.

Tip: Simple resolvers with separate tables lead to cleaner, more maintainable code than complex “all-in-one” queries.

✅ When single-table actually makes sense

Single-table design works well for tightly coupled domains with stable access patterns. If you’re building a multi-tenant SaaS where tenant isolation is critical, or your entities are always queried together (user + preferences + settings), single-table can be the right choice.

The problem isn’t the pattern itself — it’s treating it as the default instead of a tool for specific cases.

Thank you for reading, let’s chat 💬

💬 Have you been forced into single-table design when separate tables would have been simpler?
💬 What's your take on the development complexity vs runtime performance?
💬 Have you hit the 10GB partition limit?

I love hearing from readers 🫶🏻 Please feel free to drop comments, questions, and opinions below👇🏻

Leave a comment

This post breaks down how these new structural components actually fit into a your architecture. We’ll look at the shift toward multi-layer enforcement, the trade-offs of centralized policy management, and the practical constraints — from latency hits to Cedar policy limitations — that determine whether these tools are actually viable for your current stack.

1️⃣Three-Layer Guardrail Enforcement

Instead of a single on/off switch, guardrails work best when applied at multiple stages of the request lifecycle:

• Input Layer: Evaluate user input before model invocation to block prompt injection and detect PII. This reduces unnecessary model calls and expensive RAG retrievals.

• Processing Layer (AgentCore): Enforces tool-use policies for agents at runtime. If an agent attempts to invoke a delete_user action but the policy allows only read access, the runtime blocks the call.
  ⚠️ This layer controls capabilities, not content.

• Output Layer: Validates model outputs for PII and hallucination signals before returning results to the user or downstream systems.

Trade-offs: Each layer adds latency, with output checks being the most expensive. Use selectively for high-risk outputs.

2️⃣Centralized Guardrail Application Across Models

The ApplyGuardrail API enables consistent guardrail enforcement across multiple model backends:

• Native Bedrock models (Claude, Llama)

• External APIs (OpenAI, Anthropic) via a Lambda wrapper

• Self-hosted models on SageMaker

Why it matters: Policies remain consistent when switching or combining model providers.
⚠️ Enforcement for non-Bedrock models depends on your integration layer, which may add latency and reduce observability.

Trade-offs: Increases “Time to First Token” due to double-hop network calls (Guardrail API + Model API); centralized governance creates a single point of failure for all model traffic.

3️⃣RAG Verification Strategy

RAG systems fail differently from standard LLMs. This concept combines two key safeguards:

• Contextual Grounding: Ensures generated answers are supported by retrieved documents rather than being purely generative. Accuracy is strongest for English and well-structured sources.

• Automated Reasoning: Applies formal logic (SMT solvers) to verify output correctness in regulated use cases (finance, legal, compliance).
  ⚠️ ~99% accuracy claims apply only to formalizable reasoning tasks, not open-ended text.

Trade-offs: Computationally expensive; only useful when formal verification is required.

🚩 Production Gotchas

• Limited explainability: Bedrock Guardrails do not provide detailed reasons for blocks, complicating false-positive debugging.

• Service quotas: Default RPM/TPM limits may be insufficient for production workloads.

• Cedar policy constraints: No floating-point arithmetic or advanced regex, which can be limiting for fine-grained financial/technical rules.

• Non-English performance: Contextual grounding primarily supports English; performance degrades significantly for other languages.

• DLQ pattern: Route blocked requests to an SQS DLQ for analysis instead of returning generic 4xx errors.

• Versioning: Avoid LATEST. Treat guardrails as versioned, immutable infrastructure — validate in shadow environments before promotion.

Thank you for reading, let’s chat 💬

💬 Which layer of guardrails do you implement first — input, processing, or output?
💬 What's your current latency budget for safety checks in production?
💬 Have you hit service quota limits with Bedrock Guardrails?

I love hearing from readers 🫶🏻 Please feel free to drop comments, questions, and opinions below👇🏻

Leave a comment

✅ When API Gateway shines: Heavy business logic

If your GraphQL API is primarily business logic rather than data access, API Gateway + Lambda is simpler than AppSync’s multi-resolver approach.

Consider an e-commerce pricing engine with discount calculations, tax logic, and external API validations. With API Gateway, you write one Lambda that handles everything:

const resolvers = {
  Query: {
    productPrice: async (_, { productId, userId }) => {
      const user = await getUserTier(userId);
      const basePrice = await getProductPrice(productId);
      const discount = calculateDiscount(user, basePrice);
      const tax = calculateTax(basePrice, user.location);
      
      return await pricingAPI.validate({ basePrice, discount, tax });
    }
  }
};

With AppSync, you’d split this into multiple Lambda resolvers plus VTL or APPSYNC_JS orchestration to chain them together. More moving parts, more complexity.

The cost reality: For 10M requests/month where each query needs complex processing (500ms total):

• AppSync: $40 (Query) + $100 (Lambda) = ~$140 total

• API Gateway (HTTP API): $10 (Request) + $83.33 (Lambda) = ~$93.33 total

Note: These are estimates; actual costs depend on memory size and the new 2025 Lambda billing for initialization (cold starts).

The pricing model favors consolidated logic over distributed resolvers. So, API Gateway, in most cases, is around 30% cheaper because you avoid AppSync’s per-query cost and multiple resolver invocations.

✅ When API Gateway shines: Adding GraphQL to existing REST API

Here’s the migration trap many teams fall into. You have a successful REST API on API Gateway with established auth, monitoring, and deployment pipelines. Now you want to add GraphQL capabilities.

With AppSync, you’re building a second, separate API. Different auth configuration (your REST uses Lambda authorizer, now you need Cognito or IAM for AppSync). Different CloudWatch dashboards. Different rate limits and quotas. Different deployment pipelines. Your mobile clients now need to maintain connections to two different APIs with potentially different authentication flows.

With API Gateway, you add a /graphql route to your existing API. Same Lambda authorizer works for both REST and GraphQL endpoints. Same CloudWatch Logs, same X-Ray traces. Same domain name, same SSL certificate. One unified API surface that your clients interact with.

The operational overhead of maintaining two separate APIs is real and often underestimated. Double the monitoring dashboards, double the alerting rules, double the cost allocation tracking. If you already have API Gateway infrastructure and tribal knowledge, adding GraphQL there is the path of least resistance.

✅ When API Gateway shines: High-volume APIs

At high request volumes, the per-request pricing difference becomes significant. AppSync charges $4 per million query/mutation requests. API Gateway charges $1 per million requests. This 4x difference compounds at scale.

For 10M simple queries per month (single DynamoDB lookup, 100ms Lambda execution):

• AppSync costs $40 (queries) + $2.50 (DynamoDB) = $42.50

• API Gateway costs $10 (requests) + $16.67 (Lambda at 128MB, 100ms) + $2.50 (DynamoDB) = $29.17

That’s 31% cheaper for identical functionality. At 50M requests/month, you’re saving $67/month. At 100M requests/month, you’re saving $1,340/month. The math shifts heavily toward API Gateway as volume increases.

Important caveat: API Gateway’s advantage assumes your Lambda execution time stays reasonable. If you’re doing simple CRUD where AppSync can use direct APPSYNC_JS resolvers (no Lambda), AppSync might win on cost. But in practice, most production APIs need Lambda for business logic anyway, so API Gateway’s lower base cost tends to win.

Also worth noting: API Gateway supports 10MB payload sizes versus AppSync’s 256KB limit. If you’re handling file uploads, image processing requests, or large batch mutations, AppSync’s limit becomes a hard constraint. You’ll need pre-signed S3 URLs as a workaround, while API Gateway handles it natively.

✅ When API Gateway shines: Local development and debugging

Development velocity matters, especially when debugging production issues at 3 AM. API Gateway + Lambda gives you the full Node.js/Python ecosystem and standard debugging workflows.

You can test your entire GraphQL API locally with sam local start-api, set breakpoints in your IDE, and step through resolver logic. Use any npm package you want—Winston for logging, Sentry for error tracking, custom middleware for authentication. Your error handling is pure JavaScript:

const server = new ApolloServer({
  formatError: (error) => {
    logger.error(’GraphQL error’, { 
      message: error.message,
      path: error.path,
      user: context.user 
    });
    if (error.message.includes(’database’)) {
      return new Error(’Service temporarily unavailable’);
    }
    return error;
  }
});

With AppSync, you deploy to AWS to test every VTL or APPSYNC_JS change. No local testing environment. Debugging means parsing verbose JSON logs in CloudWatch and adding $util.log() statements. APPSYNC_JS doesn’t support npm packages — you’re limited to its built-in standard library. X-Ray tracing works, but it feels like an afterthought compared to Lambda’s first-class X-Ray integration.

When something breaks in production, API Gateway’s familiar patterns—CloudWatch Logs you can grep, X-Ray traces that make sense, error stack traces that point to actual code — mean faster mean time to recovery. The operational simplicity compounds over time.

Caching insight: API Gateway caches the entire GraphQL response per query. AppSync caches per resolver, meaning if your query triggers 5 resolvers, each checks its cache independently. For queries that request multiple fields, API Gateway’s query-level caching often delivers better cache hit rates than AppSync’s resolver-level approach.

⛔ The main concern: cold starts

The common objection to API Gateway + Lambda is cold start latency. Lambda cold starts for Node.js at 128MB typically add 50-200ms. AppSync’s APPSYNC_JS resolvers talking directly to DynamoDB have ~5-10ms overhead. Seems like a huge difference, right?

But here’s what matters in production: cold starts occur in less than 1% of Lambda invocations according to AWS’s analysis. Lambda reuses warm execution environments for 10-15 minutes. So 99%+ of your requests hit warm Lambda functions with no cold start penalty.

If cold starts truly matter for your use case: Provisioned Concurrency keeps functions warm for $54/month per 5 instances. Lambda SnapStart (Java only) reduces cold starts by 90%. Increasing memory allocation from 128MB to 512MB also speeds up cold starts. These options work identically whether you’re behind AppSync or API Gateway.

🎁My decision framework

Choose API Gateway + Lambda if:

1. Adding GraphQL to existing REST API (avoid two separate APIs)

2. Heavy business logic (complex transformations, external APIs, retry logic)

3. High volume (>10M requests/month where per-request cost matters)

4. Team prefers standard Node.js/Python (avoid VTL/APPSYNC_JS learning curve)

5. Need local testing and debugging flexibility

6. Custom auth requirements beyond Cognito/IAM

Choose AppSync if:

1. Real-time subscriptions are critical (WebSocket lifecycle handled automatically)

2. GraphQL-first greenfield project (no existing REST infrastructure)

3. Simple CRUD with direct data access (DynamoDB/Aurora queries)

4. Low to moderate volume (<5M requests/month)

5. Team wants schema-first discipline enforced

Reality check: Even IMDb (Amazon-owned) uses API Gateway + Lambda for GraphQL at 10,000+ TPS, not AppSync. It scales.

Thank you for reading, let’s chat 💬

💬 Does your team actually use Amplify Mock?
💬 What’s the most “creative” way you’ve bypassed the 256KB payload limit?
💬 If you had to start over today, would you go AppSync?

I love hearing from readers 🫶🏻 Please feel free to drop comments, questions, and opinions below👇🏻

Leave a comment

The typical framing is: “Do you need GraphQL? Use AppSync.” But the actual question should be: “What are you optimizing for, and what operational complexity can your team handle?” Let me show you where AppSync genuinely excels — and the hidden costs.

✅ When AppSync shines: Real-time subscriptions

AppSync subscriptions are transformative for real-time updates. They handle WebSocket connections, message filtering, and fan-out automatically.

type Subscription {
  onOrderStatusChange(customerId: ID!): Order
    @aws_subscribe(mutations: [”updateOrderStatus”])
}

With API Gateway, you’d build this yourself: WebSocket APIs + DynamoDB connection tracking + Lambda broadcasting. That’s 3-4 services vs AppSync’s built-in handling.

The cost reality: WebSocket connections cost money even when idle.

For 10,000 users connected 8 hours/day (rough estimate):

• Connections: $0.08/million minutes = ~$384/day

• Messages: $1.00/million messages

If your “real-time” is actually “poll every 30 seconds,” REST might be cheaper.

✅ When AppSync shines: Multi-source data federation

One GraphQL query, multiple data sources in parallel:

type User {
  profile: UserProfile      # DynamoDB
  orders: [Order]           # RDS Aurora
  recommendations: [Product] # Lambda + ML
}

AppSync orchestrates this automatically. With API Gateway + Lambda, you write the orchestration logic yourself.

The N+1 problem: Watch out for this query:

{
  users(limit: 100) {
    orders  # 100 separate DB calls!
  }
}

AppSync makes 1 call for users, then 100 calls for orders. You need DataLoader patterns or batch resolvers to fix this.

✅ When AppSync shines: Schema-first discipline

AppSync forces you to maintain a .graphql schema file as your API contract:

• Frontend teams work from schema immediately

• Breaking changes are explicit

• Auto-generated documentation via GraphiQL

• Prevents “nobody knows what the API actually does” syndrome

This is less about AppSync features and more about enforced discipline that pays off.

✅ When AppSync shines: Field-level authorization

AppSync’s authorization is underrated. You get four modes (API Key, IAM, Cognito, OIDC) that can be applied per-field:

type User {
  id: ID!
  email: String @aws_auth(cognito_groups: [”Users”])
  salary: Float @aws_auth(cognito_groups: [”Admins”])
}

Same type, different field-level permissions. With API Gateway, you’d implement this in every Lambda resolver manually. AppSync supports multiple auth modes per API natively. Some fields use Cognito, others use IAM, others use API keys — all in one schema.

The limitation: Authorization is tied to Cognito groups or IAM policies. Custom RBAC (permissions stored in your DB) still needs Lambda resolver logic.

⛔ When AppSync disappoints: The VTL reality

AppSync promises “no Lambda needed” via VTL (Velocity Template Language) resolvers:

{
  “version”: “2017-02-28”,
  “operation”: “Query”,
  “query”: {
    “expression”: “PK = :pk”,
    “expressionValues”: {
      “:pk”: $util.dynamodb.toDynamoDBJson($ctx.args.userId)
    }
  }
}

This works for simple CRUD. But add:

• Business logic (discounts, access control)

• External API calls with retries

• Complex data transformations

• Any if/else beyond basics

You’re writing Lambda anyway. Now you maintain VTL templates AND Lambda code.

Reality check: In every production AppSync API I’ve observed, 60%+ of resolvers use Lambda. The “no Lambda” promise rarely survives contact with real requirements.

Update (late 2022): AWS introduced APPSYNC_JS runtime as a VTL replacement. You can now write resolvers in JavaScript. It’s still not full Node.js. No npm packages, limited standard library. Async/await works for AppSync operations, but external HTTP calls need Lambda anyway. For complex business logic, you’re still reaching for Lambda. But for data access patterns, APPSYNC_JS is a significant improvement over VTL.

🎁Production gotchas:

• Pipeline resolvers: use these to chain multiple logic steps (e.g., Auth → Validation → DB call) without repeating code. They turn fragmented VTL/JS into a reusable sequence. Powerful but under-documented.

• HTTP resolvers: allow AppSync to act as a GraphQL Gateway. You can proxy existing REST or even external GraphQL APIs directly, merging legacy services into your new graph.

• Caching: AppSync now offers two layers. Resolver-level caching is the default (5 resolvers check cache 5 times), but Operation-level caching (new in 2025) finally allows you to cache the entire query response natively.

• 256KB request limit: a hard limit for the total request (payload + headers). While API Gateway handles 10MB, AppSync requires you to offload large data or file uploads to S3 using pre-signed URLs.

• VTL has no debugger: Unlike API Gateway, there is no true local runtime. You can use basic simulators like Amplify Mock, but for complex logic, you are stuck with $util.log() and CloudWatch. Local testing remains the biggest developer experience bottleneck.

Thank you for reading, let’s chat 💬

💬 Have you tried AppSync in production?
💬 Do you use APPSYNC_JS or still stuck with VTL?
💬 What’s your biggest pain point with GraphQL?

I love hearing from readers 🫶🏻 Please feel free to drop comments, questions, and opinions below👇🏻

Leave a comment

The Trap: The Lambda orchestration anti-pattern

Developers frequently fall into this trap because writing sequential logic in code is fast and familiar. However, this approach turns your business flow into fragile “spaghetti code” that is impossible to maintain and costly to run.

The three primary failure points of using Lambda for orchestration:

• The 15-Minute Timeout Wall: Your Lambda execution is capped at 15 minutes. If your workflow involves waiting for an external API, a human approval, or a long-running process, the function will simply time out and fail the entire process. No graceful pause. No resume. Just failure.

• Manual State Management: Lambda is stateless by design. To coordinate steps, you are forced to write and maintain complex custom logic that updates external state (e.g., DynamoDB) after every step. You’re essentially building your own orchestration engine from scratch — poorly.

• Expensive Retries: You must code error handling and retry logic into your function. When a service fails, you pay for your code to run the error logic, instead of having the orchestration layer handle it almost for free. Worse, if your Lambda times out mid-retry, you’ve lost all context about where you were in the process.

The Architect’s choice: Step Functions

AWS Step Functions is the dedicated, stateful orchestrator. It solves all of the above problems by defining your workflow declaratively using the Amazon States Language (ASL).

• Year-Long Workflows: Standard workflows can run for up to one year. You pay for the compute (Lambda) only when a task is executing. Waiting for days or weeks for human approval is cost-effective, unlike an idle Lambda that would timeout long before anyone even opens the approval request.

• Built-in Resilience: Error retries, exponential backoff, complex branching, and fallbacks are defined visually in the state machine. This logic is robust, versionable, and you don’t write a single line of code for it. When a downstream service fails, Step Functions automatically retries with configurable intervals — no billable Lambda invocations during the wait.

• Clear Observability: The visual workflow console shows you the precise status of every step in a clear graph, making debugging a failure instantly obvious. Instead of searching through scattered CloudWatch logs trying to reconstruct what happened, you see exactly which state failed, what input it received, and what error it threw.

The Solution: What to do about it

1. Map your current workflows: Identify Lambda functions that contain more than one business logic step. These are your refactoring candidates.

2. Start with the long-running ones: If you have any workflow that needs to wait more than 5 minutes for anything—an API, a human, a batch job—move it to Step Functions immediately.

3. Use the .waitForTaskToken pattern: For human approvals or external dependencies, leverage Step Functions’ callback pattern. Your workflow pauses indefinitely (up to one year) without any compute cost, then resumes exactly where it left off when the external event completes.

4. Let Step Functions handle the boring parts: Error handling, retries, timeouts, and branching logic should live in your state machine definition, not scattered across Lambda code.

Lambda is exceptional at executing discrete tasks. Step Functions is exceptional at coordinating them. Use each for what it does best, and your workflows become more reliable, maintainable, and cheaper to run.

Thank you for reading, let’s chat 💬

💬 What’s the longest workflow you’re currently running in a single Lambda function?
💬 Have you ever lost workflow state due to a Lambda timeout?
💬 What’s stopping you from adopting Step Functions for orchestration?

I love hearing from readers 🫶🏻 Please feel free to drop comments, questions, and opinions below👇🏻

Leave a comment

The core problem: waiting is expensive

In a modern MLOps pipeline, we often reach a critical point—such as before deploying a model to production, or tagging a batch of data—where a human expert must review the outcome.

In a serverless environment, implementing this manual gate traditionally presents a dilemma:

1. The Polling anti-pattern: Use a Lambda function that polls a database (e.g., DynamoDB) every minute until the human updates the status. This is inefficient, complex, and burns unnecessary compute costs.

2. The Container anti-pattern: Keep a container (EC2 or Fargate) running 24/7 waiting for a state change. This defeats the purpose of serverless and leads to high, fixed costs.

The solution must be both durable (it can wait days or weeks without failing) and cost-effective (we don’t pay for idle compute).

The Step Functions solution: callback task

AWS Step Functions is the ideal orchestration tool for this, specifically through its Callback Task pattern using the .waitForTaskToken integration.

This pattern allows the State Machine to initiate an action (like sending a notification to a human), pause its execution, and wait indefinitely (up to one year) for a unique Task Token to be returned.

Process flow:

1. Trigger: The Step Function is initiated (e.g., by an S3 upload, scheduled Lambda, or commit event).

2. Pause: The State Machine reaches an AWS service integration configured with .waitForTaskToken. This service could be:

   • SNS/SQS: Send a message to a human-facing queue or topic.

   • Lambda: Invoke a Lambda that creates a message with the Task Token.

3. Human Action: The unique Task Token is passed to your external interface (e.g., a simple Slack message). The human reviews the data or model output.

4. Resume: When the human hits “Approve” or “Reject,” your external service calls the Step Functions API using one of two methods, including the original Task Token:

   • SendTaskSuccess: Workflow proceeds to the next step.

   • SendTaskFailure: Workflow terminates or enters an error handling state.

During the entire wait time, you are only charged for the State Transition (the start of the waiting state) and the duration of the wait in seconds, which is significantly cheaper than hourly compute.

Implementation Details

In your Amazon States Language (ASL) definition, you can add it with the following configuration:

 “WaitApproval” : {
        “Type”     : “Task”,
        “Resource” : “arn:aws:states:::lambda:invoke.waitForTaskToken”
        “Parameters” : {
           “Parameter_1”: "value",
           “Payload”: {
              “token.$”: “$$.Task.Token”
     }
   }
 }

By leveraging the Task Token, you can get a durable, auditable, and inherently cost-optimized serverless workflow component for the MLOps process. A full example of such an approval step can be found in the MLOps Serverless pipeline project.

🎁Add SLA to your approval flow

Add a TimeoutSeconds parameter directly to your callback task. When a timeout occurs, Step Functions throws a States.Timeout error. Use a Catch block on that error to redirect to an escalation state — send SNS to backup reviewers, or trigger a Lambda that sends a reminder or escalates to a team leader.

This ensures stalled approvals don’t become black holes. The timeout becomes your SLA enforcement mechanism, automatically escalating when humans don’t respond within your defined window. It also reduces charges for the State Transition.

Thank you for reading, let’s chat 💬

💬 Beyond Step Functions, what is your preferred orchestrator for MLOps pipelines, and why?
💬 What is your solution for implementing an approval step in the pipeline?
💬 What is your best tip for creating a reliable MLOps pipeline?

I love hearing from readers 🫶🏻 Please feel free to drop comments, questions, and opinions below👇🏻

Leave a comment

Important context: AWS’s post-incident report reveals this was caused by a race condition in DynamoDB’s DNS management system and cascading operational failures—bugs AWS is now fixing, not inherent architectural limitations. However, the incident still teaches us important lessons about resilience, dependencies, and matching architecture to business requirements.

This incident prompted many teams to reassess their resilience strategies. Some realized their Multi-Region setup was overkill for their actual requirements. Others discovered gaps in their ability to handle regional issues. Where does your architecture fall?

Lesson 1: Multi-AZ protects your data plane, not your control plane

he incident demonstrated an important operational principle: while data planes can be multi-AZ, control planes are typically regional. During the incident, this meant:

• Running EC2 instances stayed healthy (data plane)

• New EC2 launches experienced failures (control plane)

• Existing containers kept running (data plane)

• New deployments faced intermittent issues (control plane)

This particular incident was caused by bugs — a race condition and cascading failures that AWS is fixing. But it reveals a broader reality: when regional services fail (whether due to bugs, misconfigurations, capacity issues, or other operational problems), Multi-AZ won’t help you scale.

What to do about it:

• Map your scaling dependencies: List every component that launches new resources during scaling—ASGs, ECS services, Lambda cold starts. These are your blind spots.

• Document your RTO honestly: If your RTO assumes you can scale during an incident, Multi-AZ won’t meet it. Write this down explicitly.

• Run a “no-launch” game day: Simulate control plane unavailability. Can your existing capacity survive peak traffic?

• Apply WAF REL10-BP04: For workloads with RTO < 4 hours, implement Multi-Region fault isolation.

Lesson 2: Match your architecture to your actual RTO requirements

Multi-AZ is designed for high availability within a region—and it works exceptionally well for that purpose. During this incident, Multi-AZ deployments kept existing workloads running without data loss.

The challenge isn’t that Multi-AZ is insufficient — it’s understanding when it’s enough. Multi-AZ protects against AZ failures and keeps running infrastructure healthy. When regional issues occur (whether architectural limitations or operational failures), you need different strategies.

What to do about it:

• Define real business impact: Ask stakeholders: “What does 6 hours of downtime actually cost us?” This drives your architecture, not preferences.

• Calculate Multi-Region costs: Multiply your AWS bill by 2.5x. Can the business justify that for rare regional failures?

• Choose your strategy based on requirements:

  • RTO > 4 hours: Multi-AZ + 20-30% excess capacity

  • RTO 1-4 hours: Multi-AZ + warm standby

  • RTO < 1 hour: Multi-Region active-active

• Over-provision capacity: If staying single-region, run 20-30% excess capacity so you don’t need to scale during incidents. This only works if your existing instances stay healthy and your traffic stays predictable.

Lesson 3: Practice Multi-Region failover before you need it

Global Tables performed well during this incident: customers with replicas in healthy regions could continue reading and writing data. Most replicas caught up within 2.5 hours of the incident start.

With asynchronous replication, there’s always some replication lag during incidents (WAF REL09-BP02). This isn’t a limitation unique to Global Tables—it’s fundamental to distributed systems. Synchronous replication across regions would add significant latency to every write operation.

For most applications, accepting temporary divergence during rare regional incidents is a reasonable trade-off for the performance and availability benefits. But having Multi-Region infrastructure deployed doesn’t mean you’re ready to use it.

What to do about it:

• Practice failover quarterly: Actually fail over to your secondary region and time it. Don’t discover issues during a real incident.

• Handle conflicts in your app logic: Identify tables with potential write conflicts and implement application-level resolution.

• Configure Route 53 failover properly: Use health checks that test actual endpoints, not just instance health (WAF REL13-BP01).

• Test your runbook: If you can’t execute failover in under 30 minutes following your documented steps, simplify the process.

• Monitor replication lag: Set CloudWatch alarms for lag > 5 minutes. Catch problems early.

Thank you for reading, let’s chat 💬

💬 Which of your mission-critical systems is still running Multi-AZ only?
💬 What’s your current RTO for your core database?
💬 Have you ever had a regional control plane failure?

I love hearing from readers 🫶🏻 Please feel free to drop comments, questions, and opinions below👇🏻

Leave a comment

What makes RDS different? Unlike on-premise setups, you can’t tune OS-level parameters, adjust I/O schedulers, or control storage directly. RDS instances use EBS-backed network storage, where performance depends on instance type, EBS bandwidth allocation, and OS page cache behavior. This fundamentally changes how you should approach performance tuning — many traditional rules of thumb simply don’t apply.

Myth 1: Keep the default value for work_mem

The default 4MB for working memory is a legacy minimum, not an optimal configuration for production.

The Problem: When a query operation (like a sort or hash) exceeds 4MB, PostgreSQL performs an expensive disk spill, writing temporary files to storage. This I/O bottleneck drastically slows queries. On a modern server with ample RAM, this is pure resource underutilization.

The Fix: Tune for memory

• Increase work_mem to a moderate, production-appropriate value (e.g., 16MB−64MB) via a custom DB Parameter Group.

• Monitor spills: Set log_temp_files to a low value (e.g., 1MB) to identify and correct the specific queries causing I/O bottlenecks.

• Balance carefully: Higher work_mem improves performance, but multiply it by your connection count to ensure you don’t exceed available RAM.

• ⚠️Critical caveat: work_mem is per-operation, per-connection. A query with 3 hash joins on one connection uses 3× work_mem. With 100 connections averaging 2 operations each, work_mem=64MB could consume 12.8GB RAM. Formula: work_mem × operations × connections.

Myth 2: Push all logic into functions and triggers

Advocating for a “fat database” architecture creates immediate scaling and maintenance challenges in cloud environments.

The Problem: Placing core business logic (e.g., order processing) in the database layer creates a monolithic bottleneck. This sacrifices the horizontal scaling capabilities of your application servers and impedes modern CI/CD pipelines, making versioning and testing complex.

The Fix: Separation of concerns

• Business logic: Belongs entirely in the application layer for easier unit testing and faster iteration.

• Database logic: Should be strictly limited to maintaining data integrity (Foreign Keys, CHECK constraints, and simple, non-blocking auditing via triggers).

Myth 3: Always use UNION vs. UNION ALL

The belief that UNION is the standard relational operator ignores its significant performance overhead.

The Problem: The UNION operator performs an implicit deduplication step (typically using hash aggregation) to ensure all rows in the combined result set are unique. While this doesn’t always involve sorting, it still adds significant computational overhead. The UNION ALL operator simply concatenates the two result sets without deduplication overhead.

The Fix: Prioritize performance

• Prefer UNION ALL unless the explicit removal of duplicate rows is a business requirement.

• The performance difference, especially on large result sets or within recursive queries, can be orders of magnitude.

Myth 4: Trust default autovacuum settings

Relying on AWS defaults for autovacuum is insufficient for databases with high write/transaction volume.

The Problem: The conservative default thresholds often allow table and index bloat to accumulate. This degrades query performance. More critically, they increase the risk of Transaction ID (XID) wraparound, which forces an emergency vacuum that can stall the entire database instance.

The Fix: Tune for velocity

• Decrease the autovacuum_vacuum_scale_factor (e.g., from 0.2 to 0.05) on high-write tables to trigger vacuuming sooner.

• Significantly increase autovacuum_work_mem (e.g., 128MB or more) to enable vacuum workers to process indexes in memory, preventing bloat efficiently.

• Consider increasing autovacuum_max_workers (default is 3 on RDS) for larger databases with many active tables — this allows parallel vacuum operations across multiple tables. However, this setting is capped by available CPU cores — on smaller instances, increasing workers won’t help and may cause contention.

Thank you for reading, let’s chat 💬

💬 Do you still use any triggers or stored functions in your application logic? If so, what specific use case justified the trade-off?
💬 Beyond the four points above, what is the single biggest “bad tip” you see when using RDS PostgreSQL?
💬 What is the single hardest habit from on-premise or local database development that you had to break when moving your PostgreSQL workload to a fully managed service like AWS RDS?

I love hearing from readers 🫶🏻 Please feel free to drop comments, questions, and opinions below👇🏻

Leave a comment