Software Architecture Weekly is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

Subscribe now

Highlights

The first highlight is the content piece of myself on how Supplied transitioned from the pure Serverless to the more traditional setup with containers in ECS and PostgreSQL instead of DynamoDB. See the full journey inside.

#casestudy

DNS Cache Memory Optimization 👷‍♂️

Whoever tells you data structures don’t matter in the age of AI - ban them. Cloudflare manages to save hundreds of terabytes of memory across the glove doing small optimizations to the ways DNS cache records are stored with the knowledge of how Rust store data and dropping dns owners when they are identical. Brilliant engineering work.

#dns #performance

Efficient Software Factory at Uber 👷‍♂️

While everyone talks about making software factories, Uber actually does it. They dropped a banger of an article explaining the 4 levels of AI adoption in the company, cost structure, measurement approach and of course the results. Great job!

#ai #casestudy

Follow-Up

DuckDB v2 - Distributed 👷‍♂️

At the core of this release is a native client/server mode powered by the quack protocol extension and the new CONNECT SQL statement. Rather than requiring developers to embed the engine directly or rely on verbose query wrappers, any DuckDB instance can now run as a daemon and accept connections over the network. Users can attach a remote DuckDB endpoint or external relational engines like PostgreSQL and MySQL, routing queries directly with pushdown optimisations.

#db

Solving 1+N problem with Acadia 👷‍♂️

1+N problem(or better known as N+1 problem) as when you need to query an object with 1 query and then make a query for every subobject(because of Joins). There are different ways to help with it. Acadia is a new programming language and application platform for building database-backed web applications, created by Evan Czaplicki, the creator of Elm. And Acadia solves it by preventing the very code that could possibly cause the said problem. Details inside.

#orm

Bucket Scanner to Context Engine 👷‍♂️

AI is no cryptocurrencies: you can imagine a dozen of valid use-cases thinking for 5 minutes. One of them is security scanning to classify the files for containing sensitive information and raising awareness. This is exactly what Wiz did for Data Protection Management. See how simple bucket scanner evolved into a security context engine.

#ai #security

Local models will not win 🍼

I tried to see if local models are viable for serious development. Spoiler: they are not. Sean Goedecke agrees: everybody tends to choose the most powerful model in their price range, and only datacenters have the corresponding capacity. Plus, they are well optimized for it. So local models will just not happen, forget it.

#ai

Unified AI Agent architecture 👷‍♂️

Embeddings for the AI agents typically require a separate database, while production data sits somewhere else. But if your data is in DynamoDB you have an opportunity to unify the data storage for both production and embeddings data. AWS shares the architecture blueprint to build AI agents with Bedrock and DynamoDB.

#casestudy #aiagents

Big thanks to Nikita, Constantin, Anatoly, Oleksandr, Dima, Pavel B, Pavel, Robert, Roman, Iyri, Andrey, Lidia, Vladimir, August, Roman, Egor, Roman, Evgeniy, Nadia, Daria, Dzmitry, Mikhail, Nikita, Dmytro, Denis and Mikhail for supporting the newsletter on Patreon! Alternatively, you can upgrade to the paid version and get access to the premium only posts!

I founded Supplied AI 2 years ago using only Serverless components: Lambda as a backend, DynamoDB for the data storage, S3 as an object storage. SQS came naturally a year later. Frankly speaking any stack works when you have one and a half clients with a very limited set of expectations. As we crossed 200,000 sellers data with 50+ clients while handful of them are mature enterprises, the requirements for scalability, performance and functionality shot up almost overnight.

In this article I will elaborate on:

• Our initial software architecture

• The problems we faced during accommodating requirements of the new clients

• A glance on the migration process

• Our current architecture.

Let’s go!

Initial Approach

At the point I co-founded Supplied I had plenty of experience running Java applications in webservers like Tomcat and WebSphere, overseeing the development of Go microservices on Kubernetes and doing some outsourcing staff with Lambdas on AWS. And one thing I knew for sure: premature complexity kills mercilessly. So I asked myself:

And boy, the answer was not Kubernetes or WebSphere on EC2. It was indeed Serverless. Here’s my initial stack:

• A single Lambda for the backend with a Functional URL

• DynamoDB with a handful of collections for data storage like customers and sellers

• S3 for the document storage.

• Web app was a React application server right from another S3 bucket.

And guess what? It was great! I could click through the entire infra in 10 minutes. Backend deployment? A shell script using aws cli. Web app deployment? A shell script using aws cli to zip the build folder and upload to the S3. That’s it! Simplicity is the king.

Once I needed to add a new, completely distinct functionality, the serverless way advised to just throw more isolated lambdas. Needed to upload files? Just make a Lambda! Need to generate reports? Separate lambda. Need to make PDF files for the sellers? You guessed. A separate lambda.

And it worked well - until it didn’t.

First caveats

The clients came and the thing which began as a side project started to prove the market demand. And all hell broke loose. Here are the problems we had to face.

Complexity

Managing infrastructure and dependencies between increasing number of components rendered increasingly difficult. Why?

Infrastructure

Testing new features required a staging environment and with high number of moving parts it become cumbersome. We eventually solved by introducing Infrastructure-as-Code, which is covered in this article.

Data model synchronisation

Several lambdas were working with the same set of collections in DynamoDB forcing us to synchronise data models between them. It meant either copy-pasting classes between different services or introducing shared libraries. You could say that this is what Lambda Layers were built for, but with layers you face a new problem.

Deployment problems

More lambdas, more layers - more deployments, and deployments could destabilise production due to mistakes in synchronisation. Here’s what happens: You make changes in two services because the new functionality require both changes. It needs an appropriate order of deployment of layers, then lambdas, then updating versions of layers in those lambdas. What does guarantee this order? Nothing. Occasionally we would break this order and make troubles for ourselves.

Performance

Cold start timing

We faced the Lambda cold start problem.

Our admin interface would take 15 seconds to load the first time. In the situation where multiple clients used the admin all the time, that would not be a problem; unfortunately it was not and still is not the case for us. We made several improvements like moving to Docker containers to leverage AWS Lambda optimizations. It improved the situation by roughly 40%, but didn’t solve the problem completely. We wanted the instant start.

Tens of thousands seller per customer

The majority of our customers had 200 to 500 sellers which data we had to manage. Then we got the first one with almost 3000 sellers. And then came the client with more than 20 000 sellers. While processing this kind of data is easy in any environment, we also had to display them quickly in the admin panel and more importantly sort them by reporting completeness. Because as the Company Admin, you really want to see the sellers who require the most attention now. This is the point we saw that DynamoDB simply does not have in-database sorting! We would have to load all the sellers for the company and sort them in-memory which had significant CPU and memory footprint while making the customer wait every time. That was not acceptable.

Multi-collection transactions and locking

With introduction of subscription and payments we had to update the data consistently across several collections. We also had to introduce proper locking mechanism for the subscription status as Stripe sends over 4 events in non-guaranteed order to our webhook. DynamoDB does not serve something useful for that too.

Functionality

Finally, with Lambdas you lose the interactivity: the statuses of the jobs do not update themselves - you need to make a poll on the frontend. And now with the introduction of Supplied Support AI Agent, you don’t get to stream the tokens because Lambdas are not built for it. In the end we ditched the lambdas all together.

Migration process

I’ve got plenty of experience migrating between different solutions in Bolt and this experience became very useful during our migrations.

Read more

Declaration: no AI is used in this newsletter.

Software Architecture Weekly is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

Subscribe now

Highlights

What it takes to achieve SOC2? 👷‍♂️

Recently Supplied AI achieved the SOC2 certification. It’s much harder than you would think: aside from technical security solutions, you need proper change management procedures, staff background checks and onboarding/offboarding procedures. Follow our journey in this post.

#security #certification

Faster Devs, Same Output

With AI agents developers became extremely productive, but do software companies ship more features or improve quality significantly? Barely. I am talking with Baruch Sadogursky, AI Enthusiast on the reasons behind same output and ways to improve it.

#video

Should You Split Into Microservices? 👷‍♂️

We recently merged all our microservices(we had a handful) in a monolith. If you consider moving in a back direction, you need to ask yourself at least 5 questions on dependencies, different NFRs for the system parts, teams blocking each other, data boundaries and independent failures.

#microservices

Follow-Up

Optimizing Prompt Cache 👷‍♂️

Prompt Caching is not mere saving the prompts. What you send to LLM consists of much more: System Instructions, Tools Descriptions, Conversation History and finally the prompt. explains what your agents send to the LLM on every turn and how to optimize it for the sake of performance and cost-efficiency. Double kil!!

Alex Ewerlöf Notes

Optimizing prompt cache

You probably know what prompt caching is (if not, there’s a quick intro) but what you may not know is the massive impact it has on cost per task (the token cost to do something…

Read more

24 days ago · 12 likes · 1 comment · Alex Ewerlöf

#ai #performance

AWS EC2 Application Status Check 👷‍♂️

After decades of custom monitoring solutions, AWS introduced the every minute status check with HTTP probes. Auto Scaling groups can replace unhealthy instances based on this application status. Try it out and tell me in the comments how it improved your life!

#ec2 #observability

MicroVMs for throw-away jobs

MicroVMs is a way to provide you a short-lived isolated execution environment. What are they best for? Right, security related tasks! How about running the virus scanning on docker images?

While I am personally skeptical on mere signature-check scans in general and ClamAV in particular, the idea of running high risk payloads in isolated envs is really appealling. Checkout how to leverage MicroVMs for it.

#security #aws

Writing code was never the point 🍼

Luca Mezzalira nails what we all think: When code is AI generated and disposable, what should we do?

However as always: abstraction level up, we now design agent harness and real systems instead of focusing on writing DTOs and generic controllers. Exciting times!

#ai

Meerkat: Cloudflare Consensus Solution 🤟

Following a good tradition finishing the newsletter with some kind of a paper, I am sharing an article from Cloudflare on their new Distributed Consensus Service called Meerkat based on the QuePaxa algorithm. Cloudflare aimed to replace leader&timeouts relying Raft with the multi-write nodes while claiming strong consistency. How do they manage that? Find the answers in the article.

#distributedsystems

Big thanks to Nikita, Constantin, Anatoly, Oleksandr, Dima, Pavel B, Pavel, Robert, Roman, Iyri, Andrey, Lidia, Vladimir, August, Roman, Egor, Roman, Evgeniy, Nadia, Daria, Dzmitry, Mikhail, Nikita, Dmytro, Denis and Mikhail for supporting the newsletter on Patreon! Alternatively, you can upgrade to the paid version and get access to the premium only posts!

The recipe behind certification

The B2B company can achieve a certification by implementing proper security measures, establishing the best practices and policies and gathering evidence. One of the artefacts is the Penetration Testing report, meaning the company needs to hire a partner which is able to perform it and make an appropriate report. Hiring a partner also helps with the checklist on all the practices, policies and evidences. There is no short-cut to a certification though, but hard work pays off.

Once every piece of evidence is gathered, the external auditor goes through them and generate a compliance report which we can share with the clients then. But why going through such a hassle in the first place?

Why SOC2 matters for a B2B tax platform

In B2C software, say a mobile application, the compliance required converges on a privacy policy, terms of service and age assessment. The corporate clients can’t be the more opposite than that, especially for Supplied AI, where we work with Personal Identification Information(PII), tax IDs, bank account numbers, and identity documents for a DAC7 and 1099K reporting product. Our customers are platforms and marketplaces. Those customers face their own regulators. Before they sign a contract with us, their security teams need proof that we protect this data correctly. And even then the process takes weeks and months. And while the certificate proofs formal compliance, it does not close the deal by itself - the actual controls behind the certificate do.

Before I became a CTO at Supplied, I got a taste of certification during my time in Bolt while leading Billing team in the Commerce Department. I had to fill some forms and questionnaire, but boy, did I not realize the whole scale of getting the whole company certified.

This article walks through the main control areas that SOC2 and ISO27001 check. Each section names the control area, then shows the real system behind it.

How the assessment happens

Your verification partner runs a software solution which solve several issues during certification:

1. Managing certifications

2. Finding gaps

3. Gathering evidence

4. Manage reviews&approvals

5. Providing TODO lists

6. Handling evidence life-cycles

However the biggest advantage of such a platform is the cloud integration: as we run AWS, our certification partner was granted the read-only access to our AWS organisations so the software could scan it and find the gaps. Imagine doing the same with on-prem resources ;)

But what the scanning is looking for?

Access control

First area of interest of the certification is access control: how our customers access their data and how this data is separated from other customers. Essentially, everyone wants to know that one client does not see the data of the other ones. The system can’t just rely on the 3rd party partner for authorisation and call it a day. Here’s how we do it at Supplied.

Supplied works with types of end users: suppliers(sellers) - think drivers, couriers, goods sellers, content creators; and administrators - members of staff of the marketplaces the suppliers operate on. We split those groups with audiences in Auth0: different applications, different keys, completely different permission models.

Admin-only endpoints carry an extra guard. The guard checks for the admin authentication strategy by name. A supplier token cannot reach these routes, even by mistake due to audience separation.

However separating people by a role is not enough; the admins should only have access to their own data, and nothing else. That’s why every request in the admin panel is scoped, so every database query for company data carries this scope. The system reads a company ID from the request context and applies it to the query. One company cannot read data that belongs to another company, as the query uses the company id from the authorization, not request parameters.

Other problem is the malicious or accidentally incorrect input. We validate everything with built-in Nest.js mechanism, so input validation runs on every write automatically. The validation pipe keeps a list of allowed fields and rejects any request with fields outside that list. This stops a common class of data-tampering attempts before the request reaches business logic.

Encryption and key management

While access control secures the data within legitimate users, encryption is what makes the data safe against everybody else. In case of traffic interception, or database leakage any data should remain useless to the attacker. Here’s the set of security measures we take.

Our customers data is of outmost priority. Data at rest uses AES-256 across S3, RDS, and DynamoDB. Data in transit - the traffic between the frontend and backend - uses TLS. The database proxy requires TLS on every connection. The developers can not access the database without a VPN tunnel, which uses mutual TLS with its own certificate authority.

However encryption is not a silver bullet, but rather one of the many measures we leverage. The web applications(and we run 2 in production) are a subject to multiple different techniques like XSS, protocol downgrade attacks, and many others. Fighting them means sending a whole set of HTTP headers so that the customer’s browser can prevent undesired behaviour.

We add security headers in layers. The CDN edge sets a Content Security Policy, a Referrer Policy, a Strict-Transport-Security header, an X-Content-Type-Options header, and an X-Frame-Options header.

The application layer sets its own Content Security Policy too, with a per-request nonce. Two layers add a small amount of duplication. Two layers also mean that one misconfigured layer does not remove the protection completely.

Last, but not least: the access to the data storage and downstream systems. meaning API keys, access tokens, connection strings etc. They should be never written anywhere in a plain text: how reckless would be storing these secrets in files or environment variables! Instead, we encrypt secrets with AWS KMS before any service uses them.

Change management and the software development lifecycle

The main principle of the secure change management:

Essentially, you need to ask a question “Why this change happened?” and “Who made it?”. That’s why there are no changes without a link to an issue in our Task Tracker system(we use Linear). But having tickets does not do the trick: the pipeline delivering changes is required too.

What to expect from a pipeline? A good one is:

• Predictable. Run on the same version, get same result

• Secure. Avoid leaks, prevent supply chain attacks.

• Helpful. Catch quality problems before it reaches any environment.

To grant aforementioned properties, we configured out delivery pipeline in the following manner.

Every deploy pipeline uses short-lived cloud credentials. GitHub Actions requests a token through OpenID Connect at run time. No long-lived AWS key sits in any repository secret.

Every pull request runs through a merge gate. The gate runs lint checks and a full test suite. The integration tests run against a set of emulated storages we use.

Every container image gets scanned on push, before it reaches any environment. A vulnerable base image gets flagged at build time, not after deployment.

We have 2 environments: staging and production. The tickets reach staging and being thoroughly tested. Only then the changes are deployed to prod.

Monitoring and incident response

Once the code is on the production, the working phase begins. At this point SOC2 asks a direct question: when something breaks, how does the team find out? What happens next? We answer with the set of monitoring tactics alongside measures applied for the identified gaps.

Our monitoring stack is classic, if not boring. CloudWatch for logs and alerts, Sentry for crash reporting, CloudTrail for audit. The team is monitoring the email alerts and reacts to urgent one ASAP; we configured Sentry for the Linear integration so independent of urgency we make a ticket in Linear right away to follow our change management policies.

But logic fixes is one story; real incidents is another. We had an incident when after one of the migrations we completely broke the Supplier App. After getting the indication from Sentry we indeed delivered a hotfix, but the interesting part is indeed in the Post Mortem.

We follow the best practices regarding learning from our own mistakes: The incident owner filled in the Post Mortem template, and then we hold a meeting to analyze it, make the remediation measure to later implement them.

Vendor and third-party risk

Customers data does not flow only through our applications and data storages, it is processed by external downstream systems. The formal term for them is “Subprocessors”. Data Protection standards and legislations demand at least a disclosure on these lists; the ideal situation includes minimisation of data processed alongside data locality. Meaning? Don’t transfer data oversee, but if you do - make sure there is no sensitive info transferred.

Supplied connects to more than fifteen outside services. These include payment processors, crash reporting software, an email provider, tax authorities in multiple countries, identity verification providers, and indeed hosted LLMs for AI payloads.

Here are our principles:

1. Use EU data residency to the maximum. Auth0 leverages EU tenants for both staging and production, EU-based models in AWS Bedrock, compute and storage resources are located in EU regions too.

2. Use minimum amount of information, avoid PII. Crash reporting software for example anonymise everything, customer emails are not included replaced by non-PII user ids, all the text on crash-associated screen recording is masked.

3. Be explicit about your subprocessors.

Not to mention, the clients ask about subprocessors and data locality every second time, so having this in order is essential.

Policies and Trainings

Reading the article so far you could have thought that certification is about technical measures, it is definitely not true. The organization should adopt several dozens of policies on hiring(onboarding and offboarding), access controls, communication, security trainings, backups, data retention and other aspects of software business.

Templates help to draft the policies, but real work is filling them in and proving they are actually in place instead of dead documents.

Consider the policy of running background checks: the organization should provide the proof, that criminal records were actually checked for the employees, their education suits the role and the employees have the skills corresponding to their roles.

The same affects the board meeting policies: it’s not sufficient to state the board should gather at least once a months; a proof of scheduled meetings and written meeting minutes are required. Thanks tech gods, we have automated summary nowadays.

Final Steps

Once the security measures are complete, evidence gathered and the pen-test conducted(with the gaps closed), it’s time to get an external auditor going through evidence. In our case our certification partner ordered this audit on our behalf and shared all the findings. Then you just wait for 2-3 weeks and get your SOC2 report.

I hope you can pass your certification as well. If you need a contact of the certification partner, leave a comment down below, I will refer your company.

Our Offering

Running a digital marketplace? Struggling with supplier onboarding and DAC7 reporting? Nervous about safety of the sellers data? Checkout Supplied AI - compliance infrastructure for Digital Marketplace. Get compliant in no time!

The limits on Claude Code, Codex and Antigravity keep shrinking with more and more powerful models consuming weekly limits in 3 prompts. The rise in prices for inference is imminent as the unit economy just does not converge. I decided to see if I can run a viable alternative on my own hardware with open-source models. See what I managed to achieve in this post filled with nothing but personal experience.

The First Purchase Attempt

My younger self was exciting of building everything from scratch; the grown up man looks for efficiency. The first idea I came up was to buy a Mac Studio with 64GB of RAM. This is a custom build - a subject to ordering and waiting for a couple of months. By the date of delivery I dropped the dealer a call and figured out they forgot to order it, offered no apology and suggested to wait for “5-6 months”. Apparently, the price for the memory spiked again and they were forcing me to take the money back and disappear, which I did. I ordered an AMD based PC instead which arrived in 3 weeks. Check out the LLM Machine Unboxing. The whole setup ended up with a cost of €4250.

The Machine

I nicknamed it beast, because it is supposed to be an Inference Beast. Under the hood: an AMD Ryzen 9 9950X3D, an AMD Radeon AI PRO R9700 with 32GB of VRAM, running Arch Linux (Omarchy). The 32GB card is the whole story here — it's the single number that constrains almost every decision in this post. Everything I did for the next week was, in one way or another, a negotiation with those 32 gigabytes.

For the inference engine I landed on llama.cpp's llama-server, running as a systemd service, GPU-accelerated through Vulkan. Multiple guides across the Internet recommends AMD's own ROCm/HIP stack to run AI workloads on this hardware, but after couple of experiments I found Vulkan backend (specifically the RADV driver) 2 times more performant on this card.

You'll Need A Bridge

Claude Code speaks Anthropic's Messages API. llama-server speaks an OpenAI-compatible chat-completions API. These are not the same shape of JSON, and nothing translates between them for free.

So I generated a translator: a small Node.js proxy that sits between Claude Code and llama-server, translating requests and responses in both directions - streaming Server-Sent Events, tool-call and tool-result mapping, the works. It runs as its own systemd service.

This proxy turned out to be the source of some of the real bugs. More on that shortly.

UPD: llama-server supports Anthropic API.

A Crash Course in Quantization

The Large Language Models are indeed large: they are so big your generic 5090 won’t be able to host a performant one in memory. The solution? Let’s compact the parameters of the said LLM into smaller bit size, so that instead of 16 bit per parameter we would have 3, 4 or 8. That’s call quantization.

For a 35-billion-parameter model, it takes roughly 70GB just to store the numbers. My card has 32GB total. The model simply does not fit into VRAM. And if it does not fit into VRAM, it needs to be offloaded to RAM and eventually to, god forbid, SSD.

Quantization is the fix: you round those 16-bit numbers down to something smaller - 8 bits, 6 bits, 4 bits - and store that instead. You lose a little precision. In exchange, the model shrinks dramatically and runs faster, because there's simply less data to move around. The whole game is figuring out how much precision you can throw away before the model gets noticeably dumber.

The naming scheme on the files you'll download (Q4_K_M, Q5_K_M, Q6_K) decodes like this:

• The number (`Q4`, `Q5`, `Q6`) is roughly how many bits are used per weight, on average.

• `K` means "k-quant" — a smarter scheme that doesn't quantize every layer equally. Some layers matter more to output quality than others, so k-quants spend more bits where it counts and fewer where it doesn't.

• `M` (or `S`/`L`) is a size variant within that quant level — a further knob on the same precision-vs-size tradeoff.

There's also a newer variant I used, from Unsloth, called a Dynamic quant ("UD" in the filename). Instead of applying one blanket rule per layer type, it analyzes the actual model and picks precision per layer more surgically — the pitch is meaningfully better quality at the same file size, and in practice it's what I reached for by default.

The problem with compressing weight is losing intelligence. The smaller you go, the more the model's reasoning quality erodes. I was hoping the model would be slow, but smart enough to fix at least simple bugs, but I’ll get to what I got instead.

If you would like a visual guide for quantization, here’s the article which does a great job.

The Model Rotation

Over about a week I ran through four distinct configurations:

1. Qwen3-8B, Q4_K_M (~5GB). That was just a test out for the setup. It ran instantly and was, unsurprisingly, too small to do real engineering work.

2. Qwen3.6-35B-A3B, Q4_K_M (~22GB). This is a Mixture-of-Experts model — 35B parameters total, but only about 3B "active" per token, which is why a "35B" model runs at a speed you'd expect from something much smaller. This became my long-term baseline.

3. Ornith-1.0-35B, Q5_K_M (~25GB). Frankly I got this model recommended in one of the twitter threads. On paper, this should have been a straight upgrade.

4. Back to Qwen3.6-35B-A3B, this time at Q6_K (~29GB)** — after Ornith's benchmark scores turned out to have nothing to do with how it behaved in my actual harness. I'll get to why.

That last jump, from Q4KM to Q6K, is a good illustration of the VRAM negotiation: at Q6K the model weights alone occupy 29.3GB out of 32GB. That leaves a sliver of the card for everything else inference needs (the KV cache, i.e. the model's working memory for the current conversation) - a constraint that comes back to bite me later in this post.

Building the Autonomous Pipeline

My ideal picture was to make a list of bugs and improvements in a task tracker(we use Linear), point the pipeline at it and wait for the pull requests to land. Here’s how I imagine the harness would work on the beast machine:

1. Reads tickets assigned to me in my ticket tracker.

2. Checks whether a ticket is actually ready to work (clear problem, acceptance criteria, no open questions) — and if not, asks a clarifying question as a comment and stops.

3. Drafts a design and posts it as a comment, then **stops and waits for approval.** This is a hard rule, not a suggestion — the pipeline is not allowed to talk itself into skipping this step, no matter how "obvious" the fix looks.

4. Once approved, implements the change in an isolated git worktree, verifies it (build/lint/tests), and opens a draft pull request.

5. Watches open PRs for review feedback — from a human, or from an automated reviewer — and reacts to it.

It should run unattended, once an hour, off a systemd timer. All of its progress is tracked in a small state file, so a run that gets interrupted can pick back up where it left off.

That's the design. Now here's where it actually fell over.

Where It Fell On Its Face

Ornith simply didn't show up. Despite the leaderboard scores, one triggered run against Ornith produced a clean, zero-error exit and a grand total of one line in the log: "I'm ready to help! What would you like to work on?" No ticket read, no tool call, nothing. The backend logs were completely healthy — this wasn't a crash, the model just never engaged with its own instructions. Rolling back to Qwen3.6-35B-A3B and re-running the identical trigger worked correctly on the first try. Benchmark leaderboards, it turns out, tell you almost nothing about whether a model will actually follow a long, detailed operating manual.

The silent empty-turn bug. This one was the scariest, because it produced no error at all. llama-server streams a model's reasoning as a separate field (reasoning_content), distinct from its actual visible output (content). My proxy only ever read the second one. So on a turn where the model burned its entire response on internal reasoning and produced zero real output, the proxy still emitted a syntactically valid, totally empty "the assistant is done" message. Claude Code has no way to tell "finished normally" apart from "finished having done nothing" — it just exits cleanly. My first real test run vanished this way: no commit, no comment, no state update, no error anywhere. I only caught it by noticing the ticket simply hadn't moved.

Read more

Software Architecture Weekly is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

Subscribe now

Highlights

The State of Streaming to Apache Iceberg in July 2026: Every Path, Its Latency, and What to Do When Seconds Are Not Fast Enough 👨‍💼

In mid-2026, teams no longer ask, “Should we use Iceberg?” They ask, “How current can our Iceberg tables be?”. And this is where the main tradeoff relies dictating your data architecture and tools to go with. From tuned Flink to Kafka connect latency numbers varies from 30 seconds to 15 minutes. How to choose? Well, Alex Merced explains in his piece.

#db #distributed #architecture #softwareengineering

Clustering Billions of Products for Agentic Commerce with Catalog API 🤓

Shopify Catalog groups billions of listings without a common schema. It first matches products inside each store, then connects them across stores with a Universal Product Identifier (UPI). LLMs assign a structured label to every product. This enables consistent grouping, high precision, and better recall. AI searches based on Catalog data convert twice as often as searches based on scraped data.

#ai #architecture #softwareengineering #engineering

On-prem DBaaS in 2026: Platforms, standards, and gaps 👨‍💼

On-premises database provisioning in 2026 splits into two camps that rarely interoperate: developer-operated models built on Kubernetes operators (CloudNativePG, Vitess) and platform-team models backed by Crossplane or internal IDP tooling that abstract the infra layer. The critical gap is the absence of any widely adopted standard connecting database lifecycle management to the internal developer portal — leaving teams to rebuild equivalent abstractions repeatedly with incompatible APIs and operational assumptions. Until a common binding layer emerges, the tradeoff is real: tight operator integration with manual self-service wiring, or platform abstraction that substitutes cloud lock-in with a home-grown one. #db #architecture #devops #cloud

Follow-Up

Running a self-hosted LLM in Kubernetes with vLLM 👨‍💼

With the rise of cost for the LLMs and the privacy concerns more and more enterprises opt to run local models(and I am experimenting with them myself). Grab a guy how to setup an open-source LLM with Kubernetes!

#llm #cloud #devops #architecture

Claude Code Sandbox for iPad 👨‍💼

On the other handside you might want to launch some coding or research jobs from your handheld device - like an iPhone or an iPad. Edjgeek offers a solution: a MicroVM spawing once you open a special web page, mounting a home directory from S3 and doing the work you need. Of course, you need to throw in some security measures too!

#cloud #ai

A Visual Guide to Quantization 🤟

The Large Language Models are indeed large: they are so big your generic 5090 won’t be able to host a performant one in memory. The solution? Let’s compact the parameters of the said LLM into smaller bit size, so that instead of 32 bit per parameter we would have 3, 4 or 8. That’s call quantization. How does that work? Well, grab a great visual guide!

#llm

Reduce Logging Costs 🍼

When your solution landscape grows, the amout of logs compounds into hundreds of gygabytes or terabytes monthly. At a certain scale it hits the cloud bill significantly and it makes sense to start optimizing the cost. The first 2 strategies are obvious: sampling and tiered storage, but there are 3 more which you will find in the article By Michael Shpilt.

#observability #cost #cloud

Clickhouse is winning the observability wars! 🍼

Observability 2.0 is about structured logs and you know what’s the ideal storage for them is? Clickhouse! You don’t need to trust myself, trust Charity Majors - the CTO of Honeycomb, an observability platform. Clickhouse is efficient, open-source, scalable storage and a hope for observability. Details by Charity inside.

charity.wtf

Have you heard? Clickhouse is winning the observability wars!

We interrupt our regularly scheduled series of posts on AI norms and values to bring you this incredible piece from Mat Duggan, “Why Clickhouse Is Winning the Observability Wars,” which you should go read right now. Go! I’ll wait…

Read more

2 months ago · 28 likes · 4 comments · Charity Majors

#observability

Big thanks to Nikita, Constantin, Anatoly, Oleksandr, Dima, Pavel B, Pavel, Robert, Roman, Iyri, Andrey, Lidia, Vladimir, August, Roman, Egor, Roman, Evgeniy, Nadia, Daria, Dzmitry, Mikhail, Nikita, Dmytro, Denis and Mikhail for supporting the newsletter on Patreon! Alternatively, you can upgrade to the paid version and get access to the premium only posts!

Why people attend conferences? Is it insightful talks? Maybe looking for a new job? Free snacks? The answer is yes - to all of it. But do people get the best value out of expensive tickets? Absolutely not. Here’s the guide on getting the most of any professional conference you attend. Save it for the next one.

I’ve been going to the conferences since the beginning of my career and promptly became a speaker as well. Fast forward several years, I was organizing technical conferences myself ending up as a content director of the biggest technical mobile conference called Mobius in Moscow a decade ago.

In this post I cover how conferences work, what are the strategies for using conferences for getting a promotion, changing a work place and expanding your network for a long game.

TL;DR: understand your goal of visiting the conferences, study the program, find relevant speakers and companies, network as much as possible.

How conferences work

The majority of the conferences including even small meetups work in the same mode. It requires 4 parts to be successful: organisers, speakers, sponsors and attendees.

The organisers obviously want to have to a financially successful event. The conference budget consists of sold tickets and sponsor’s packages. The sponsor packages include the booths, promotional materials, banners etc. The sponsors are looking to make an HR Brand mostly, showing how advanced they are technologically and how fun and beneficial it is to work for them. Companies rarely search for clients at tech conferences, mostly talent.

However the sponsors are only willing to pay the sponsorship if they are sure enough they can get some value from it. And the value comes from the attendees. So conference organisers need to attract the attendees to the conference to make the whole even work.

The strategy here is straightforward: invite well-known speakers with new, insightful talks. An ideal speaker should have a track of well-given talks previously, bring a fascinating topic and be able to deliver it in an engaging manner. There is a separate topic on how to invite speakers to your conference in the first place, but that’s off topic for today.

Anyway, here’s how the circle completes: the conference invites speakers, attendees come to listen to speakers, sponsors pay to get access to attendees.

The Attendees Strategies

Indeed you as an attendee may just buy a ticket, come to a venue and listen to a bunch of talks which happen on the conference, not to mentioning grabbing an included lunch and some free merch and call it a day. Anyway, your ticket is probably paid by the company, you also get a day or two off work, what else to desire?

But I argue that would be a waste of time, money and potential. The best value is extracted not out of talks.

For God’s Sake, the talks will be recorded - you can rewatch them any time later, although few actually does.

Here’s the flip: the talks are mere a quarter of value you should be getting out of the conference. But then what to do instead? Well, it depends on your short term and long term goals. So, first step is to understand why you attend this conference in the first place besides free snacks.

Accelerating the promotion

The careers are built with promotions followed by achievements. If you goal is to rise up the corporate ladder, you would really like to get some experience from people who already done this from the companies similar to yours or at least with the problems similar to the problems of your company. So your strategy should reaching out to the people who can help you with that. And the obvious tactic to pick in this strategy is doing your homework. Don’t read the conference program on the opening day. Do this instead.

Every conference will publish it’s schedule, it’s speaker line up and list of partners and sponsors. This is your primary input. Study it well: write down the companies which are similar to yours and find if there are speakers from there. If so - great! Mark the talks from those people in your agenda. Some conferences will provide you with an app where you can bookmarks talks; if there is none, make the records in your calendar. Make sure to attend those talks and listen to the talks carefully - omit scrolling YouTube shorts during the talk. Instead, make notes. Write down 3 to 5 questions about those talks, because when the speaker goes into Q&A, you want to become the first person to ask a question.

You probably wonder why it’s important: practice shows that the audience is not ready to ask questions right away and it creates an awkward silence. And man, the worst feeling of the speaker is when there are no questions after the talk. Save them, be their hero. Ask a question. Don’t know what to ask? Go with the templates like these:

• What would you do differently knowing those problems in advance?

• What was your biggest learning during solving this problem?

• What would you recommend to the developers in the similar situations?

The main point is to make the speaker feel good to prepare for your phase 2. What is phase 2? Well, after the talk the speaker stays for several minutes to answer the individual questions. This is your moment - you approach them and ask for the advice. Be explicit about your intention to learn from them and get to know how to get promoted. The best case scenario you will get to know the promotion mechanics. The worst case - well, you get a contact who could refer you in case you decide to change the workplace.

Talking about a good conference to attend, Packs runs a virtual architecture summit with Lead Engineers, Directors and Data Scientists from Amazon, Google, Splunk Netflix and other companies:

Software Architecture in the Age of AI

ARC: Software Architecture in the Age of AI is a two-day virtual summit for CTOs, software architects, engineering leaders, and senior engineers exploring how AI is reshaping modern software systems.

Hear from industry leaders including Dr. Ali Arsanjani (Director, Google Applied AI), David Ping (Head of Solutions Architecture, AWS), Juan Bustos (Lead GenAI Partner Engineer, Google), Chi Wang (Google DeepMind), and other experts from leading technology organizations.

Sessions will cover AI governance, agentic systems, reliable AI architectures, engineering leadership, inference at scale, and practical strategies for designing, deploying, and operating production-ready AI systems.

Whether you’re defining your organization’s AI strategy or modernizing existing software platforms, ARC offers practical insights from practitioners building enterprise AI today.

You can sign up using the following link: ARC 2026

Changing Jobs

Being promoted at your place is one story, looking for a new place is completely other. Good news is there is plenty of people for whom helping you is in their best interest.

Remember the sponsors of conferences? Their goal is to HIRE YOU. Literally the Talent Acquisition specialists will be on booths there gathering contacts. But again, you need to do your homework first.

Similar to previous point you should write down the companies that you intend to get an interview for. Once you’re at the venue, reach out to people from those companies -some will be probably at booths, others will be speakers, 3rd one will be the attendees. Try to talk to the maximum amount of people and make the most meaningful conversations with each of them. How to do that though?

Talking to Engineers: figure out the company culture. Find out how do they work on features, how they prioritize things, how the releases happen, if the post mortem are run. Now in the AI era, ask for the AI-approach. Is it token-maxxing or a sane strategy. Don’t forget to ask about the particular team, manager name, their style and overall mood at the team.

Talking to Talent Acquisition: figure out the hiring process. How many interviews before offer, how do they happen, who make the decision, how the promotions happen, how learning is managed in the company etc.

Try to make a good first impression, but don’t try hard. Do have a solid reason why you’re changing a workplace. Good reasons include lack of development, layoffs. Bad reasons? Don’t ever blame the management, nobody likes complainers.

Your end goal is to get to your first interview. Ideally you would like to get a referal - in such case you increase the chances to actually get into the process.

General Networking

Believe it or not, your career is a long-term game. You never now what you would like to do in a couple of years. So the idea here is to extend your perspective on the industry and find out what is happening around.

Use the talk breaks for wondering around and talking to people. I realize that software developers are typically shy and a bit afraid of initiating a conversation. Here’s an algorithm to stop worrying:

1. Grab a glass of water, but don’t have more than a few sips there

2. Approach a person you’re interested in(may be it’s a lead developer in an interesting company)

3. Tell them “Hello, I am <Your Name>, I am an engineer at <Company>. What do you do?”.

4. Now you have a conversation!

5. Once you want to end the conversation, end the water and say you need a refill

6. Repeat

Grab people’s contacts, add them to your contact list, mark them with notes why you though the person was interesting.

Some time later you might want to invite them to a meetup. Maybe you would ask for a referal. Who knows? Play the long game.

On Online Conferences

Online conferences are pretty common nowadays especially after the pandemic. Unfortunately you don’t get a physical presense, but that does not mean you can not network there too. What you’re looking for is the networking breaks. Some conferences call them Random coffees, where you’re randomly matched with a person, some have just breakout rooms, others have different formats. But make sure to attend those calls.

You would also probably get invited to Slack, Telegram or a platform where the online conference happens. Study what communication option the platform offers and follow the approaches mentioned above: DM people! But don’t DM people with just asks: offer something instead. Make a conversation interesting for them too. Offer to share your experience with the technology at your company, share an insight you got, offer to tell something valuable. Don’t make a conversation one-sided.

If you read up to this point, don’t forget to leave a like and share the post!

Thanks to Patrons!

Big thanks to Nikita, Constantin, Anatoly, Oleksandr, Dima, Pavel B, Pavel, Robert, Roman, Iyri, Andrey, Lidia, Vladimir, August, Roman, Egor, Roman, Evgeniy, Nadia, Daria, Dzmitry, Mikhail, Nikita, Dmytro, Denis and Mikhail for supporting the newsletter on Patreon! Alternatively, you can upgrade to the paid version and get access to the premium only posts!

Software Architecture Weekly is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

Subscribe now

Highlights

Introducing Durable Functions in PostgreSQL 🤓

Postgres will happily store your data, run your queries, and scale with you for years. But the moment you need to do more with that data, such as running multi-step transformation, scheduling nightly rollups, generating embeddings or waiting on an approval, you hit a wall. Postgres has no built-in way to run long-lived, fault-tolerant work. Microsoft's pg_durable makes PostgreSQL an orchestration engine: workflows are defined as graphs of SQL steps, each checkpointed after execution, so database crashes and restarts resume from the last completed step with no external state reconstruction. The architecture details inside!

#db #architecture #engineering #softwareengineering

Stragglers, Not Failures: How Adaptive Hedged Requests Reduce p99 Latency by 74 Percent 🤓

In fan-out microservice architectures, the dominant cause of high p99 latency is stragglers — slow-completing requests rather than failures — because one straggler in a fan-out blocks the entire composite response. While retries are a solution for failed requests, the stragglers require a parallel request if slow response is detected. Indeed new issues come with the solution, like write amplification, but if the goal is p99 optimized, raced requests are a way to go. The result is a 74% reduction in p99 latency with zero call-site configuration changes, with a reference implementation available as an open-source Go library.

#distributed #architecture #engineering #softwareengineering

The Inference Paradox: How Split-Brain LLMs Are Killing Your GPU ROI 🤓

LLM inference has a structural hardware mismatch: the prefill phase is compute-bound (processing all input tokens in a single forward pass) while decode is memory-bandwidth-bound (reading the full KV cache to emit one token per step), so coupling both phases on the same GPU means each permanently starves the other. Kubex's enterprise audits surface average GPU utilization near 5% — monolithic vLLM serializes all prefill before decode can continue, and under high concurrency the delay compounds across every request in the batch. Disaggregating prefill and decode onto separate hardware-optimized node pools — as llm-d (now a CNCF sandbox project) implements on Kubernetes with prefix-cache-aware routing — yields 2–3x throughput at high concurrency by keeping the decode pool continuously active while the prefill pool handles bursts in parallel.

#ai #llm #engineering #cloud

Scaling MCP Adoption: Cloudflare's Enterprise Reference Architecture 👨‍💼

MCP Servers bring a set of vulnerabilities like prompt injections and indeed supply chain attacks. Allowing the developers to make the MCP servers is a big security risk and Cloudflare tackes it at scale with MCP platform handled by a single platform team, which routes all inter-service calls through AI Gateway for per-user spend tracking and budget enforcement, and uses Cloudflare Access JWT as the OAuth provider. For platform teams standardizing on MCP at enterprise scale, this is a production-complete blueprint: remote hosting, cost controls, identity, audit logging, and progressive tool disclosure in one reference design.

#architecture #cloud #ai #engineering

Quick: An Internal Hosting Platform for the AI Era 👨‍💼

Shopify's Quick started as the simplest possible internal hosting: HTML files in a GCS bucket, a wildcard NGINX config mapping *.quick.shopify.io to folder paths via GCS Fuse, and Google Identity-Aware Proxy ensuring every request is a verified Shopify employee. In under a year it hosts 50,000+ sites used by more than half the company — AI code generation created demand for internal hosting that outpaced every alternative tool. The architecture grew only when forced: an LLM proxy with server-side API keys, optional CloudSQL+Node.js backends, BigQuery data connectors, and WebSocket support, each added when multiple teams independently needed it.

#engineering #devops #cloud #softwareengineering

AI Is Moving up the Software Lifecycle: from Code Review to PRD Governance 👨‍💼

Code review was the easy part — Uber, DoorDash, and Cloudflare are now pushing AI review further left, into PRDs and design docs, before a single line of code exists. Uber runs a first-pass AI filter on product requirement documents while keeping engineers as final validators, and DoorDash’s internal code reviewer is explicitly tuned for fewer, higher-signal comments after teams complained about noisy automated feedback. In Bolt a good product document was stumbling features for weeks so reviewing Product Documents should be a great feature. See Uber’s experience in the article.

#ai #softwareengineering #engineering #quality

Building Reliable Agentic AI Systems 👨‍💼

Production-grade agentic AI hinges on two disciplines: context engineering (routing the right information to the right stage, not stuffing everything into one prompt) and harness engineering (controlled LangGraph orchestration with defined pause points, retries, and state persistence). Bayer’s PRINCE platform uses a hybrid RAG pipeline — semantic search weighted 0.7, keyword search 0.3 — reranking 20 initial chunks down to top 7, with Text-to-SQL capped at 50 records and 3 auto-correction attempts. Complementary reflection loops (process, data, and draft) catch trajectory drift, information gaps, and output completeness failures respectively. Daily live-traffic evaluation via RAGAS metrics closes the production feedback loop.

#llm #architecture #ai #softwareengineering

The Cost YAGNI Was Never About 🍼

Developers think that YAGNI - You Ain’t Gonna Need It - was a warning against overengineering for the imaginary use cases. And while the cost of writing code was obvious, the cost of maintaining a complex structure is still hidden in the plane sight. However, LLM being able to generate code almost for free surfaces this problem: YAGNI is not so much about code writing, it’s about designing for a wrong thing. Don’t listen to me, listen to Kent Beck.

#principles

Big thanks to Nikita, Constantin, Anatoly, Oleksandr, Dima, Pavel B, Pavel, Robert, Roman, Iyri, Andrey, Lidia, Vladimir, August, Roman, Egor, Roman, Evgeniy, Nadia, Daria, Dzmitry, Mikhail, Nikita, Dmytro, Denis and Mikhail for supporting the newsletter on Patreon! Alternatively, you can upgrade to the paid version and get access to the premium only posts!

This article is a practical guide to choosing the right Stripe integration for read-only data access, understanding what building a Stripe App actually entails, and avoiding the specific traps we fell into.

Four Ways to Read Data from a Stripe Account

Before you write a line of code, you need to pick your integration model. There are four meaningfully different approaches, and they’re not interchangeable.

Option 1: User-Generated Restricted API Key

The merchant creates a restricted API key in their Stripe Dashboard (Developers → API keys → Create restricted key), selects the read-only permissions your app needs, and pastes it into your product.

When it works: Internal tooling, low merchant count, you control onboarding.

Why it breaks at scale: Fully manual. No programmatic rotation. If the key leaks, the merchant has to notice and revoke it themselves. For a public SaaS with hundreds of clients, the support burden alone makes it impractical.

Option 2: Stripe Connect (Platform Model)

In Stripe Connect, merchants are connected accounts on your platform. You process payments on their behalf and can read their data using your platform key with a Stripe-Account header.

When it works: You’re building a marketplace or platform that already processes payments through Stripe. The merchants signed up through you, not directly with Stripe.

Why it doesn’t work here: This is a deep structural commitment — merchants have to onboard through your Stripe platform, putting you in the payment flow. If your product is downstream of Stripe (you’re reading data it already holds, not facilitating payments), Connect is the wrong model. You’re acquiring platform-level authority when all you need is read access.

Option 3: Stripe Connect OAuth (Standalone)

Standard OAuth: you redirect the merchant to connect.stripe.com/oauth/authorize, they grant access, Stripe redirects back with a code, you exchange it for an access token scoped to that account.

When it works: You want programmatic, per-account read access, you’re not building a Stripe App, and you want a standard OAuth UX rooted in your own web app.

Why consider the App instead: The merchant has to leave their Stripe Dashboard to connect. If your customers live in Stripe daily, a Stripe App removes that friction. If most of your customers come from outside Stripe, standalone OAuth is simpler to build and requires no App Marketplace review. However this approach bears a significant downside: you can’t be found on the Marketplace either removing a marketing channel.

Option 4: Stripe App

Stripe Apps can embed your product inside the Stripe Dashboard. The merchant installs the app from the marketplace (or a direct link), and a panel appears in their sidebar - but not necessarily. Stripe Apps can leverage UI - this is exactly the panel we mentioned. But there might be a UI-less apps, which are backend-only. Anyway, there are two sub-types that matter for data access.

4a — App with Restricted Key (stripe_api_access_type: "restricted")

You declare permissions in stripe-app.json. Stripe automatically provisions a per-install restricted key with exactly those scopes. No OAuth token exchange, no refresh logic — you read the key from the app context and call Stripe’s API directly.

Simplest token model. The ceiling: restricted keys are scoped strictly to your declared permissions. If you need the token to outlast a session or call Stripe’s API from a scheduled backend job, the automatic key won’t work.

4b — App with Connect OAuth (stripe_api_access_type: "oauth")

During install, the merchant goes through the standard Stripe Connect OAuth flow. You get a full access token and refresh token per account — identical to Option 3, but the entry point is the Stripe App install rather than a button in your web app.

More token management work, but full flexibility. You store, refresh, and encrypt the token yourself. This is what we ended up building.

How to Choose

By data access need:

• Small number of accounts, internal tooling, willing to ask users to generate a key → restricted API key

• Already a Stripe Connect platform processing payments on behalf of merchants → Stripe Connect platform model

• Want OAuth-based access, most customers come from outside Stripe → standalone Connect OAuth

• Customers live in Stripe daily, install UX matters for conversion → Stripe App

If you’re building a Stripe App:

• Want simplest token model, no refresh logic, permissions declared upfront → App + restricted key

• Need long-lived tokens, schedule-based backend API calls, full flexibility → App + Connect OAuth

• Core workflow happens inside the Stripe Dashboard → UI Extension (visible panel)

• The app is a one-time connection step, your product lives elsewhere → backend-only (extensions: null)

If You Build a Stripe App: UI Extension vs. Backend-Only

A Stripe App can have a visible UI — a React panel rendered inside the Stripe Dashboard — or no UI at all, with only OAuth and webhooks.

UI Extension means your app renders inside Stripe’s iframe using their proprietary SDK (@stripe/ui-extension-sdk). The sandbox is strict: no cookies, no localStorage, no arbitrary external fetch calls. Requests to your backend must go through fetchStripeSignature(), which signs the request so you can verify it server-side. State doesn’t persist across page loads.

Backend-only means "extensions": null in stripe-app.json. The app exists only as an OAuth grant mechanism and webhook receiver. The merchant installs it, completes the OAuth flow, and returns to their normal Stripe workflow. Your product’s UI lives in your own web app.

We started with a UI Extension and ended with backend-only. The UI felt valuable — connection status, sync history, a disconnect button, all without leaving Stripe. But it was built on Auth0 login happening inside the sandboxed iframe, which required a popup, clean popup closure, and state surviving an OAuth round-trip inside a sandbox. Every piece of that is fragile. We deleted 282 lines of TypeScript on the same day we simplified the OAuth architecture.

When a UI Extension is worth it: The app’s core workflow happens in Stripe. The panel provides value while the user is working in the Dashboard, not just during setup.

When backend-only is the right call: The app is a data pipeline — Stripe is the source, your product is the destination. The install flow is a one-time event. Once connected, the user has no reason to return to Stripe to interact with your product.

What We Built and What Went Wrong

We’re a DAC7 tax reporting platform. Marketplace operators on Stripe must report seller earnings data to EU tax authorities annually. The Stripe App lets them connect their Stripe account to our reporting pipeline without leaving the Dashboard. We only needed to read — invoices, customers, payments, transfers — never write.

We ruled out:

• Restricted API key — implemented before the app, feels clunky and not exactly trustworthy

• Stripe Connect platform — we’re downstream of Stripe, not a payment processor

• Standalone Connect OAuth — our customers are Stripe-native; the App install UX matters

We chose the Stripe App with Connect OAuth. Here’s how the architecture evolved.

Read more

Software Architecture Weekly is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

Subscribe now

Highlights

How Discord Automates ScyllaDB Clusters at Scale 👷‍♂️

A small infrastructure team operating 20+ ScyllaDB clusters with nearly 500 nodes cannot afford purely manual runbooks for cluster-wide operations. Discord’s Scylla Control Plane (SCP) encodes operations — rolling OS upgrades, cluster expansion, shadow cluster provisioning, node recovery — as YAML-defined workflows with explicit retry counts, parallelism controls, and abort-on-failure semantics; idempotency is a hard requirement for every task so that any retry is safe. Shadow clusters — temporary production replicas receiving real traffic — let the team validate new ScyllaDB versions before they touch live data; SCP automates the full shadow cluster lifecycle, cutting what once required more than a day of continuous engineer attention to largely unattended runs.

#db #distributed #architecture #engineering

Zero Trust for AI Agents 🍼

Each new technology bears security risks and the agentic systems is no exception. Both common vulnerabilities like supply chain attacks and excessive permissions stay; new ones like direct and indirect prompt injection, RAG poisoning and others comes. Follow the Antropic guide on applying Zero Trust to your agentic systems.

#security #ai

How we built Cloudflare’s data platform and an AI agent on top of it 👨‍💼

An analyst working at Cloudflare and asking themselves “How many domains that signed up today are in the Top 100 by traffic?” finds it difficult to answer searching for various systems, consolidating data and preparing a query. Cloudflare built a data lakehouse entirely on Cloudflare’s own stack — Apache Trino as the query engine joins Postgres, ClickHouse, and Iceberg-on-R2 tables in a single SQL statement without intermediate materialization, with Workers handling compute and Workflows handling orchestration. Skipper, the AI agent sitting on top, routes through Anthropic Claude models treated as commodity APIs. Nice case study from Cloudflare!

#architecture #cloud #ai #engineering

Follow-Up

Better Experiments with LLM Evals — A Funnel, Not a Fork 👨‍💼

Code Generation, Leads Classification, Content Writing… LLM can be useful in A/B testing! At Spotify, only 12% of A/B tests produce a shipped positive result and teams roll back 42% of launched experiments before they fully ship, which means evaluation effort spent on experiments that shouldn’t run at all is the biggest waste in the pipeline. The fix is positioning LLM evals as a funnel upstream of experiments: automated judges score relevance, coherence, and quality across large candidate sets, catching trust-breaking content or regressions that make a treatment unworthy of a full test, while experiments remain the authority on whether actual user outcomes moved. Follow Spotify’s post for details.

#ai #softwareengineering #engineering #quality

Migrating Data Ingestion Systems at Meta Scale 👨‍💼

Big migrations are scary and effortful. Imagine one with the biggest MySQL deployment in the world like at Meta’s data ingestion system.

The team ran both legacy and new systems in a shadow testing mode, diffing outputs for correctness so individual pipelines could be validated and migrated independently; partition-level metadata flags automatically halted new delta landings and forced merges with known-good partitions whenever a partition was flagged as bad, bounding data quality risk during the transition.

At @boltapp we followed the same shadow testing approach during our migration from MySQL to TitaniumDB.

#db #architecture #distributed #engineering

Securing Private Video Content 👨‍💼

Sharing a video via YouTube is easy with it’s visibility settings and paid walls. But if you’re running your own video platform like in a learning platform? Lee Gilmore showcases how to use AWS Lambda, CloudFront, S3 and Secrets manager to manage the access to video content reliably in a serverless environment.

#casestudy #serverless

From Latency to Instant: Modernizing GitHub Issues Navigation Performance 👨‍💼

At the start of 2026, only 2% of GitHub Issues page views completed in under 200ms — a meaningful gap against local-first tools that set the modern expectation for instant navigation. The fix layers a client-side cache backed by IndexedDB with a “preheating” strategy: a service worker proactively walks high-intent issue references and pre-warms cache entries just before a user navigates, hitting the network only on misses, which keeps the approach both fast and cheap. Rendering from local cache first and revalidating in the background pushed 35% of issue views under 200ms; the team is now investing in an edge delivery layer closer to users and backend rewrites optimized explicitly for low-latency delivery.

#architecture #softwareengineering #engineering #quality

How Linear is so fast? 👨‍💼

We started with Linear at Supplied AI right away due to it’s rich intergration capabilities, but more importantly - it was and still is fast. Apparently, the secret lies not in the fat DB hosts or extensive caching. It’s optimizing the web bundle, making optimistic right into local db and syncing it later, preloading, rendering before authentication and multiple other optimizations. Find all the details in the article by Dennis Brotzky.

#casestudy #performance

Big thanks to Nikita, Constantin, Anatoly, Oleksandr, Dima, Pavel B, Pavel, Robert, Roman, Iyri, Andrey, Lidia, Vladimir, August, Roman, Egor, Roman, Evgeniy, Nadia, Daria, Dzmitry, Mikhail, Nikita, Dmytro, Denis and Mikhail for supporting the newsletter on Patreon! Alternatively, you can upgrade to the paid version and get access to the premium only posts!

Software Architecture Weekly is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

Subscribe now

Business Oriented System Design Course

Wanna learn how to design systems which truly impact business and get promoted? Sign up for Business Oriented System Design Course: starts 8th of June. Begin the year with learning crucial skills and advance your career! 100+ engineers from Bolt, Booking.com, Semrush, TripleTen etc. who completed the course and brought their architecture skills to a new level.

Highlights

RAG in Production 👷‍♂️

We faced the problem of choosing data sources for our AI Agents and obviously we started weighing retriaval augamented generation. And it looks like RAG is surprisingly complex: tokenization process, committing to the model, reindexing cost and many more. Grab a great article on the topic.

#rag #ai

Temporal Primer — Building Long-Running Systems 🍼

Temporal is a durable execution engine that records every workflow event — activity scheduled, timer fired, signal arrived — as an immutable entry in an event history persisted to Cassandra, PostgreSQL, or MySQL, then replays that history on worker restart so a process survives network partitions and VM reboots without any custom state management code. Workflow code must be strictly deterministic (no random values, no wall-clock reads, no direct I/O); all side effects live in Activities, which carry independent retry policies. For agentic systems that chain dozens of tool calls over minutes or hours, this durability model solves exactly the failure class that cron + queue + DB hacks break under.

#distributed #architecture #softwareengineering #llm

Structured prompt driven development 👷‍♂️

Everybody who hasn’t spent last 2 years in a cave came up with their own way of working in the AI for software development era. Thoughtworks is no exception. Their core idea is prompt should become a first class citizen, just like code: be saved, versioned and reviewed. Based on this idea they develop the structured prompt process and tools and show how to implement a feature with it. To my taste the approach is a bit naive because creating holistic, full and correct test cases is much more complicated than “make me test cases, avoid duplicates” like it is shown in the article, but interest approach anyway.

#softwaredevelopment

Follow-Up

5 Ways to make CDC with Postgres

Short article not only on the CDC blueprint from Pinterest, but also mentioning 5 different ways of how Change Data Capture can be implemented in PostgreSQL. I knew only half of them!

#db #cdc

What’s Easy Now? What’s Hard Now? 👨‍💼

AI coding agents are fundamentally feedback loops built around LLMs, and the quality of available feedback determines where they succeed or fail. Tasks with fast, clear, objective feedback — like building a UI — are much easier for agents than tasks where feedback is delayed, silent, or subjective: writing concurrent code (where bugs manifest as silent data corruption at runtime), or making architecture decisions (where feedback is inherently contextual and often never arrives). Brooker frames this as the most important axis for evaluating agent capability, whether the problem domain supports tight feedback loops.

#ai #softwareengineering #engineering #llm

We replaced Redis with MySQL for inventory reservations—and it scaled 👨‍💼

Shopify replaced a Redis-backed inventory reservation system with MySQL by switching from a single quantity-per-item counter to one row per sellable unit, using MySQL 8’s `SKIP LOCKED` to eliminate row-level contention without waiting. The old Redis model had no atomicity across the reserve and claim steps — payment could succeed while inventory went unclaimed — and had no multi-location awareness; the new MySQL design fixes both with ACID guarantees in a single database transaction. A bounded pool capped at 1,000 rows per item/location pair keeps write paths narrow. Under peak Black Friday 2025 traffic ($5.1M in sales per minute), writer CPU stayed under 50% and reader CPU under 16%, with headroom to spare. #db #architecture #softwareengineering engineering

Valkey 9.1 delivers improvements in security, performance, and more 👨‍💼

Valkey 9.1 cuts per-key memory usage by up to 10% for common workloads — up to 20% for strings under 128 bytes and 10% for sorted sets via skiplist optimizations — with no reconfiguration required. A redesigned I/O threading model improves throughput by up to 17%, and a new database-level ACL system enables fine-grained multi-tenant isolation within a single instance. The headline addition is Valkey Search 1.2, which folds full-text search, tag filtering, numeric lookup, and AI-native vector search directly into the same system as the cache — eliminating the need to run a separate search platform alongside it.

#db #distributed #softwareengineering #cloud

Your AI wants to nuke your database 👷‍♂️

AI deleting your production database is not a nightmare anymore: it’s a reality which happened to PocketOS running their systems on Railway. To be fair to the latter company, they learned from the incident and implemented 48 hour window for soft deletion and backups for backups. Learn the full story in the article.

#ai #resilience

Big thanks to Nikita, Constantin, Anatoly, Oleksandr, Dima, Pavel B, Pavel, Robert, Roman, Iyri, Andrey, Lidia, Vladimir, August, Roman, Egor, Roman, Evgeniy, Nadia, Daria, Dzmitry, Mikhail, Nikita, Dmytro, Denis and Mikhail for supporting the newsletter on Patreon! Alternatively, you can upgrade to the paid version and get access to the premium only posts!

Software Architecture Weekly is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

Business Oriented System Design Course

Wanna learn how to design systems which truly impact business and get promoted? Sign up for Business Oriented System Design Course: starts 5th of May. Begin the year with learning crucial skills and advance your career! 100+ engineers from Bolt, Booking.com, Semrush, TripleTen etc. who completed the course and brought their architecture skills to a new level.

Highlights

Redis Cluster 👷‍♂️

Redis is single-threaded and despite that, can handle hundred of thousands requests per second while guaranteeing atomicity. This is enough for the majority of systems, but not for Stripe. They moved from a single hot node to a 10-nodes cluster and told everyone how Redis cluster operates in this well-written article.

#redis #performance

Why GitHub Actions suck big time 👷‍♂️

At Supplied we’re using GitHub Actions, and it is mostly fine... until it does not. Slow builds, tricky YAML, awful log explorer and the (un)availability start to bite altogether. Apparently, we’re not the only one experiencing the troubles. Ian Duncan digs into each issues and actually suggest couple of alternatives.

#ci

The CTO’s Entropy War 🍼

First of all, the code writing was never a bottleneck: product understanding, architecture, quality control was. Second, AI will generate the code but why the engineer has not reviewed it? How it passed the peer review with such grave architecture problems? Those are my questions to the article, but it is good anyway: whatever entropy you have in your codebase, the AI will only amplify, not solve them.

The CTO Substack

The CTO’s Entropy War

I’m standing behind Marcus, one of my best engineers, watching him demo a feature he built in forty minutes using Claude Code. A reporting dashboard. Clean layout. Responsive. Tests passing. He’s beaming…

Read more

5 months ago · 10 likes · 2 comments · Etienne de Bruin

#ai #cto

Follow-Up

Compromised Axios 👷‍♂️

Another week, another compromised npm package. Axios - the most popular js library for HTTP requests - got infected with a malicious dependency via… right, social engineering. Make sure to check out the infection signals and apply all the remediation and prevention steps from the article!

#security

Practical Lessons From the Claude Code Leak 🍼

While Axios was compromised on purpose, the Claude Code sources leaked accidentally as source maps during the update publishing. A great chance to learn from it though to get best practices on Claude.MD, multi-agent orchestration, permissions and many more.

The Generative Programmer

Practical Lessons From the Claude Code Leak

On March 31, 2026, security researcher Chaofan Shou posted (more than 34 million views) that Claude Code’s entire source had leaked through a sourcemap left in Anthropic’s public npm package. Roughly 500,000 lines of TypeScript across about 1,900 files. Not the model or weights, but the harness around them: system prompts, tool routing, memory, permissi…

Read more

5 months ago · 11 likes · Bilgin Ibryam

#security #ai

Every query gets a receipt 👷‍♂️

Everyone knows that running an sql SELECT query results with a list of rows, but only few is conscious that “31337 rows affected” is part of the same response too. While scaling the system especially with exposed external API accessing your DB, it’s important to understand how costly each query becomes and there is a nice trick to use this standard part of sql response to report on consumed DB resources. Find out how exactly here.

#db

A decade of Docker containers 👷‍♂️

Docker containers fuel modern software development from small companies to large enterprises, from tiny Docker Compose instances to large Kubernetes Clusters. This article explains the need for the containers, shows the components of Docker and discusses the Docker application in the age of AI. Looong read indeed.

#docker

Why fakes beat mocks and testcontainers 👷‍♂️

Mocks and Testcontainers are the two tools most developers reach for. Both have fundamental limitations. Testcontainers are binary failing to provide partial failure modes, and mocks test implementation rather than results. Fakes fix both issues: they are the in-memory implementations which allow to test results, while providing partial failures. Grab the best practice guide!

#qa #tests #testcontainers

Big thanks to Nikita, Constantin, Anatoly, Oleksandr, Dima, Pavel B, Pavel, Robert, Roman, Iyri, Andrey, Lidia, Vladimir, August, Roman, Egor, Roman, Evgeniy, Nadia, Daria, Dzmitry, Mikhail, Nikita, Dmytro, Denis and Mikhail for supporting the newsletter on Patreon! Alternatively, you can upgrade to the paid version and get access to the premium only posts!

Software Architecture Weekly is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

Highlights

Evals in Skill-Creator 👷‍♂️

People created over half a million Claude Code skills, but do they improve the claude performance? Actually, developer-written context files improved agent task completion by just 4% on average, while LLM-generated ones actually made performance worse by 3%, and both increased costs by over 20%. The solution is to evaluate the skill. Antropic shipped exactly that - find more in the Tessl’s blog.

#ai

Context Anchoring 👷‍♂️

Context anchoring is the practice of externalizing the reasoning, decisions, and intent behind work (e.g., into commits or shared documents) so AI systems can maintain continuity across sessions and avoid losing critical context. By turning context into a persistent, queryable artifact rather than ephemeral chat history, it reduces drift, improves collaboration with AI, and enables more reliable long-running development workflows. See more in Martin Fowler’s blog.

#ai

Bcrypt vs Argon2: What’s better? 🤟

If you control which algorightm you use for hashing password and haven’t made the choice yet, this article will be handy. It benchmarks the resource consumption and cost of cracking a password hashed with both bcrypt and argon2 on CPU and GPUs from the CPU time and memory perspective. Grab a bunch of tables!

#security #encryption

Follow-Up

Database Transactions 👷‍♂️

A great breakdown of database transactions: commits, rollbacks, consistent reads, isolation levels, multiversion control in PostgreSQL. Perfect structure to learn as a beginner, great refresher for experienced developers.

#db

The monolith on a Single EC2 instance 👷‍♂️

Every architecture decision is a trade-off. And while we tell everybody to go back to the monolith, it is worth remembering not to put everything in one basket. Grab a list of considerations of what can go wrong with a solution on a single EC2 instance.

#architecture

Why software engineering jobs are surging 🍼

AI is a platform shift (like the internet or cloud) that expands what can be built, causing an explosion in software demand rather than reducing it, so more code—and thus more developers—are needed. Instead of replacing developers, AI changes their role toward higher-level tasks (architecture, orchestration, innovation), creating new jobs and increasing the overall demand for specialized human-written software.

#ai

Break Paxos 🍼

Consensus protocols are boring, but what if we can make them fun? Murat Demirbas vibecoded a game where your task is to break Paxos sync for 3 nodes by killing and slowing them down. I tried it and it was actually fun!

#distributedsystems

Big thanks to Nikita, Constantin, Anatoly, Oleksandr, Dima, Pavel B, Pavel, Robert, Roman, Iyri, Andrey, Lidia, Vladimir, August, Roman, Egor, Roman, Evgeniy, Nadia, Daria, Dzmitry, Mikhail, Nikita, Dmytro, Denis and Mikhail for supporting the newsletter on Patreon! Alternatively, you can upgrade to the paid version and get access to the premium only posts!

Architecture Weekly Issue #194. Articles, books, and playlists on architecture and related topics. Split by sections, highlighted with complexity: 🤟 means hardcore, 👷‍♂️ is technically applicable right away, 🍼 - is an introduction to the topic or an overview. Available in telegram as well.

Software Architecture Weekly is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

Highlights

The CTO’s New Engineering Ladder 🍼

Traditional corporate ladders for Engineers looked like Intern-Junior-Middle-Senior-Principle-Distinguish and hinged on ability to delivery changes and make impact. However, anyone can ship now with Cursor and Claude Code. That forces technical leaders into rethinking the career. Take an idea from Etienne de Bruin.

P.S. I am a bit doubtful. In Supplied we have just Product Engineers. But we are tiny yet.

The CTO Substack

The CTO’s New Engineering Ladder

I’m sitting across from a senior engineer named Priya on a Tuesday afternoon. Eight years in the industry. Runs the platform team. Every company I’ve ever worked in would have called her a senior engineer without hesitation. I’m doing a routine level review and I ask her, as casually as I can, “What does senior actually mean on your team right now…

Read more

6 months ago · 11 likes · Etienne de Bruin

#management

AI firewall 👷‍♂️

Remember courtcase when a person won against a Canadian airline which AI chat bot offered a discount which didn’t exist in reality. The problem is not a mere hallucination, but a reason to rethink what any AI solution does in your system. AI Firewall is an idea to sanitize and evaluate the customer input that ends up in an LLM. Alex Ewerlöf nails the topic.

Alex Ewerlöf Notes

AI firewall

We’ve seen many ridiculous AI incidents over the past few years…

Read more

6 months ago · 7 likes · Alex Ewerlöf

#ai #llm

The AI Agent Blueprint 👷‍♂️

Talking about AI - what’s the difference between an AI-driven workflow and an AI Agent? What constitutes the AI agent itself? Find in my own article!

#ai

Follow-Up

A need for Resilience 🍼

Here’s another side of the coin: AI amplifies your company culture. It does not solve problems, it amplififes both good and bad traits. Typically, the companies would use AI to increase speed(Supplied is no exception), and with greater speed comes worse resilience. Think trees: the one grows the fastest, falls first under a storm. Don’t listen to me, better read what Uwe Friedrichson thinks about it.

#resilience

Software Testing Strategies in 2026 🍼

The software development changed a lot with AI Assistants and Coding Agents, the the buzz is everywhere. What is talked about less frequent, it’s the Testing Strategies and how it changed with AI. Starting with the foundations, this article covers the decisions trees on what and when to automate and touches on how AI contributes to testing strategies nowadays. Would be nice though to get some sort of insights into tooling and practical approaches too.

#testing #qa

Chaos Engineering at Monday.com 👷‍♂️

Chaos Engineering is a string tactic to ensure the reliability of your system. Monday.com shares their adoption of Chaos-Mesh for Kubernetes to simulate various failures and see the consequences.

#resilience #chaosengineering

The Serial Safety Net 🤟

Having Serializable Isolation Level leads to significant performance hit in databases. This problem led researches to introduce The Serial Safety net which tracks the dependencies between transactions and ensures either that it’s safe to commit both, or prevents one from doing so. Murat Demirbas made a nice overview.

#db

Lambda Managed Instances for Steady Workloads 👷‍♂️

Some time ago AWS announced you can pick and reserve EC2 instances for your AWS Lambda getting best of serverless and serverfull worlds. The example of the practical use case and reference architecture you will find in this blog post.

#serverlesss

Big thanks to Nikita, Constantin, Anatoly, Oleksandr, Dima, Pavel B, Pavel, Robert, Roman, Iyri, Andrey, Lidia, Vladimir, August, Roman, Egor, Roman, Evgeniy, Nadia, Daria, Dzmitry, Mikhail, Nikita, Dmytro, Denis and Mikhail for supporting the newsletter on Patreon! Alternatively, you can upgrade to the paid version and get access to the premium only posts!

Traditional software follows predefined paths: the frontend makes an HTTP Request, the backend queries the database, transforms the result into JSON format and yields a response. Any variation is handled by if-else or switch statements.

The problems like evaluating if a Jira ticket has enough information for creating test cases may hardly be solved with if-else statements. Sounds more like a job for an LLM.

People build UiPath, Zapier or n8n workflows where the flow is predefined, but one or several steps are using LLMs. For the Jira ticket example, we can build one like this:

This is not an agent: this is a workflow. We just solve one step of it with an LLM. The LLM is not in the control. However, LLMs are powerful: they can generate texts, write code, analyse Jira tickets, summarise meeting transcripts. They can also make decisions. What if we grant them the power to decide what to do?

Here’s the key idea:

Let’s ask an LLM what step to take. LLM will tell us what tool to call, and what parameters to pass to this tool. Our job is to call the tool, feed the result back and ask what step to take next.

Now let’s dig into the details.

System Prompt

Every agent begins with a system prompt. The purpose of the prompt is to tell the agent what it is supposed to do. This is where you provide an agent role, the end results, guidelines and tools. If your agent was an employee, system prompt is the job instruction. For example(shortened and simplified):

“You’re a senior technical writer. Create technical documentation for our software product and generate appropriate diagrams. Make sure the diagrams are clear and comprehensible. The technical documentation should include architecture decisions, API contracts and addressed non-functional requirements”.

Typically this is your AGENTS.md/CLAUDE.md file or any other file that you pass to an agentic framework; more on them later.

The real system prompts though are typically much longer and elaborate; they should include not only the end results, but also the available tools, references to subagents, guidelines and guardrails. Talking about tools: why we need them and what they are?

Tools

A majority of software solutions exchange data with a database, read and write files, call APIs and take other actions in external world. In AI Agents these capabilities are implemented as tools: dedicated instruments to perform a certain job. Formally we can define a tool as the following set:

Let’s imagine we want to query Jira Issue Tracker for the content of a certain ticket. First, we need to create a script which sends an API request to Jira. Then we need to understand what parameters we pass into the script, and what we get back. That’s schema. And finally the tool should have a name and description to be useful for our LLM.

We can add more tools: like connection to code storage or a knowledge base. For example, we might want our LLM to read Notion pages to learn more about the product.

But how the LLM is going to know about the tool in the first place? Right, we need to tell LLM about available tools in system prompt. Here’s how it looks like:

### Your tools

‘read_tickets’ | Queries Jira for a ticket’s title, description and comments by id

‘notion’ | Queries Notion Knowledge Base for the additional product information.

If you’re building your AI Agent using Claude Code, there is no need to specify the contract strictly, however it’s still beneficial for efficiency.

Here’s a profound difference between an AI powered workflow and an agent: now LLM controls what to call and when instead of us. Rather than making the LLM a tool, we make LLM think which tools to call.

Of course, direct API integration is not the only way to add tools to LLMs. You likely heard of MCP - Model Context Protocol. It is even more convenient, because if a tool exposes an MCP Server, it becomes a matter only of configuration and authentication. Notion for example, exposes one. Here’s how to add it.

State

LLMs themselves are mere token predictors, thus they got neither memory, nor state. So our agent should solve the problem of providing both. Consider state: as the LLM does not know if the ticket was already queried, it will query it again which makes no sense. Here’s how it’s solved in AI Agents.

First we define what state we need: in our case it should indeed have the ticket id, the ticket content, content of relevant notion pages, etc. A JSON object will work just fine:

class QAAgentState(TypedDict):

ticket_id: str

ticket_data: dict

user_decision: str | None # “replace” | “supplement” | “skip”

notion_queries: List[str]

notion_search_results: List[dict]

notion_pages: List[dict]

user_feedback: str | None

What happens to state? Our AI Agent will update the state after every LLM interaction and will pass this state to an LLM together with the system prompt. Then LLM will know at what step it is because again: LLM is stateless itself.

Memory

AI agents maintain Short-Term Memory (STM) for the current conversation and Long-Term Memory (LTM) for persistent knowledge. STM functions like human working memory: temporary, conversation-bound, and used for immediate reasoning. It typically includes the conversation history, tool outputs, system prompts, working state, file diffs, and injected artifacts.

Agent frameworks construct prompts by combining these elements with the latest user message. Tools such as LangChain Checkpointer store conversation state using a key-value store tied to a thread_id. However, context windows are limited, even in modern models with up to ~1M tokens, so conversations must sometimes be summarized or compressed.

Important insights discovered during interactions—such as recurring mistakes or learned preferences—should be saved to Long-Term Memory. Systems like Claude Code persist this memory in files (e.g., project_memory.md, decisions.md) or dedicated memory folders that the agent reads at session start. Frameworks like LangGraph implement persistent memory using Stores (e.g., PostgreSQL, MongoDB) and Namespaces for organizing data.

In practice, STM keeps the agent coherent within a session, while LTM preserves important knowledge across sessions. Together they enable AI agents to behave consistently and improve over time.

Agent Loop

How it all works together? Any AI agent operates the loop:

1. take the system prompt, add the state and memory

2. send a request to LLM

3. parse a response

4. if this is final output, finish work

5. otherwise, determine which tool to call

6. Call the tool

7. Update the state and memory

8. Start from Step 1

That’s the whole blueprint. However, as any software systems, it requires cross-cutting concerns to be addressed.

Observability

Proper introspection is a must for any system. Let’s see how to add observability to see what our agent is doing under the hood.

Modern systems rely on OpenTelemetry and AI Agents are no exception. Each agentic solution, be it Claude Code or LangGraph, OpenAI SDK or others support OTEL for sending traces, tool calls, LLM calls, costs, timings and other signals.

Configuring the agent can be as easy as providing the required environment variables if using a dedicated observability solution like LangSmith. Find the setup instructions there.

Guardrails

Traditional software is hard to operate well, and AI agents are no easier. To the integration and logic problems we add cost considerations, tools overusage, infinite loops problems and many more. Thus we need guardrails to solve them.

Guardrails typically operate at several layers. Input guardrails validate or sanitize user prompts to block prompt injections, malicious instructions, or sensitive data requests. Output guardrails inspect the model’s response before it is returned, filtering hallucinations, toxic content, policy violations, or unsafe instructions. Tool guardrails restrict how the agent can call external tools or APIs—for example enforcing schemas, permission checks, rate limits, and argument validation.

In agent frameworks such as LangChain, LangGraph, or OpenAI Agents SDK, guardrails are often implemented using middleware, structured output validation (e.g., JSON schema, Pydantic models), and policy engines. Some systems also add runtime monitoring and human-in-the-loop checkpoints for critical actions.

In practice, guardrails combine prompt policies, validation logic, security checks, and monitoring to ensure the agent remains reliable, safe, and aligned with application constraints.

Evaluation Harness

Traditional software quality relies on unit, integration, and end-to-end tests — all of them deterministic. AI agents are not. The same input can produce different outputs on different runs, which makes classical test assertions fragile. That doesn’t mean evaluation is impossible; it just requires a different toolkit.

Manual review is the baseline: run the agent against a set of representative inputs and inspect the outputs by hand. It’s slow, but it builds intuition and is the right starting point before you invest in automation.

LLM-as-judge scales manual review. You write an evaluation prompt describing what “good output” looks like, then feed the agent’s response to a second LLM and ask it to score or explain the result. This works well for subjective quality criteria — correctness, tone, completeness — that are hard to encode as assertions.

Deterministic assertions cover what you can pin down: does the agent call the right tools? Does the output contain required fields? Does it stay within token budget? These are your regression tests — fast, cheap, and automatable with any standard test runner.

Observability as a signal closes the loop in production. Tracking tool call frequency, latency, cost per run, and error rates over time surfaces regressions that no pre-deployment test would catch.

In practice, you layer all four: manual review to bootstrap a golden dataset, deterministic assertions for known-good behavior, LLM-as-judge for open-ended quality, and observability for production drift. Frameworks like LangSmith, Braintrust, and PromptFoo provide ready-made infrastructure for running and tracking these evaluations at scale.

Summary

An AI agent is not magic — it is a loop. An LLM decides what to do next; the execution environment carries out the decision, updates state, and feeds the result back. Repeat until done.

The components covered here — system prompt, tools, state, memory, observability, guardrails, and evaluation — are not optional extras. Each one addresses a failure mode that will surface in production. Skip the system prompt and the agent has no purpose. Skip guardrails and it will run up your API bill in an infinite loop. Skip evaluation and you won’t know when it breaks.

The good news: you don’t have to build this from scratch. Frameworks like LangGraph, OpenAI Agents SDK, and Claude Code handle the scaffolding. Your job is to define the goal, design the tools, and put guardrails around the things that matter.

Software Architecture Weekly is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

Highlights

Harness Engineering 👷‍♂️

OpenAI built a software writing 1 million lines of code, from which exactly 0 was manually written - it was done by Codex agents instead. However, engineers are needed to navigate the development, construct context, write relevant rules and enforce the guardrails. The article shows how software development changed in just 3 months and how strict rules, well-defined architecture, API contracts and feedback loops are not simply important - they are necessary.

#ai #softwareengineering #architecture

Supabase Incident on February 12, 2026 👷‍♂️

A deployment of a new internal monitoring service inadvertently toggled AWS VPC Block Public Access to “block-bidirectional” across the entire us-east-2 region, killing all internet gateway traffic for 3 hours and 42 minutes. Resolution dragged on because the networking change wasn’t surfaced quickly in audit logs, pre-production environments didn’t include us-east-2, and the right infrastructure teams weren’t escalated to early. Supabase’s remediation plan: isolate non-customer services in separate AWS accounts, add external connectivity monitoring, enforce production/pre-production region parity, and overhaul escalation and customer communication protocols. Pretty surprising they didn’t use different AWS account in the first place!

#devops #cloud #engineering

CDC Approaches and Best Practices 👷‍♂️

Ensuring the data in downstream systems like analytical databases and business intelligence platforms remain consistent with the central transactional database is critical for making quick and accurate business decisions. Change Data Capture - CDC - is an approach to making it happen. Redpanda got a nice article from their “Fundamentals of Data Engineering” series with the design references, push and pull implementations and best practices for CDC.

#dataengineering #cdc

Follow-Up

Database Federation: Decentralized and ACL-Compliant Hive™ Databases 🤟

Uber’s Delivery business ran a monolithic Hive database holding 16,000 datasets and 10 petabytes of data — a single failure domain causing cascading outages and overly permissive cross-team access. Their migration uses pointer manipulation rather than data duplication: a Bootstrap Migrator handles the initial copy with checksum verification, a real-time Apache Flink synchronizer drives 10 concurrent Piper pipelines for ongoing changes, and a Batch Synchronizer runs every 2 hours as a consistency backstop. The result: zero-downtime migration across thousands of datasets, 7 million HMS sync operations, and 1+ petabyte of redundant storage eliminated — plus granular ACL enforcement that was impossible in the monolith.

#architecture #distributed #db #migration

Heartbeats in Distributed Systems 🤟

Heartbeats are the primitive that separates a slow node from a dead one — and the parameters matter more than most engineers realize. Kubernetes uses 10-second kubelet intervals with 40-second timeouts; etcd uses 100ms Raft heartbeats; the rule of thumb is that timeouts should be at least 10x the round-trip time to absorb network variance. Beyond naive ping-based detection, the article covers the Phi Accrual Detector (continuous suspicion scores instead of binary alive/dead) and gossip protocols that distribute failure detection across all nodes, eliminating centralized bottlenecks.

#distributed #architecture

The new AI-driven SDLC 👷‍♂️

AI compresses planning from weeks to minutes and parallelizes development, but the bottleneck shifts to code review — humans can’t keep pace with AI-generated changeset volume. CircleCI identifies three friction points: infrastructure that must validate contextual fit at machine speed, review processes that don’t scale, and an identity shift for engineers from execution to judgment. Their recommendations: connect AI tools to pipeline data via MCP, redesign review workflows with automation and fast-path approvals for large changesets, and reframe senior engineering roles around architecture and strategic oversight.

#ai #devops #softwareengineering

All Agentic Architectures 👷‍♂️

A practical catalog of 17+ AI agent design patterns implemented with LangChain and LangGraph, organized into five progressive stages. Coverage spans foundational patterns (ReAct, Planning, Reflection, Tool Use), multi-agent collaboration (Meta-Controller routing, Blackboard shared memory, Ensemble methods), advanced memory and reasoning (Tree of Thoughts, graph-based world models), safety mechanisms (Plan-Execute-Verify, dry-run approval, simulator-based risk assessment), and self-improvement loops. The repository includes LLM-as-a-Judge evaluation patterns for measuring agent performance — making it more than a pattern catalog.

#ai #llm #architecture

Big thanks to Nikita, Constantin, Anatoly, Oleksandr, Dima, Pavel B, Pavel, Robert, Roman, Iyri, Andrey, Lidia, Vladimir, August, Roman, Egor, Roman, Evgeniy, Nadia, Daria, Dzmitry, Mikhail, Nikita, Dmytro, Denis and Mikhail for supporting the newsletter on Patreon! Alternatively, you can upgrade to the paid version and get access to the premium only posts!

I’ve been a solution architect for years. My job was picking technologies for clients and then convincing directors and VPs of Engineering that my choice isn’t insane. I’ve failed at this. I’ve also succeeded. The difference was never the technology — it was always the pitch.

Here’s the uncomfortable truth most engineers never learn: CTOs don’t care about your technology. They care about their problems. And their problems are measured in dollars, headcount, and quarters — not in developer experience or elegant APIs.

What Your CTO Actually Thinks About

Forget the tech radar for a second. A CTO is a manager first and an engineer second (if at all). Their daily anxiety revolves around five things:

1. Will this product succeed? — Does the technology help users solve real problems, or is it resume-driven development?

2. Can I explain this to the CEO? — The CFO doesn’t know what Kubernetes is. The CTO has to translate your architectural decisions into plain business language. Every. Single. Board meeting.

3. Can we launch new products with this? — Companies grow. A technology that only solves today’s problem is a liability tomorrow.

4. Can I hire for this? — Brilliant tech with zero talent pool is a dead end. Your CTO is thinking about the next 10 job postings, not your benchmark results.

5. Does this make the team faster? — Tooling, libraries, CI/CD, monitoring — the whole ecosystem matters. A fast language with no ecosystem is a slow language in practice.

If your pitch doesn’t address at least three of these, you’ve already lost.

The Pitch: Four Rules That Actually Work

Rule 1: Tell a Story, Not a Spec Sheet

Nobody was ever persuaded by a bullet list of features. Structure your pitch as a narrative:

“Our mobile team ships two apps — iOS and Android. We have 7 engineers. We ship roughly the same features on both platforms, 3 weeks apart. That delay costs us users. Here’s how we fix it.”

Problem → Solution → Prize. That’s it. The prize is always time or money saved. Everything else is decoration.

Rule 2: Translate Developer-Speak into CTO-Speak

Check this out:

What You Say What the CTO Hears

Short feedback loop Faster velocity

Cross-platform Lower cost

Type safety Fewer production incidents

Great DX Nothing. This means nothing to them

All arguments must be denominated in time and money. If you can’t convert a benefit into one of those two currencies, leave it out. It’s noise.

Rule 3: Show Alternatives

Walk in with one option, and the first question will be: “What else did you consider?”

Walk in with three options — with honest tradeoffs for each — and you look like someone who did their homework. Because you did.

Rule 4: Show the Downsides (Before They Find Them)

Nothing destroys credibility faster than pretending a technology is perfect. Every technology has warts. Name them. Rank them. Explain how you’d mitigate each one.

This isn’t weakness — it’s the strongest signal of competence you can send.

The Math: A Simple Framework for ROI

Here’s where most engineers stop and most successful pitches start. You need a spreadsheet, not a slide deck.

The Variables

• CC (Current Cost) — What you spend today: salaries + licenses. This is your OPEX baseline.

• NC (New Cost) — Same calculation for the proposed technology. Ideally lower.

• MC (Migration Cost) — The one-time CAPEX to transition. Be honest here; underestimating this is the #1 way pitches die post-approval.

• NO (New Opportunities) — Revenue from things the new tech enables that the old tech simply can’t do.

• RC (Risk Cost) — For each risk: probability × mitigation cost. Sum them up.

The Formula

(CC - NC) + NO - (RC + MC) > 0

That’s it. If this number is positive, you have a business case. If it’s not, you either need better numbers or a different technology.

The Velocity Check

Cost per feature (new) < Cost per feature (old)

Or equivalently: same team, more output. If you can’t demonstrate this, the migration is a lateral move at best.

Putting It Together: Flutter vs. Native

Let’s make this concrete. A team builds iOS and Android apps with Swift and Kotlin. Someone wants to switch to Flutter.

Current State

| Role | Count | Cost |

| Android developers | 2 | $200k |

| iOS developers | 2 | $200k |

| Mobile lead | 1 | $100k |

| QA engineers | 2 | $200k |

Total: 7 | $700k/year

Proposed State (Flutter)

| Role | Count | Cost |

| Flutter developers | 2 | $200k |

| Mobile lead | 1 | $100k |

| QA engineers | 2 | $200k |

| Total | 5 | $500k/year |

Annual savings: $200k.

The CTO Translation Table

Developer Thinks | CTO Hears | Verdict

Hot reload = fast iteration | Higher velocity | ✅

Single codebase | 30% cost reduction | ✅

Fewer platform bugs | Better user retention | ✅

Dart is... interesting | Hiring risk | ⚠️

Young ecosystem | Missing tools | ⚠️

Moving fast | Migration risk | ⚠️

New Opportunities

Flutter runs on web and desktop. That’s new user channels the native stack simply couldn’t reach. Estimated value: $200k/year.

Risks

| Risk | Probability | Mitigation Cost | Exposure

| Missing critical libraries | 20% | $100k | $20k

| No security scanning tools | 20% | $100k | $20k

| Can’t hire senior Flutter devs | 20% | $100k | $20k

| Unexpected technical blockers | 20% | $100k | $20k

Total exposure: $80k

Migration Cost

Two developers + one QA, three months of transition: $300k.

The Final Number

($700k - $500k) + $200k - ($300k + $80k) = $20k

Profitable in year one. And in year two? The migration cost is gone, and you’re looking at $320k annual net benefit.

That’s not a technology pitch. That’s a business case.

Before You Walk In

Find an ally. One colleague who believes in the idea gives you social proof. A solo pitch is an opinion; a supported pitch is a movement.

Practice the story. Dry-run it with someone who’ll push back. If your pitch survives a skeptic, it’ll survive the CTO.

Show enthusiasm — but earned enthusiasm. If you’ve done the math, run the PoC, and mapped the risks, your confidence isn’t arrogance. It’s preparation.

Be ready for “no.” Sometimes the timing is wrong, the budget isn’t there, or the risk appetite is low. A well-constructed pitch that gets rejected today becomes the reference document when priorities shift next quarter.

The Bottom Line

Engineers love to debate technologies on technical merits. CTOs make decisions on business merits. The gap between these two conversations is where most good ideas go to die.

Bridge that gap with numbers, honesty, and a story worth telling. The technology sells itself once you’ve proven it solves a business problem — not a developer problem.

Business Oriented System Design Course

Wanna learn how to design systems which truly impact business and get promoted? Sign up for Business Oriented System Design Course: starts 5th of May. Begin the year with learning crucial skills and advance your career! 100+ engineers from Bolt, Booking.com, Semrush, TripleTen etc. who completed the course and brought their architecture skills to a new level.

Software Architecture Weekly is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

Business Oriented System Design Course

Wanna learn how to design systems which truly impact business and get promoted? Sign up for Business Oriented System Design Course: starts 5th of May. Begin the year with learning crucial skills and advance your career! 100+ engineers from Bolt, Booking.com, Semrush, TripleTen etc. who completed the course and brought their architecture skills to a new level.

Highlights

The Great Software Quality Collapse 🤟

Software quality has been degrading for years — and it’s not just an AI problem. Apple Calculator leaking 32GB of RAM, CrowdStrike crashing 8.5 million machines due to missing bounds checking, Teams consuming 100% CPU. The industry is spending $364 billion on infrastructure to compensate for engineering failures instead of fixing root causes. The AI-generated code era with its 322% more security vulnerabilities only accelerates the trend. What we are all going to do about?

#softwareengineering #quality

Explicit Architecture: DDD, Hexagonal, Onion, Clean, CQRS — How I Put It All Together 👷‍♂️

A comprehensive synthesis of major architectural patterns into one coherent model. Instead of treating Ports & Adapters, Onion Architecture, DDD, and CQRS as separate concepts, the author combines them into three fundamental blocks: User Interface, Application Core, and Infrastructure — with all dependencies pointing inwards.Try out the “package by component” approach organizing code by business domains rather than technical layers is especially practical.

#architecture #ddd

Unconventional PostgreSQL Optimizations 🤟

Three creative PostgreSQL optimization techniques beyond the usual indexing advice. Constraint exclusion to skip full table scans when query conditions contradict check constraints. Function-based indexes reducing index size from 214MB to 66MB through deduplication. Hash indexes for uniqueness enforcement on large text columns consuming 5x less space than B-tree alternatives. Strategic use of PostgreSQL’s lesser-known features can dramatically change performance characteristics.

#postgresql #db

Why Spec-Driven Development Can Be Iterative, Incremental and Agile 👷‍♂️

Spec-driven development is not waterfall. The author argues it iterates on understanding rather than solely on code — short, focused, Markdown-based specifications stored in version control and treated as living documents. Without clear specs, developers guess and AI confidently generates the wrong code. The system grows use case by use case, and the time invested in writing a clear spec is saved multiple times during implementation.

#development #agile

Follow-Up

Bolt: Modernizing MySQL with TiDB to Scale Thousands of Microservices on AWS 👷‍♂️

I was fortunate enough to witness the migration from MySQL to TiDB at Bolt myself. Sevearl years ago we hit severe scaling limits with MySQL at 100 million users — adding a column to a 1TB table took a week. After working with Vitess for a while, we migrated to TiDB for its MySQL wire compatibility, horizontal scalability, and strong consistency via Raft. The results: 3:1 data compression ratio (3TB MySQL to 1TB TiDB), 35,000 queries per second, and seven clusters supporting core business services. The migration went smooth well too - each team migrated at their own pace and we completed the whole migration in like a year.

#db #migration #aws

We Built a Spreadsheet Engine from Scratch — Here’s What We Learned 🤟

GRID built one of the most sophisticated spreadsheet engines that runs entirely in the browser. The complexity is staggering: Excel has 510 functions, Google Sheets 494, and together there are 600+ unique operations to implement. A spreadsheet is essentially a program where users declare relationships in a 2D grid, and the engine must maintain deep dependency graphs with thousands of relationships. Their engine covers 398 functions handling 99%+ of real-world spreadsheets.

#architecture #engineering

Without Benchmarking LLMs, You’re Overpaying 👷‍♂️

Generic LLM benchmarks like GPQA or MMLU don’t predict performance on your specific task. The author benchmarked actual customer support prompts across 100+ models via OpenRouter and found cheaper alternatives delivering comparable quality — cutting costs by 80% and saving over $1,000 monthly. The key: collect real examples, define expected outputs, and score results using LLM-as-judge rather than trusting marketing leaderboards.

#ai #llm

You Are Here 🤟

Marc Brooker argues that AI has drastically reduced the cost of writing code, but building reliable end-to-end systems remains hard. He frames it through Amdahl’s Law: massive speedups in one area yield diminishing returns unless other bottlenecks are addressed.(sounds familiar to the Theory of Constraint, right?) The real opportunity lies not in clinging to traditional programming (the analog electronics mistake) but in leveraging AI tools to solve previously intractable integration and system design problems. Software’s first era has ended.

#softwareengineering #ai

Big thanks to Nikita, Constantin, Anatoly, Oleksandr, Dima, Pavel B, Pavel, Robert, Roman, Iyri, Andrey, Lidia, Vladimir, August, Roman, Egor, Roman, Evgeniy, Nadia, Daria, Dzmitry, Mikhail, Nikita, Dmytro, Denis and Mikhail for supporting the newsletter on Patreon! Alternatively, you can upgrade to the paid version and get access to the premium only posts!

Building compliance infrastructure starts with one thing above all else: making data ingestion easy. When we first launched, the only way to get data into our system was by uploading a CSV file. Since then, we’ve come a long way. Today, our clients can seamlessly connect to more than 700 data sources including Stripe, Google Sheets, Salesforce, cloud-hosted managed databases, AFAS, Booking Experts, XLSX uploads, and many more. How did we get there? That’s what this post is about.

Initial Attempt

Stripe became the first system to connect to. Our client used it to process payouts to their suppliers(no surprise!), so naturally Stripe was a perfect source of truth for DAC7 reports we create for our customers.

Stripe allows 2 forms of integrations: via an API Key and OAuth authorization for the approved Stripe applications. We are yet to become one, so we chose the API Key approach. Stripe has the root public/private key pair - which you’re not expected to share with anybody. However, Stripe has a feature of restricted API keys: you can create a read-only key with an access to certain Stripe collections like invoices, payouts, accounts and other ones you deem necessary.

It is indeed important to discuss how to store those keys securely, this is discussed later.

The API integration seems to be pretty straightforward: initialise a Stripe object with the obtained restricted key and call the appropriate API:

// query invoices from Stripe
(s: Stripe) => params => s.invoices.list(params)

Background Jobs

This works perfectly for small accounts with less than a hundred of entities, because Stripe limits the query to exactly 100 records. That means 2 things: first, the query should not request more than that, so querying all relevant invoices should be done in chunks; second, such approach will take minutes for a typical customer with tens of thousands of records. So we need to apply several mechanics at once:

1. Introduce parallelisation. It should not be a high number of parallel request, otherwise you will be rate limited.

2. Limit the time span to query. In order to make a report for a single year, you don’t need to query the whole history from the beginning.

3. The querying should not happen on the main request, so we moved the data import off to a job queue.

   

While the original scheme could be just fire request and wait for the response, the scheme with the background job requires a proper support in the database following the status of the job: started, in progress, failure, succeeded. We store this in the special table in DynamoDB and use it to provide appropriate UI.

We also implemented an integration with Google Sheets in the same manner. Although the implementation approach is the same, the Google Sheets import requires picking up what sheets to import from a file, which provide a convenient UI for.

Scaling Integrations

Our customer’s software is not limited to Stripe and Google Sheets. In real world our customers use a variety of Databases, ERP Systems(like Salesforce), Accounting Software, CRM Solutions(like Hubspot and Booking Experts) and bespoke software systems like AFAS.

One way to work with them is building the integrations one-by-one slowly increasing the number of integrations. This approach though does not scale well: it consumes the valuable time for building an integration. That’s why we started looking at solutions which would unlock the most popular connections right away.

Aside from several companies which focus on these integrations, we concluded that the best possible option for us is using Fivetran.

Fivetran is data movement platform which allows to ingest data from 700 different system types into a handful of destinations like Snowflake, S3 and others. While Fivetran solves the connection problem, we still had to integrate it into our import subsystem. Let’s find out what we did exactly.

Connections, Connectors and Destinations

Fivetran operates with two important entities: Connectors(also known as Connections) and Destinations.

The Connection is ultimately the configured data source. Let’s say you want to connect to a MySQL database in the AWS Cloud. You will need a connection host, port, username and password. The Database Admin should also whitelist the connection IPs so it can be established(do you allow anybody to connect to your DB? yeah, exactly, you don’t). Those are the parameters of the connection which should be provided while establishing the connection. Luckily, Fivetran provides a convenient wizard to do that with any connection.

Fivetran will then establish it’s own connection once the parameters are provided. We don’t store those parameters anywhere in Supplied system - luckily we don’t need to. We only require knowing our own Fivetran credentials, which are safely stored in AWS Secrets Manager.

And here’s where it gets interesting: Fivetran will let us know that the connection was established - but there is no data synced yet!

The required part for a synchronisation is the Destination: where the data goes. Fivetran supports S3 bucket as a destination, but it uses parquet files for that. However, we use parquet data for merged data, not the originally imported one. In order to have it uniformly, we want to translate it once the import is completed. The way we solve it, we wait for Fivetran to call our web hook to indicate the import is completed. With this call we trigger moving the data from an S3 bucket made as a Fivetran destination into the our common Data S3 bucket. Find the schema below:

Using Fivetran we can seamlessly connect to 700+ sources like Salesforce, SAP, Facebook, Paypal and hundreds of others. However, even Fivetran does not have all: for example we built a custom integration with AFAS and Booking Experts, but those integrations are pretty similar to Stripe with the difference that we don’t store a key, but rather using OAuth flow and store temporary access_token/refresh_token. Even using temporary credentials, we still want to store them securely.

Storing Credentials Securely

Storing the access tokens in the plain text in DynamoDB is not secure, despite having the built-in encryption at rest of the db itself - because the developers still can access those tokens. That’s why we use the security-in-depth approach: we secure every layer we can, and our customers value this truly. So how the data is protected in the end?

Let’s take a Booking Experts(BEX) example integration. In order to import data from the BEX itself you need to create an application at their marketplace. Each application has a list of used permissions - and our app use only a dozen of read-only permissions to access owners, invoices, rentable objects and other required data for reports like DAC7. The application is also being installed to a particular BEX account, so it can only access the data of said account.

So the app will only access the data of authorized accounts, and only according to granted permissions. The next level or protection is going with the OAuth, so the app’s client id and client secret are used to exchange the authorization code for the access_token and refresh_token. The access_token is short-lived and will expire within hours, after which we renew it using the refresh_token. And even those tokens should not be saved in the plain text. That’s why we encrypt them using a key, which we store in the AWS Key Management Services for maximum protection. So even in an event of an attacker accessing the database itself, they won’t be able to use whatever client keys are saved. Here’s the example from AWS themselves:

Summary

Supplied provides the Enterprise-level data sources integration connecting to hundreds of source systems in the secure manner. The design of the system also allowing to connect any additional connections if there is an API available in a scalable, performant and secure manner.

If you liked this article, press this like button, it helps understanding what kind of content is interesting.

Have a question? Leave a comment down below!

Leave a comment

Software Architecture Weekly is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

Business Oriented System Design Course

Wanna learn how to design systems which truly impact business and get promoted? Sign up for Business Oriented System Design Course: starts 4th of February. Begin the year with learning crucial skills and advance your career! 100+ engineers from Bolt, Booking.com, Semrush, TripleTen etc. who completed the course and brought their architecture skills to a new level.

Highlights

Scaling Postgres at OpenAI 👷‍♂️

OpenAI has 800 million customers across the globe. Surprisingly enough they manage to run it with a single master HA PostgreSQL setup offloading the read queries as much as possible to read replicas and having a hot standby for primary in case of a failure for fast recovery. More in their article.

#db #scalability #performance

Building effective AI Agents 👷‍♂️

AI Agents shows the biggest shift in the software development: instead of having a solid, predefined execution flow you let LLM decide what to do. However before building agents, you still need to master AI-driven workflow, where the flow is still predefined, but the program becomes more powerful due to the usage of LLM processing. Grab a manual from Antropic themselves to learn it!

#llm

The rise of industrial software 🍼

The industrialization allowed to craft goods(including growing food) in greater volume than before due to decreased cost and thus increased demand. The same happens with software: as the code is not required to be written manually anymore, the demand for software will blow up. Follow the article on future economics of software development.

Follow Up

Holistic Engineering for organic problem solving 🍼

The biggest issues in the sofware development don’t come from technical complexity: they come from the external forces like business trends, internal factors like organization design and other attributes. Go through the insightful infoq article on the topic.

#softwaredevelopment

Distributed Cache Invalidation with Redis and HybridCache

In-memory cache is nice, but what about a case when you have several instances of the application? Apparently, it can become quite tricky to solve. Fortunately, Redis has some mechanism which can help with it. To get to know which exactly, open the article.

#performance #consistency

Operational Maturity is a must now

2025 marks a borderline where observability ceases to be a technical concern to become a necessity for business to trust the underlying system and to have explainability in it. Andrew Mallaband elaborates on the issue.

#o11y #observability

Foundation vs Instruct vs Thinking Models

AI space evolves fast, but it is important to understand the difference in the models which can be used for AI-driven applications. Alex Ewerlöf wrote a short note on the difference between foundation, instruct and thinking models. Grab a read!

Alex Ewerlöf Notes

Foundation vs. Instruct vs. Thinking Models

If you are an engineering leader exploring LLMs, you have likely encountered a confusing naming convention on HuggingFace. You see Llama-3-8b (the Base model) and Llama-3-8b-Instruct…

Read more

8 months ago · 10 likes · Alex Ewerlöf

#ai

Big thanks to Nikita, Constantin, Anatoly, Oleksandr, Dima, Pavel B, Pavel, Robert, Roman, Iyri, Andrey, Lidia, Vladimir, August, Roman, Egor, Roman, Evgeniy, Nadia, Daria, Dzmitry, Mikhail, Nikita, Dmytro, Denis and Mikhail for supporting the newsletter on Patreon! Alternatively, you can upgrade to the paid version and get access to the premium only posts!