Post 28 covers two tools in the .NET diagnostics toolchain. dotnet-counters streams lightweight runtime metrics to the terminal — CPU usage, GC heap size, allocation rate, thread pool queue length — with zero performance overhead. dotnet-trace captures a .nettrace file during a 30-second peak load window. PerfView (Windows) or SpeedScope (cross-platform) opens that file as a flame graph showing exactly which function is consuming CPU at 200 RPS.

The answer — in this system — is TenantConfigStore.GetTenantConfigAsync. Every order creation request hits Azure Table Storage once for the tenant configuration lookup (the TenantAuthorizationHandler from Post 21). At 200 RPS that is 200 Table Storage round trips per second. Adding IMemoryCache with a 30-second TTL reduces that to one round trip every 30 seconds and drops p99 by approximately 40%.

No new code this post — only two commits: docs/profiling-notes.md (the profiling workflow) and a MemoryCache addition to TenantConfigStore. verify.ps1 grows to 61 checks.

Career skill this post adds: dotnet-counters + dotnet-trace + PerfView / SpeedScope flame graph analysis — the four-step .NET performance engineering workflow. Every Azure .NET senior engineer interview that asks “how did you diagnose a latency regression?” expects this answer: counters for situation awareness, trace for root cause, flame graph for the hot path, targeted fix, re-run load test. The TenantConfigStore cache optimisation pattern — per-key TTL cache for a lookup called on every authenticated request — is a recurring pattern in multi-tenant ASP.NET Core APIs.

• dotnet-counters — which counters to watch at 200 RPS and what each signals

• dotnet-trace — capturing a .nettrace file, which providers to enable

• Flame graph anatomy — width, height, hot leaves vs wide roots

• The hot path: TenantConfigStore → TableClient.GetEntityAsync at 200 RPS

• IMemoryCache addition — the fix and why 30-second TTL is appropriate

System state: where we are

The full Phase 1–6 stack is deployed: 7 services, 4 emulators, Dapr sidecars, 8 integration tests, NBomber load test. Post 28 adds profiling tooling and the first performance optimisation — a cache in TenantConfigStore. No new NuGet packages beyond what is already in the base library.

Starting point: git checkout post/28-perfview Install once: dotnet tool install -g dotnet-counters · dotnet tool install -g dotnet-trace Both are official Microsoft .NET diagnostics tools. Cross-platform.

Read more

The load test reveals something integration tests cannot: where the system degrades. At 50 RPS the p99 is 85ms. At 100 RPS it is 210ms. At 200 RPS it is 1100ms. That non-linear jump between 100 and 200 RPS is the signal — something saturates around 100 RPS. The HTML report produced by NBomber identifies whether the bottleneck is the Service Bus emulator CPU, the Azurite Table Storage connection pool in TenantAuthorizationHandler, or the JWT validation overhead accumulating under parallel requests.

NBomber is code-first. The load plan is a C# console application — not a JMeter .jmx XML file, not a Visual Studio .loadtest XML file. The scenario references the same appsettings.Development.json JWT configuration as the application. The Simulation.Inject model matches the arrival rate pattern of real HTTP traffic: N new requests per second, regardless of system speed.

verify.ps1 grows to 59 checks. Two new checks confirm the tests/LoadTests/ project exists and contains the NBomber package reference.

Career skill this post adds: NBomber load testing with stepped load simulations — Simulation.Inject for constant arrival rate, WithWarmUpDuration for JIT/connection stabilisation, and the interpretation of p50 vs p99 divergence under load. Every Azure .NET performance engineering role asks “how did you measure the throughput ceiling of the system?” The answer is a stepped load test with a clear bottleneck identification process.

• Simulation.Inject vs KeepConstant — arrival rate vs fixed concurrency, when each applies

• Warm-up duration — why first-JIT and cold connection overhead contaminates results without it

• p50 vs p99 divergence — the signal that a non-linear bottleneck exists

• Reading the NBomber HTML report — latency histogram, RPS achieved vs requested, error table

• Common bottlenecks on local emulators and their production equivalents

System state: where we are

Posts 24–25 added Dapr. Post 26 added Testcontainers integration tests. Post 27 adds NBomber as a second test type in tests/ — a console application that measures sustained throughput behaviour. The application under test is unchanged; the full Phase 1–5 stack runs via Aspire.

Starting point: git checkout post/27-nbomber Prerequisite: Aspire stack must be running (dotnet run --project src/AppHost). The load test acquires a JWT from UserService and then drives POST /orders/create on OrderService.

Read more

Post 26 adds integration tests. Not unit tests with mocked Azure SDK clients — mocking TableServiceClient tests that your code calls GetEntityAsync with specific arguments, not that it returns the right result when the table contains specific data. Integration tests with Testcontainers start a real mcr.microsoft.com/azure-storage/azurite Docker container inside the test process, run TenantConfigStore and OrderBlobStore against it, and confirm behaviour that matters: the path structure is correct, second writes are idempotent, unknown tenants return null.

Eight tests across two test classes. Each class starts one Azurite container, runs four tests against it, and stops the container. The same Azurite image used in docker-compose.yml and .WithDaprSidecar() runs in the tests. There is no configuration difference between local development and CI.

Career skill this post adds: Testcontainers integration testing for Azure SDK components — AzuriteContainer from Testcontainers.Azurite 3.9.0, IAsyncLifetime for container lifecycle, and the principle that integration tests should use the same Docker images as local development. Appears in senior .NET engineer roles that list “integration testing” and “CI/CD” alongside “Azure SDK.” The argument for Testcontainers over mocks — testing behaviour not implementation — is a system design interview answer for “how do you test code that uses external services?”

• Why mocks of Azure SDK clients give false confidence — what they test vs what they don’t

• Testcontainers.Azurite — the typed container builder and IAsyncLifetime lifecycle

• AzuriteContainer.GetConnectionString() — the bridge from container to SDK client

• Test isolation without container restart — unique IDs per test

• What the 8 tests cover: path convention, idempotency, null for unknown tenant, ContentType

System state: where we are

Phase 5 complete (Posts 24–25): Dapr service invocation and Dapr pub/sub. Phase 6 begins with testing infrastructure. The application code under test (TenantConfigStore, OrderBlobStore) has not changed since Posts 17 and 16. Post 26 adds tests/IntegrationTests/ as a new project that references the service projects directly.

Starting point: git checkout post/26-testcontainers Prerequisite: Docker must be running. Testcontainers pulls the Azurite image and starts containers. dotnet test tests/IntegrationTests must pass all 8 tests before running .\verify.ps1.

Read more

The key property of Dapr pub/sub: the component YAML (config/dapr/components/pubsub.yaml) is the only thing that changes between Redis locally and Azure Service Bus in production. The PublishEventAsync call — and the [Topic] subscriber attribute — are identical in both environments. The application code has no namespace.servicebus.windows.net, no SharedAccessKey, no ServiceBusClient. Those are broker configuration, not application code.

verify.ps1 grows to 55 checks. The final check calls /orders/create-dapr and verifies the “Order published via Dapr pub/sub” log record in the OTel Collector — confirming the Dapr sidecar received the publish call and routed it to the Redis component.

Career skill this post adds: Dapr pub/sub with PublishEventAsync and [Topic] subscription — the broker-agnostic event publishing pattern. The pubsub.yaml component swap (Redis → Azure Service Bus → Azure Event Hubs → RabbitMQ) is the key demonstration: the same application code works with four different brokers. Appears in Azure .NET distributed systems roles that list Dapr alongside “event-driven architecture.”

• Dapr pub/sub vs direct Azure Service Bus SDK — what changes, what stays the same

• pubsub.yaml component YAML — the three lines that change for production

• daprClient.PublishEventAsync — the publish call

• [Topic] attribute / .WithTopic() — the subscriber binding

• ACK/NACK via HTTP response codes — simpler than Service Bus CompleteAsync/AbandonAsync

System state: where we are

Post 24 added Dapr sidecar support (WithDaprSidecar()) to both UserService and OrderService. Post 25 adds the pub/sub building block on top of the same sidecars. The Redis component (dapr init installs a local Redis container automatically) handles message routing.

The Phase 2 Service Bus pipeline remains active and unchanged. Post 25 adds a parallel Dapr pub/sub path at /orders/create-dapr. Both paths publish OrderCreatedEvent. The goal is comparison, not replacement — at this stage in the series.

Starting point: git checkout post/25-dapr-pubsub Redis must be running (dapr init starts it automatically). Verify: docker ps | grep redis.

Read more

Post 24 introduces the Dapr Distributed Application Runtime — the sidecar architecture that moves infrastructure capabilities (service discovery, retries, circuit breakers) out of application code and into a process that runs alongside every service. Two packages, two lines in AppHost/Program.cs, one new endpoint. verify.ps1 grows to 53 checks.

Career skill this post adds: Dapr service invocation with DaprClient.InvokeMethodAsync — the pattern that replaces HttpClient for service-to-service calls in a Dapr-enabled microservice system. Appears on Azure .NET senior engineer roles listing Dapr or Microsoft.Extensions.Http.Resilience. The sidecar model explanation is a system design interview answer for “how do you build resilient microservices without coupling resilience to business logic?”

• The Dapr sidecar model — why it exists, what it handles, what it doesn’t

• WithDaprSidecar() in Aspire AppHost — the one-line addition per service

• DaprClient.InvokeMethodAsync — the application-side call

• Distributed trace propagation through the sidecar — cross-service span in Aspire Dashboard

• Prerequisites: dapr init and the daprd binary

System state: where we are

Phase 4 is complete (Posts 20–23). Phase 5 introduces Dapr as an infrastructure layer that augments the existing Service Bus and Event Hubs messaging with sidecar-managed service-to-service resilience.

Starting point: git checkout post/24-dapr-sidecar Prerequisites: dapr init must have been run. On Windows: winget install Dapr.CLI then dapr init. Verify: dapr --version.

Read more

Post 23 adds two documents that together answer that question. docs/sre-dashboard.json is an Azure Dashboard JSON template with eight tiles, each backed by a KQL query that was already in docs/kql-queries.md. docs/sre-runbook.md is the operational companion — for each tile, what the threshold is, what the root cause is, and what to do in the first five minutes.

Both documents are deployable artifacts. The dashboard imports into the Azure Portal via a JSON upload or deploys via az resource create. The runbook is the document that gets linked from the PagerDuty alert body and opened at 3 AM. Both live in docs/ and ship with the repository — the same git pull that deploys new application code also updates the operational documentation.

Career skill this post adds: SRE dashboard design thinking — which metrics to show (actionable, alert-triggering) vs which to hide (interesting, review-only). The four golden signals framework from the Google SRE book provides the structure. Every senior Azure SRE and platform engineer interview includes “walk me through the observability stack you would build for a distributed system.” The four golden signals + dashboard + runbook is the answer.

• The four golden signals: latency, traffic, errors, saturation — definitions and this system’s equivalents

• What belongs on an on-call dashboard vs a weekly review dashboard

• Azure Dashboard JSON structure — lenses, parts, positions, KQL-backed tiles

• The runbook structure — threshold, severity, immediate steps, root causes

• Deploying the dashboard via the Azure Portal and az CLI

The four golden signals

Google’s Site Reliability Engineering book defines four signals that, together, provide a complete picture of system health. Every production distributed system should expose all four. Post 23 maps each signal to the series’ existing telemetry:

1. Latency — how long does a request take?

• This system: p99 of /orders/create requests in App Insights requests table

• The right metric: p99, not average. Averages hide slow outliers. A p99 of 2000ms means 1% of users wait 2 seconds — which is a real problem that an average of 150ms would conceal.

• Alert threshold: > 1000ms for 5 consecutive minutes

2. Traffic — how many requests per unit time?

• This system: /orders/create count per 5-minute bucket

• Why traffic matters: a traffic drop is often the first signal of an upstream issue (API gateway, DNS, client app failure). The service appears healthy but orders are not arriving.

• Alert threshold: < 10% of 7-day rolling average for 15 consecutive minutes

3. Errors — what fraction of requests fail?

• This system: success == false in requests table + exception count in exceptions table

• Separate from latency: a fast error (50ms 500 response) is an error, not a slow request. Track both independently.

• Alert threshold: > 1% error rate for 5 consecutive minutes

4. Saturation — how “full” is the service?

• This system: Service Bus DLQ depth + Polly retry rate

• Why these two: the DLQ is the lagging saturation indicator (messages already failed), while Polly retry rate is the leading indicator (messages about to fail). Monitoring both gives early warning.

• Alert threshold: DLQ > 0 for any 5-minute bucket; Polly retries > 5/min for 10 minutes

Bridge from familiar: Windows Performance Monitor (PerfMon) custom views on a Windows server show CPU %, memory %, disk I/O, and network throughput — equivalent to the four golden signals for infrastructure. The counter log is equivalent to the App Insights timechart. The alert threshold in PerfMon is equivalent to the Azure Monitor alert rule that fires when a KQL query returns results. The four golden signals are the application-layer equivalent of the four infrastructure metrics every Windows administrator has been monitoring since NT 4.0.

Design dashboards for the person who opens them at 3 AM. That person is usually not the architect who knows the system intimately. They need: one tile per signal (not 30 tiles of interesting metrics), clear threshold labels, and a link to the runbook. “DLQ Depth” with no axis label or threshold annotation is useless. “DLQ Depth — alert if > 0 (see runbook)” is actionable. The eight tiles in docs/sre-dashboard.json each have a _comment field with the alert threshold.

Core concept: On-call dashboard vs weekly review dashboard

The wrong approach: put every interesting metric on one dashboard. Engineers who build the system know what every metric means. The on-call dashboard for someone unfamiliar with the system becomes noise — 30 tiles, all green, no idea which one would turn red first.

The right approach: two dashboards with different audiences.

On-call dashboard (this post): Eight tiles. Four golden signals only. Each tile has a threshold. Opening this dashboard takes 10 seconds to scan. If all eight tiles are green, the system is healthy. If any tile is red, the runbook tells you what to do.

Weekly review dashboard (not in this series, but the design principle): Tenant order volume trends, SLA percentile over the last 30 days, cost-per-1000-orders, exception rate comparison week-over-week. These are the metrics that inform architectural decisions and capacity planning. They do not belong on the on-call dashboard.

The distinction is temporal resolution: on-call dashboards show the last 1 hour with 5-minute resolution. Weekly review dashboards show the last 30 days with 1-day resolution. The same metric — error rate — appears in both, but the time window and alert purpose are different.

Bridge from familiar: A Windows Server farm administrator has two views: the System Center Operations Manager (SCOM) alert console (on-call) and the performance reporting dashboard (weekly review). The SCOM console shows only active alerts — the equivalent of the on-call KQL dashboard. The performance reports show capacity trends — the equivalent of the weekly review dashboard.

Core concept: Azure Dashboard JSON structure

Github Link:

https://github.com/sysdr/dotnet-system-design-p/tree/main/post-23-v3-source-code

The JSON schema:

{
  "type": "Microsoft.Portal/dashboards",
  "properties": {
    "lenses": {
      "0": {                          // first row of tiles
        "parts": {
          "0": {                      // first tile
            "position": { "x": 0, "y": 0, "colSpan": 6, "rowSpan": 4 },
            "metadata": {
              "type": "Extension/AppInsightsExtension/PartType/AnalyticsPart",
              "inputs": [
                { "name": "Query", "value": "requests | where ..." },
                { "name": "Title", "value": "p99 Latency" }
              ]
            }
          }
        }
      }
    }
  }
}

Position system: The dashboard grid is 12 columns wide. colSpan: 6 makes a tile half-width. colSpan: 12 makes a tile full-width. rowSpan: 4 is the standard chart height. Tiles are positioned with x (column) and y (row) — a colSpan: 6 tile at x: 0 and another at x: 6 are side by side.

Part type for KQL tiles: Extension/AppInsightsExtension/PartType/AnalyticsPart — this is the App Insights Logs tile type. The ComponentId input is the full Azure resource ID of the App Insights instance. The Query input is the KQL string.

Tokens requiring replacement before deployment:

• SUBSCRIPTION_ID — your Azure subscription GUID

• YOUR_RG — the resource group containing App Insights

• YOUR_APP_INSIGHTS — the App Insights resource name

# Replace tokens in PowerShell before deployment
$dashboard = Get-Content docs/sre-dashboard.json -Raw
$dashboard = $dashboard.Replace("SUBSCRIPTION_ID", $env:AZURE_SUBSCRIPTION_ID)
$dashboard = $dashboard.Replace("YOUR_RG",         $env:AZURE_RESOURCE_GROUP)
$dashboard = $dashboard.Replace("YOUR_APP_INSIGHTS", "hyperscale-appinsights")
$dashboard | Set-Content docs/sre-dashboard-deployed.json

Deployment:

az resource create \
  --resource-group YOUR_RG \
  --resource-type Microsoft.Portal/dashboards \
  --name hyperscale-sre-dashboard \
  --properties @docs/sre-dashboard-deployed.json

Working Demo Link:

Verify

.\verify.ps1

-- Post 23 checks (new — SRE dashboard) ----------------
  docs/sre-dashboard.json exists ... PASS
  docs/sre-runbook.md exists      ... PASS
========================================================
 All 51 checks passed. Post 23 complete.

Read more

The claim is not marketing. AppHost/Program.cs is a dependency graph. The graph says: SQL Edge must be healthy before the Service Bus emulator starts. The Service Bus emulator must be healthy before OrderService starts. UserService and OrderService must share the same JWT signing key. OrderService must know UserService‘s URL for Phase 5 Dapr integration. Every one of those relationships is now expressed as a compile-time-checkable C# declaration, not as a YAML string that fails at runtime.

WaitFor() is the first new API. It replaces the 35-second Start-Sleep that has been in every verify.ps1 since Post 9. WithEnvironment("Jwt__SigningKey", key) is the second — it injects the shared JWT key from the AppHost into both services, so a key change requires one edit in one place. WithReference(userService) is the third — it injects UserService‘s URL into OrderService via Aspire’s service discovery conventions.

The Aspire manifest — generated by dotnet run --project src/AppHost --publisher manifest — is JSON that describes the same graph for deployment tools: azd for Azure Container Apps, Aspirate for Kubernetes Helm charts, and the Bicep generator that Post 35 uses.

Career skill this post adds: .NET Aspire AppHost IaC patterns — WaitFor, WithReference, WithEnvironment, and the manifest. Appears on every Azure .NET role that lists .NET Aspire in its stack requirements. The AppHost pattern is the mechanism that makes microservice local development tractable: one dotnet run command for the entire system, with correct startup ordering and environment configuration, without manual port management or YAML fragility.

• WaitFor(resource) — explicit startup ordering vs Start-Sleep polling

• WithEnvironment("Section__Key", value) — double-underscore nested config injection

• WithReference(service) — URL injection for service-to-service discovery

• The Aspire manifest — the deployment artifact generated from the same graph

• Docker Compose comparison table — depends_on, environment, networks equivalents

System state: where we are

AppHost/Program.cs is rewritten in Post 22. All functionality is identical — the services start in the same order, the same configuration is applied. What changes: the startup ordering is now explicit (WaitFor) rather than implicit (35-second sleep), the JWT key is injected by the AppHost rather than read only from appsettings.Development.json, and the service dependency graph is type-safe (renaming a resource breaks compilation, not runtime).

Starting point: git checkout post/22-aspire-iac Run .\verify.ps1 — all 47 Post 1–21 checks must pass. The 49th check verifies WaitFor() is in AppHost/Program.cs; the 48th verifies Jwt__SigningKey environment injection.

Read more

Post 21 closes that gap with a custom authorization policy. The RequireEnabledTenant policy combines the Post 20 JWT validation with a runtime lookup in the Post 17 TenantConfigStore: does this tenant exist, and is it enabled? If the Table Storage entry says IsEnabled = false, the request returns 403 Forbidden — even with a fully valid JWT.

This post also covers the production upgrade from the local symmetric key to Microsoft.Identity.Web — the NuGet package that wraps JwtBearer for Azure AD / Entra ID, eliminating the shared secret and replacing it with OIDC discovery. DefaultAzureCredential, the credential chain used for managed identities, is covered here because it is the same pattern that connects Microsoft.Identity.Web in AKS to TenantConfigStore in Azure Table Storage without any stored credentials.

Career skill this post adds: ASP.NET Core custom authorization policies — IAuthorizationRequirement + AuthorizationHandler<T> + AddAuthorization(options => options.AddPolicy(...)). The three-type pattern is the correct approach when authorization logic requires DI-injected services. It separates the what (requirement) from the how (handler) from the where (policy registration). Appears in every enterprise ASP.NET Core codebase that has multi-tenant or role-based access control beyond simple [Authorize(Roles = "Admin")].

• Three authorization policy types: role-based, claims-based, resource-based — when each applies

• IAuthorizationRequirement + AuthorizationHandler<T> — the two types and why both are needed

• context.Succeed(requirement) vs context.Fail() — the explicit contract of authorization handlers

• Microsoft.Identity.Web — what changes and what stays the same when upgrading from local JWT

• DefaultAzureCredential — the credential chain for managed identities

System state: where we are

OrderService has JWT bearer validation (Post 20) and TenantConfigStore (Post 17). Post 21 connects them: the JWT claim tenant_id is looked up in TenantConfigStore at authorization time. If the tenant is not found or IsEnabled = false, the request is denied before the handler runs.

Starting point: git checkout post/21-custom-auth-policy Run .\verify.ps1 — all 45 Post 1–20 checks must pass. Checks that call /orders/create use the Get-Token helper function that calls /auth/token with the tenant-01 defaults.

Read more

Post 20 adds JWT bearer authentication to OrderService. Unprotected POST to /orders/create → 401 Unauthorized. POST with a valid Authorization: Bearer {token} header → 200, message published to Service Bus. The token is issued by a new /auth/token test endpoint in UserService — a stand-in for the Azure AD token issuance that happens in production.

Two packages, one configuration change, two source files. The local development path uses a symmetric HMAC-SHA256 signing key in appsettings.Development.json. The production path replaces four lines with Microsoft.Identity.Web and an AzureAd configuration section — no code change, no new package, no subscription required to build and test it locally.

Career skill this post adds: JWT bearer authentication in ASP.NET Core 8 Minimal APIs — builder.Services.AddAuthentication().AddJwtBearer(), app.UseAuthentication(), .RequireAuthorization(). The OTel event hooks that propagate claim values to the Aspire Dashboard trace are the production debugging pattern that turns a 401 from “unknown failure” to “specific validation error in the span.” Appears on every Azure .NET senior role that lists API security alongside observability.

• JWT structure — header, claims payload, signature — and what [Authorize] verifies

• AddAuthentication().AddJwtBearer() — the three-line middleware registration

• TokenValidationParameters — the six properties and what each does

• Local symmetric key → production OIDC discovery — the one-section configuration swap

• OTel event hooks on OnAuthenticationFailed and OnTokenValidated

• .RequireAuthorization() vs .AllowAnonymous() — selective endpoint protection

System state: where we are

Phase 3 is complete. OrderService has Service Bus publishing, Event Hubs dual-streaming, Blob Storage writes, Table Storage config lookups, full App Insights telemetry, and zero authentication. Post 20 adds the security layer without changing any of the messaging or storage code.

Starting point: git checkout post/20-jwt-auth Run .\verify.ps1 — all 43 Phase 1–3 checks must pass. Important: verify.ps1 checks 17–29 now fetch a token from /auth/token before calling /orders/create. If /auth/token fails, those checks will fail. Confirm UserService is healthy before running the full suite.

Read more

Post 19 covers what makes KQL unique: the operators with no SQL equivalent. make-series converts a tabular result into a time-series array. series_decompose_anomalies runs a machine-learning decomposition against that array and returns which points are statistically anomalous. mv-expand explodes array columns back into rows so you can filter on the anomaly scores. And workspace() lets a single query span multiple Log Analytics workspaces — the multi-region observability join.

Post 19 also covers Sentinel detection rules — the KQL queries that run on a schedule and create Incidents when their results are non-empty. The five rules in docs/sentinel-detection-rules.md cover brute-force login detection, anomalous order volume, DLQ accumulation, elevated DeliveryCount, and Polly retry storm proximity. All five have App Insights equivalents testable against real series telemetry without a Sentinel licence.

/users/login-failed is added to UserService — a new endpoint that generates failed login records for the brute-force detection exercise. Call it 10 times in rapid succession; the KQL rule returns the IP and user count that triggered the threshold.

Career skill this post adds: make-series + series_decompose_anomalies — the KQL pattern that appears in advanced Azure Monitor alert rules and Sentinel hunting queries. Sentinel SIEM KQL authoring appears on Azure security engineer and senior architect job specifications. The detection rule pattern — “query on a schedule, alert if rows returned” — is the same structure used in every enterprise SIEM including Splunk, Elastic SIEM, and IBM QRadar. Learning it in Sentinel transfers across products.

• make-series — converting tabular counts into a time-series array

• series_decompose_anomalies — ML anomaly detection in one function call

• mv-expand — exploding array results back into filterable rows

• workspace() and cluster() — cross-workspace and cross-cluster queries

• Sentinel detection rules — the “query on schedule → alert if non-empty” pattern

System state: where we are

Phase 3 is complete after Post 19. The documentation set (docs/) now contains:

• telemetry-schema.md — the OTel attribute contract

• storage-decision.md — 5-service decision framework

• blob-lifecycle-policy.json — 4 lifecycle rules for Azurite containers

• kql-queries.md — 12+ production App Insights queries

• kql-operators-reference.md — operator quick reference

• sentinel-detection-rules.md — 5 Sentinel/App Insights detection rules ← Post 19

Starting point: git checkout post/19-kql-advanced Run .\verify.ps1 — all 41 Post 1–18 checks must pass. Run /users/login-failed 10 times to generate exercise data before starting the brute-force rule exercise

Read more

Post 18 teaches you the query language that reads them.

KQL is a read-only query language designed for time-series telemetry. The same pipe syntax — table | operator | operator | operator — works in Application Insights, Azure Monitor Log Analytics, Azure Data Explorer, and Microsoft Sentinel. The table names change. The operators do not.

Post 18 adds docs/kql-operators-reference.md (the operator quick reference added to the repository alongside the production queries in docs/kql-queries.md) and /orders/metrics (a new endpoint that generates the Tenant stats log records the lesson’s exercises are built around). Three concept sections cover the pipe operator, time-based queries, and aggregation with rendering. The exercises build five queries from scratch.

Career skill this post adds: KQL query writing — the skill that separates engineers who use Application Insights from engineers who understand it. Post-incident reviews, capacity planning, cost analysis, and alert rule authoring all require KQL. It appears on Azure senior engineer job specifications alongside “Azure Monitor” and “Application Insights” as a required skill, not a nice-to-have.

• The pipe operator and execution order — why the order of where | project | summarize matters

• ago(), bin(), datetime_diff() — the time functions that make time-series queries possible

• summarize count() by bin(timestamp, 1h) — the canonical volume-over-time query

• percentile(duration, 99) — p99 latency without downloading raw data

• render timechart — one line that turns a KQL result into a chart

• let statements — named subqueries that make complex queries readable

System state: where we are

Post 15 mapped the five App Insights tables. Post 18 teaches the queries. The system has been generating real telemetry since Post 1 — every order created, every DLQ entry, every Polly retry, every EH checkpoint update is in App Insights. The exercises in this post are written against that real data.

Starting point: git checkout post/18-kql-fundamentals Run .\verify.ps1 — all 39 Post 1–17 checks must pass. Then call /orders/metrics a few times to generate the Tenant stats telemetry the exercises use.

Read more

Two deliverables, two storage concerns. The lifecycle policy (docs/blob-lifecycle-policy.json) is four JSON rules that run in Azure Storage without any application code — a cron job built into the storage layer itself. The tenant configuration store (TenantConfigStore) uses Azurite’s Table endpoint (port 10002, already running) to store per-tenant rate limits and feature flags as Table entities with O(1) point lookup.

Post 17 closes out Phase 3’s storage layer. After Post 17, the system has a storage service for every data type in docs/storage-decision.md: Service Bus for commands (Phase 2), Event Hubs for events (Phase 2), Blob Storage for objects (Post 16), Table Storage for key-value (Post 17). Phase 8 adds ADX for time-series. Every service has a local Docker emulator and a one-line connection string swap for production.

Career skill this post adds: Azure Blob lifecycle policies and Azure.Data.Tables SDK — TableServiceClient, TableClient, ITableEntity, GetEntityAsync, QueryAsync, UpsertEntityAsync. The PartitionKey design question (”how to distribute data across storage nodes while enabling efficient queries”) is asked in every Azure senior engineer interview that mentions Table Storage. The lifecycle policy JSON structure appears in infrastructure-as-code reviews whenever Blob Storage has more than one tier.

• Blob lifecycle policy rules — prefixMatch, tierToCool, tierToArchive, delete, daysAfterModificationGreaterThan

• Azure.Data.Tables hierarchy — TableServiceClient → TableClient → TableEntity

• PartitionKey design — co-location vs isolation trade-off

• GetEntityAsync vs QueryAsync — O(1) lookup vs partition scan

• UpsertEntityAsync with Replace vs Merge — full replace vs field-level update

System state: where we are

Azurite has been running since Post 13 (checkpoint store). Post 16 added the processed-orders Blob container. Post 17 adds:

1. docs/blob-lifecycle-policy.json — four lifecycle rules applied in Azure Storage

2. TenantConfigStore — reads/writes Table entities in Azurite’s Table endpoint (port 10002)

3. Azure.Data.Tables 12.8.3 in OrderService.csproj

Starting point: git checkout post/17-table-storage Run .\verify.ps1 — all 37 Post 1–16 checks must pass.

Read more

Service Bus is not a database. It delivers commands and discards them on success. Event Hubs retains events for 90 days and then discards them too. Neither service was designed to answer the question: “Show me the processed state of order abc123 from last Tuesday.”

Post 16 answers that question with Azure Blob Storage. When OrderProcessorWorker completes a message, it writes a processed order record to Azurite as a JSON blob at processed-orders/{year}/{month}/{tenantId}/{orderId}.json. The record persists indefinitely. The path structure enables lifecycle policy tiering (year/month prefix) and per-tenant GDPR deletion (tenantId segment) — both patterns run against Azurite locally before touching Azure.

Azure.Storage.Blobs 12.21.2 is the only new NuGet package. The blob client hierarchy, the CreateIfNotExistsAsync startup pattern, blob metadata, and the “Complete first, write second” ordering rule — these are the four concepts Post 16 teaches. All four are verifiable in Azurite with no Azure subscription.

Career skill this post adds: Azure.Storage.Blobs SDK — BlobServiceClient, BlobContainerClient, BlobClient. Appears on every Azure .NET role that mentions blob storage, export pipelines, or document storage. The CreateIfNotExistsAsync idempotency pattern and the overwrite: false duplicate protection are the two most common interview questions about Blob Storage in a distributed system.

• The BlobServiceClient → BlobContainerClient → BlobClient hierarchy

• CreateIfNotExistsAsync — idempotent container setup at startup

• Blob path structure as a query and lifecycle surface ({year}/{month}/{tenantId}/{orderId})

• Blob metadata — the properties you can filter on without downloading the blob

• “Complete first, write second” — the ordering rule for Service Bus + Blob integration

System state: where we are

Post 13 already added Azurite to docker-compose.yml for Event Hubs checkpoint storage. The eh-checkpoints container is there. Post 16 adds a second container — processed-orders — for business data. Same Azurite instance, same connection string (UseDevelopmentStorage=true), second purpose.

Starting point: git checkout post/16-azurite-blob Run .\verify.ps1 — all 35 Post 1–15 checks must pass before adding Post 16 components.

Read more

Since Post 2, every request to UserService and OrderService has been landing in Application Insights. Every exception, every dependency call, every log record — all flowing via UseAzureMonitor() to an App Insights workspace. Fourteen posts of data. Most engineers check App Insights when something is broken. Post 15 covers what is in those tables all the time, and the KQL queries that surface problems before they become incidents.

App Insights organises incoming telemetry into five tables: requests, dependencies, exceptions, traces, and customEvents. Post 15 maps each table to the code that populates it, shows the KQL queries that answer the most common production questions for each, and demonstrates how operation_Id — the correlation key the OTel SDK writes automatically — links all five tables back to a single trace visible in the Aspire Dashboard.

Post 15 also adds an ActivityEvent to OrderProcessorWorker: a customEvent called OrderProcessed that appears in the App Insights customEvents table and carries business properties — OrderId, TenantId, and DeliveryCount — as structured dimensions queryable in KQL.

Career skill this post adds: Application Insights KQL querying — the first skill most Azure teams ask about in observability interviews. “Show me how you would find all failed requests for a specific tenant in the last hour” is a question with a specific KQL answer, and engineers who can write it in 30 seconds stand out from those who can only click through the portal UI.

• The five App Insights tables and what each receives from the series’ OTel setup

• operation_Id — how one field links requests, dependencies, exceptions, traces, and custom events

• KQL patterns for each table: latency percentiles, error regression detection, cross-service exception grouping

• ActivityEvent as the App Insights customEvents primitive

• Adaptive sampling and the itemCount field — how to detect what is being excluded

System state: where we are

Phase 3 has started. Post 14 added the storage decision framework. Post 15 goes deep on the storage service that has been running since Post 2: Application Insights. No new infrastructure is added — the App Insights workspace and OTel instrumentation are already in place.

What Post 15 adds:

• activity.AddEvent("OrderProcessed") in OrderProcessorWorker — a customEvent with business dimensions

• docs/appinsights-queries.md — 20+ production KQL queries across all five tables

Starting point: git checkout post/15-appinsights-deep-dive Run .\verify.ps1 — all 33 Post 1–14 checks must pass before adding Post 15 components.

Read more

• Accept order commands via Service Bus and process them exactly once

• Stream order events to Event Hubs for fan-out consumption

• Monitor queue depth via sb.orders.deadletter_messages OTel gauges

• Resume after restarts at the last Azurite checkpoint position

What it cannot yet do: persist anything durably. The order is processed. The event is consumed. But where does the order record live? Where does the processed telemetry go for long-term analysis? Where are the audit log entries stored?

Phase 3 answers those questions across six posts. Post 14 maps the terrain before building in it: five Azure storage services, what each is designed for, when to choose each, and which emulators are already running.

Career skill this post adds: Azure storage service selection — the most common architecture design interview question for senior Azure .NET roles. “Which storage service would you use for X?” appears in every Azure architecture assessment. The decision tree in this post is the framework that turns that question from guesswork into a structured answer.

• Azure Service Bus vs Event Hubs vs Blob vs Table vs ADX — the data-type taxonomy

• The emulator landscape: what runs locally, what the connection strings look like

• docs/storage-decision.md — the living reference added to the repository

• The series progression: which posts cover which services

• The one question to ask first before choosing any storage service

System state: where we are

Phase 2 is complete. Two local emulators are running: the Service Bus Emulator (port 5672) and the Event Hubs Emulator (port 5673). Azurite is already running from Post 13’s checkpoint store (port 10000). The Kusto Emulator runs in Posts 40–43. Post 14’s code addition is docs/storage-decision.md — a reference that accumulates as each service is added.

Starting point: git checkout post/14-storage-decision Run .\verify.ps1 — all 31 Post 1–13 checks must pass before adding Post 14 components.

Read more

Post 13 fixes it. EventProcessorClient replaces EventHubConsumerClient. Azurite provides the checkpoint store — a local Azure Blob Storage emulator that stores one small blob per partition: the last successfully processed Offset and SequenceNumber. On restart, EventProcessorClient reads those blobs and calls ProcessEventAsync starting from Offset + 1. Replay cost: at most one event — the event being processed when the crash or restart occurred.

Post 13 also formalises the TenantId partitioning that Post 12 started. PartitionKey = tenantId in the producer routes all events for a tenant to the same partition. PartitionInitializingAsync logs which partition each processor instance claims. In a multi-instance deployment (Post 37 with KEDA), four processor instances each own one of the four partitions — each instance processes exactly one tenant shard in order, without coordination.

Career skill this post adds: EventProcessorClient + checkpoint store integration — the production Event Hubs consumer pattern. EventProcessorClient with Azure Blob Storage checkpointing is the standard for any .NET team processing Event Hubs at scale. It is the counterpart to Service Bus AutoCompleteMessages = false: the explicit acknowledgement pattern for streaming consumers. Appears on senior Azure .NET engineer role specifications alongside ServiceBusProcessor.

• EventProcessorClient vs EventHubConsumerClient — when to use each

• UpdateCheckpointAsync — per-event vs per-N-events, the cost trade-off

• Azurite as the checkpoint store — one blob per partition, same SDK as production

• PartitionInitializingAsync and PartitionClosingAsync — the partition lifecycle callbacks

• Consumer group order-processor — dedicated, independent from the $Default group Post 12 used

System state: where we are

Post 12 added Event Hubs dual-publish and a basic EventHubConsumerClient consumer. Post 13 replaces that consumer with the production-grade EventProcessorClient, adds Azurite as the checkpoint store, and uses the order-processor consumer group (separate from $Default).

After Post 13, Phase 2 is complete. Both messaging pipelines are production-patterned:

• Service Bus: ServiceBusProcessor + AutoCompleteMessages = false + Polly + MaxDeliveryCount: 3 + DLQ monitoring

• Event Hubs: EventProcessorClient + Azurite per-event checkpoint + order-processor consumer group + PartitionKey = TenantId

Starting point: git checkout post/13-tenantid-partitioning Run .\verify.ps1 — all 29 Post 1–12 checks must pass before adding Post 13 components.

Read more

AutoCompleteMessages = false in ServiceBusProcessorOptions is not a best practice. It is the only setting that gives your handler meaningful control over what happens to a message when something goes wrong.

With AutoCompleteMessages = true — the default — the SDK calls CompleteMessageAsync immediately after ProcessMessageAsync returns, regardless of whether an exception was thrown inside the handler and caught by a try/catch you added. The message is gone. Nothing in the Service Bus logs, nothing in Application Insights, nothing in the DLQ. The order was lost silently.

The scenario:

// AutoCompleteMessages = true (the dangerous default)
processor.ProcessMessageAsync += async args =>
{
    try
    {
        await SaveToDatabase(order);
    }
    catch (Exception ex)
    {
        logger.LogError(ex, "Database save failed. {OrderId}", order.OrderId);
        // You think: "I handled this."
        // The SDK thinks: "ProcessMessageAsync returned. Complete the message."
        // Result: message completed, order lost, no DLQ entry, no retry.
    }
};

With AutoCompleteMessages = false:

// AutoCompleteMessages = false (correct)
processor.ProcessMessageAsync += async args =>
{
    try
    {
        await SaveToDatabase(order);
        await args.CompleteMessageAsync(args.Message); // only called on success
    }
    catch (Exception ex)
    {
        logger.LogError(ex, "Database save failed. {OrderId}", order.OrderId);
        // ProcessMessageAsync throws → SDK calls AbandonMessageAsync
        // DeliveryCount increments → Polly retries (Post 10)
        // After MaxDeliveryCount: DLQ entry with DeadLetterReason
        throw;
    }
};

The difference: one setting. The consequence: the difference between silent data loss and a recoverable failure with full audit trail.

Post 8 of the series sets AutoCompleteMessages = false from the start and never changes it. Post 10 adds Polly so that AbandonMessageAsync is only called after retries are exhausted. The combination — manual complete + Polly retry — is the correct pattern for any production Service Bus consumer.

────────────────────────────────────────────────────── THREE THINGS WORTH YOUR TIME THIS WEEK ──────────────────────────────────────────────────────

1. Azure Service Bus message settlement — the full settlement model https://learn.microsoft.com/en-us/azure/service-bus-messaging/message-transfers-locks-settlement#settlement-of-transfers Covers all four settlement operations (Complete, Abandon, Defer, DeadLetter) and exactly what each does to the message’s position in the queue. The Defer operation is one the series has not covered yet — it moves the message off the main queue into a deferred state, addressable only by sequence number. Useful for out-of-order processing.

2. Polly v8 breaking changes from v7 https://www.pollydocs.org/migration/v8 If your team is upgrading from Polly v7 to v8 (which Microsoft.Extensions. Http.Resilience uses), this page covers the renamed types and the new ResiliencePipelineBuilder pattern. The most common breakage: Policy.Handle is now new PredicateBuilder().Handle. The mental model is the same; the syntax changed.

3. What’s new in .NET Aspire 8.3.0 https://learn.microsoft.com/en-us/dotnet/aspire/whats-new/aspire-sdk-8-3 The series uses Aspire 8.3.0 throughout. The release notes cover the new Azure Service Bus integration (Aspire.Azure.Messaging.ServiceBus) that Post 7 does not use yet — it handles connection string injection automatically without the manual configuration in appsettings.Development.json. Post 11 adopts this integration.

──────────────────────────────────────────────────────ONE QUESTION ──────────────────────────────────────────────────────

Phase 3 starts after Phase 2 wraps. Phase 3 is storage and observability: Application Insights deep dive, Azurite (local Blob Storage), and KQL — the query language that works in Application Insights, Azure Monitor, Azure Data Explorer, and Microsoft Sentinel.

The KQL post is the one I get asked about most.

Before I write it: what do you actually use KQL for right now?

A) Debugging — finding a specific request or exception B) Alerting — writing detection rules for Azure Monitor C) Dashboards — building workbooks in App Insights D) Reporting — querying custom events for business metrics E) Nothing yet — KQL is new to me

Hit reply. The answer shapes how I write the KQL posts (Posts 18–19 in the series).

──────────────────────────────────────────────────────

See you next Sunday,

Distributed Systems with .NET distrib-dotnet.substack.com

War Story #3 was about checkpointing. After a partition rebalance, the consumer was 20 minutes behind — not because it was slow, but because it started reading from a 60-second-old checkpoint. The event hub had accumulated 20 minutes of events during the rebalance. The consumer had no way to know how far behind it was.

Post 12 adds the Azure Event Hubs Emulator: a free Microsoft Docker container that runs a full Event Hubs namespace locally. By the end, /orders/create dual-publishes to both the Service Bus orders queue (command, exactly-once delivery) and the Event Hubs order-events hub (event, fan-out to multiple consumer groups, retained for replay). The OrderEventHubConsumer reads from the $Default consumer group and logs each event with its PartitionId, Offset, and SequenceNumber — the three values that define where a consumer is in a partition’s stream.

Post 13 adds EventProcessorClient with Azurite checkpoint storage and TenantId-based partition routing. Post 12 establishes the foundation.

Career skill this post adds: Azure Event Hubs SDK — EventHubProducerClient, EventHubConsumerClient, partition keys, consumer groups, and the difference between offset-based and time-based checkpointing. Event Hubs appears on senior Azure .NET engineer job specifications alongside Service Bus, and the “when to use which” decision (covered in this post) is a common interview question.

• Event Hubs vs Service Bus — the data model difference, not just a feature comparison

• The append-only partition — why Offset and SequenceNumber are not the same thing

• EventHubProducerClient with CreateBatchAsync and PartitionKey

• EventHubConsumerClient.ReadEventsAsync() — all partitions, one stream

• Per-event vs time-based checkpointing — the War Story #3 fix

• Dual-publish: Service Bus for commands, Event Hubs for events

System state: where we are

Phase 2 Service Bus pipeline is complete (Posts 7–11). Post 12 adds the Event Hubs pipeline in parallel. Both pipelines share the same OrderCreatedEvent contract from src/Contracts/. Both use BinaryData.FromObjectAsJson() with the source generator. The event serialisation knowledge from Post 7 carries forward without change.

After Post 12 the system has two independent publish paths from OrderService:

1. Service Bus orders queue → OrderProcessorWorker — transactional, exactly-once, DLQ on failure

2. Event Hubs order-events hub → OrderEventHubConsumer — streaming, fan-out, retained

Starting point: git checkout post/12-eventhubs-emulator Run .\verify.ps1 — all 27 Post 1–11 checks must pass before adding Post 12 components.

Read more

The downstream payment API returned HTTP 503 for 2 seconds. Not a crash — a blip. The infrastructure team confirmed later: it was a rolling restart that lasted 1.8 seconds.

Without Polly, OrderProcessorWorker threw HttpRequestException. The SDK called AbandonMessageAsync. DeliveryCount became 2. One more transient failure and the message would land in the DLQ with MaxDeliveryCountExceeded — for a fault that had nothing to do with the message.

With Polly — one line of configuration — the handler retried internally, waited 1 second, tried again, succeeded. The SDK called CompleteMessageAsync. DeliveryCount stayed at 1. The 2-second payment API blip was invisible to Service Bus.

The distinction matters because MaxDeliveryCount is your last-resort protection against poison messages — messages that will never succeed regardless of retries. If transient failures eat that budget, MaxDeliveryCount: 3 means three transient blips send a message to the DLQ. Polly reserves the DeliveryCount budget for genuine failures.

Post 10 adds a ResiliencePipeline to OrderProcessorWorker: exponential backoff with jitter for transient HTTP failures, a 25-second timeout per attempt (shorter than the 30-second LockDuration), and the TRANSIENT_ prefix test endpoint that shows — with log records in the Aspire Dashboard — exactly when Polly retries and when the SDK’s DeliveryCount changes.

Career skill this post adds: Microsoft.Extensions.Http.Resilience and ResiliencePipelineBuilder — the Polly integration shipped with .NET 8. These appear on every Azure .NET senior engineer job specification that mentions Service Bus, Event Hubs, or any downstream HTTP dependency. The MaxConcurrentCalls × Polly retry interaction is a common interview topic.

• The resilience pipeline — retry, timeout, and the order they execute in

• DeliveryCount vs Polly retries — the exact accounting of who increments what

• UseJitter = true — why randomised delays prevent the thundering herd problem

• AddTimeout(TimeSpan.FromSeconds(25)) — why the timeout must be less than LockDuration

• ShouldHandle — which exceptions Polly retries vs which propagate immediately

System state: where we are

Posts 7–9 built the Service Bus messaging loop: publish (Post 7), consume (Post 8), and DLQ forensics (Post 9). Post 10 adds the resilience layer — the in-handler policy that separates transient failures (retry automatically) from permanent failures (propagate to the SDK, let DeliveryCount increment).

After Post 10, the error handling hierarchy is:

1. Polly catches: transient HttpRequestException, TimeoutRejectedException — retried internally, DeliveryCount unchanged

2. SDK catches: unhandled exceptions after Polly retries exhausted — DeliveryCount increments

3. Explicit dead-letter: deserialisation failures, poison messages — DeadLetterMessageAsync called directly, DeliveryCount irrelevant

Starting point: git checkout post/10-resilience Run .\verify.ps1 — all 23 Post 1–9 checks must pass before adding Post 10 components.

Read more

The DLQ is not a failure mode. It is a feature. Service Bus created it to protect you from permanently losing messages you cannot process. The failure mode is leaving the DLQ unmonitored.

Post 9 is War Story #2 run locally, step by step. You send a deliberately poison message — an order with an OrderId prefixed POISON_ that the handler throws on every attempt. You watch DeliveryCount climb from 1 to 2 to 3. After MaxDeliveryCount (3) exhaustions, Service Bus moves the message to the DLQ. You inspect the DLQ using ServiceBusReceiver in dead-letter mode, read the DeadLetterReason that explains why the message was dead-lettered, and replay it to the main queue using a PowerShell script — the same script that cleared War Story #2’s 40,000 messages in production.

This post also introduces Service Bus topics and subscriptions — the fan-out pattern that lets one published message reach multiple subscribers with SQL filter routing.

Career skill this post adds: Service Bus DLQ forensics — inspecting dead-letter queues, reading DeadLetterReason and DeadLetterErrorDescription, and replaying DLQ messages safely. This is a common interview question and a production skill for any .NET engineer operating Azure Service Bus.

• DeliveryCount — what it counts, how it increments, and why MaxDeliveryCount: 3 is safer than the default of 10

• ServiceBusReceiver in dead-letter mode — same API as a normal receiver, different constructor argument

• Topics vs queues — when to use each; the SQL filter subscription pattern

• The replay script — moving dead-lettered messages back to the main queue

• RequiresDuplicateDetection — how to prevent the same message being processed twice during replay

System state: where we are

Post 8 gave OrderProcessorWorker the consumer loop. Post 9 adds the failure path: a message that cannot be processed, exhausts its delivery attempts, and lands in the DLQ. The DLQ always existed — it is a built-in sibling of every queue and subscription. Post 9 makes it visible and shows how to use it as a diagnostic tool, not a graveyard.

Starting point: git checkout post/09-servicebus-emulator Run .\verify.ps1 — all 21 Post 1–8 checks must pass before adding Post 9 components.

Read more