The mentorship is tailored for interview preparation, backend engineering, distributed systems, AI infrastructure, and building a strong systems-thinking foundation.

📩 Contact: programmingappliedai@gmail.com

1. Functional Requirements

1. Unified LLM API

   • Provide a common API interface for all LLM providers.

   • Client should not need to know whether the request goes to OpenAI, Anthropic, Gemini, or an open-source model.

2. Intelligent Model Routing

   • Route requests based on:

     • Model capability

     • Latency

     • Cost

     • Provider availability

     • Context-window requirements

     • Tenant preferences

   • Example: route simple queries to a cheaper model and complex queries to a stronger model.

3. Fallback

   • If the selected provider/model fails, automatically route to another eligible model/provider.

   • Example:

OpenAI → Anthropic → Gemini → Self-hosted

1. Rate Limiting & Quotas

   • Support limits at:

     • Tenant/user level

     • API-key level

     • Provider level

     • Model level

   • Support both RPM and TPM limits.

2. Cost-Aware Scheduling

   • Track input/output tokens and model prices.

   • Prefer the cheapest model satisfying the request’s latency and quality requirements.

   • Enforce tenant-level budgets.

3. Retries

   • Retry transient failures such as:

     • HTTP 429

     • 500/502/503

     • Connection timeout

   • Use exponential backoff with jitter.

   • Avoid retrying permanent errors such as invalid API keys or malformed requests.

4. Health Monitoring

   • Continuously monitor:

     • Error rate

     • Latency

     • Token throughput

     • Rate-limit responses

     • Provider availability

   • Temporarily remove unhealthy models/providers from routing.

2. Non-Functional Requirements

1. High Availability

   • Gateway should have no single point of failure.

   • Target availability: 99.99%+.

2. Low Latency

   • Gateway overhead should ideally be <10–20 ms excluding LLM latency.

   • Support streaming responses.

3. Scalability

   • Horizontally scale gateway instances.

   • Support millions of requests/day and thousands of concurrent streaming requests.

4. Fault Tolerance

   • Provider failures should not bring down the gateway.

   • Support retries, circuit breakers, fallback and graceful degradation.

5. Security

   • Encrypt provider API keys.

   • Authentication and authorization for clients.

   • Tenant isolation.

   • Don’t expose provider credentials to clients.

6. Observability

   • Distributed tracing.

   • Metrics and structured logs.

   • Track latency, cost, tokens, errors and fallback frequency.

7. Cost Efficiency

   • Avoid unnecessary calls to expensive models.

   • Support configurable cost budgets and routing policies.

3. High-Level Architecture

                         ┌──────────────────┐
                         │     Clients      │
                         │ Apps / Services   │
                         └────────┬─────────┘
                                  │
                                  ▼
                         ┌──────────────────┐
                         │   API Gateway    │
                         │ Auth + Validation│
                         └────────┬─────────┘
                                  │
                                  ▼
                    ┌──────────────────────────┐
                    │       AI Gateway         │
                    │                          │
                    │ Request Manager           │
                    │ Model Router              │
                    │ Rate Limiter              │
                    │ Cost Manager              │
                    │ Retry / Circuit Breaker   │
                    └────────────┬─────────────┘
                                 │
                    ┌────────────┴────────────┐
                    │                         │
                    ▼                         ▼
             ┌──────────────┐          ┌──────────────┐
             │ Model Router │          │ Health Engine│
             └──────┬───────┘          └──────────────┘
                    │
       ┌────────────┼─────────────┬─────────────┐
       ▼            ▼             ▼             ▼
 ┌──────────┐ ┌───────────┐ ┌──────────┐ ┌─────────────┐
 │ OpenAI   │ │ Anthropic │ │ Gemini   │ │ Open-source │
 │ Adapter  │ │ Adapter   │ │ Adapter  │ │ Adapter     │
 └────┬─────┘ └─────┬─────┘ └────┬─────┘ └──────┬──────┘
      │              │            │               │
      ▼              ▼            ▼               ▼
   OpenAI        Anthropic      Gemini       vLLM/TGI

4. Microservices Architecture

1. API Gateway

Responsibilities:

• Authentication

• API-key validation

• Request validation

• Tenant identification

• Request size limits

• TLS termination

Client
  ↓
API Gateway
  ↓
AI Gateway

2. Request/Inference Service

Responsible for:

• Normalizing requests

• Creating request IDs

• Managing streaming/non-streaming requests

• Sending request to Model Router

• Returning normalized response

3. Model Router

Core component.

It maintains a list of candidate models:

Request
   ↓
Model Router
   ↓
Candidate Models

GPT-X
Claude-X
Gemini-X
Llama-X

Routing score can be:

Score =
    capability_score
  - latency_penalty
  - cost_penalty
  - failure_penalty

Example:

GPT      cost = $$
Claude   cost = $
Gemini   cost = $
Llama    cost = very low

For a simple classification request:

Llama → selected

For complex reasoning:

Claude/GPT → selected

4. Provider Adapter Service

Each provider has a standardized adapter.

                 AI Gateway
                     |
        ┌────────────┼────────────┐
        ▼            ▼            ▼
    OpenAI        Anthropic     Gemini
    Adapter       Adapter       Adapter

Adapters convert the internal format into provider-specific APIs.

Internal:

{
  "model": "best",
  "messages": [],
  "temperature": 0.2
}

OpenAI adapter converts it into OpenAI’s format.

Anthropic adapter converts it into Anthropic’s format.

This prevents provider-specific logic from leaking into the core gateway.

5. Rate Limiter

Use Redis for distributed rate limiting.

Example:

Tenant A
   │
   ├── 100 RPM
   ├── 1M TPM
   └── $100/day

Tenant B
   │
   ├── 1000 RPM
   ├── 10M TPM
   └── $1000/day

Use Token Bucket or Leaky Bucket.

For LLMs, we need both:

RPM = Requests Per Minute
TPM = Tokens Per Minute

Example:

Request contains 10,000 tokens

Check:
RPM quota
+
TPM quota

If either exceeded → 429

6. Cost-Aware Scheduling

Maintain model pricing:

Model       Input       Output
--------------------------------
Model A     $5/M        $15/M
Model B     $1/M        $5/M
Model C     $0.2/M      $0.8/M

For every request:

estimated_cost =
input_tokens × input_price
+
estimated_output_tokens × output_price

Router can use:

Quality requirement
        +
Latency requirement
        +
Cost budget
        ↓
Candidate models
        ↓
Select cheapest suitable model

Example:

User asks:
"Translate this sentence"

Possible models:

GPT      $0.01
Claude   $0.008
Gemini   $0.004
Llama    $0.001

→ Llama selected

For complex reasoning:

Quality requirement = HIGH

Llama ❌
Gemini ⚠️
Claude ✅

→ Claude selected

7. Fallback Strategy

Use priority-based fallback.

Primary
   ↓
OpenAI GPT
   ↓ failure
Anthropic Claude
   ↓ failure
Gemini
   ↓ failure
Self-hosted Llama

But don’t fallback for every error.

Retryable

429
500
502
503
Timeout
Connection reset

Non-retryable

400
Invalid request
Invalid API key
Context too large
Unsupported model

8. Circuit Breaker

Suppose OpenAI starts returning:

70% requests → 503

Health service marks it unhealthy.

             OpenAI
               │
          Circuit OPEN
               │
               X
               
Requests → Anthropic

After a cooldown:

OPEN
 ↓
HALF-OPEN
 ↓
Send few requests
 ↓
Healthy → CLOSED

This prevents sending thousands of failing requests to a broken provider.

9. Retry Strategy

Use exponential backoff + jitter.

Attempt 1 → immediate
Attempt 2 → 100ms
Attempt 3 → 300ms
Attempt 4 → 700ms

With jitter:

delay = base × 2^attempt + random()

Also enforce:

max_retries = 2 or 3

Important: don’t retry blindly, especially for streaming requests or requests that may have already been processed.

Use an idempotency key where supported.

10. Health Monitoring

Create a Health Service.

Monitor every provider/model:

Provider
 ├── Availability
 ├── P50 latency
 ├── P95 latency
 ├── P99 latency
 ├── Error rate
 ├── 429 rate
 ├── Token throughput
 └── Cost

Example:

OpenAI GPT
Error Rate:    2%
P95:           1.2 sec
429 Rate:      0.5%
Status:        HEALTHY

Claude
Error Rate:    35%
P95:           5 sec
Status:        UNHEALTHY

Router then automatically reduces or stops traffic to Claude.

11. API Design

Generate Completion

POST /v1/chat/completions

Request:

{
  "model": "auto",
  "messages": [
    {
      "role": "user",
      "content": "Explain distributed systems"
    }
  ],
  "temperature": 0.2,
  "max_tokens": 1000,
  "stream": true
}

Response:

{
  "request_id": "req_123",
  "model": "claude-x",
  "provider": "anthropic",
  "usage": {
    "input_tokens": 100,
    "output_tokens": 500
  },
  "cost": 0.004
}

For streaming:

Client
  ↓
AI Gateway
  ↓
Provider

token → token → token → token
  ↑
stream back to client

Model Information

GET /v1/models

Response:

{
  "models": [
    {
      "id": "model-a",
      "provider": "openai",
      "context_window": 128000,
      "input_cost": 5,
      "output_cost": 15,
      "healthy": true
    }
  ]
}

Usage

GET /v1/usage

Response:

{
  "tenant_id": "tenant_123",
  "requests": 120000,
  "input_tokens": 50000000,
  "output_tokens": 10000000,
  "cost": 245.50
}

Routing Configuration

PUT /v1/routing/policy

Request:

{
  "strategy": "cost_optimized",
  "fallback": [
    "anthropic",
    "gemini",
    "self-hosted"
  ],
  "max_cost_per_request": 0.05
}

12. Database Choice

Use different databases for different workloads.

| Data                    | Database                 | Reason                          |
| ----------------------- | ------------------------ | ------------------------------- |
| Tenant/configuration    | PostgreSQL               | Strong consistency + relational |
| Model/provider metadata | PostgreSQL               | Configuration data              |
| Rate limiting           | Redis                    | Very low latency                |
| Cost/usage events       | Kafka + ClickHouse       | High-volume analytics           |
| Health metrics          | Prometheus               | Time-series metrics             |
| Logs                    | Elasticsearch/OpenSearch | Log search                      |
| Secrets                 | Vault/KMS                | Secure credential storage       |

13. Important DB Schemas

tenants

tenant_id PK
name
status
monthly_budget
created_at

models

model_id PK
provider
model_name
context_window
input_price
output_price
max_rpm
max_tpm
status

routing_policies

policy_id PK
tenant_id
strategy
primary_model
fallback_models
max_latency
max_cost

usage

request_id PK
tenant_id
model_id
provider
input_tokens
output_tokens
latency
cost
status
created_at

14. Kafka Usage

Kafka can be used for asynchronous events:

LLM Request
     │
     ▼
AI Gateway
     │
     ├──────────────→ Provider
     │
     └──────────────→ Kafka
                         │
             ┌───────────┼───────────┐
             ▼           ▼           ▼
         Cost Service  Analytics  Monitoring

Events:

LLM_REQUEST_COMPLETED
LLM_REQUEST_FAILED
MODEL_FALLBACK
PROVIDER_RATE_LIMITED
MODEL_HEALTH_CHANGED

This keeps the request path fast.

15. Redis Usage

Redis is used for:

1. Rate limiting
2. Provider health state
3. Circuit breaker state
4. Model routing cache
5. Tenant quotas
6. Short-lived request metadata

Example:

rate_limit:{tenant_id}
quota:{tenant_id}
health:{provider}:{model}
circuit:{provider}

16. Complete Request Flow

Client
  │
  ▼
API Gateway
  │
  ├── Authentication
  ├── Authorization
  └── Request validation
  │
  ▼
Rate Limiter
  │
  ├── RPM check
  └── TPM check
  │
  ▼
Request Service
  │
  ▼
Model Router
  │
  ├── Cost
  ├── Quality
  ├── Latency
  ├── Health
  └── Tenant policy
  │
  ▼
Provider Adapter
  │
  ▼
OpenAI / Anthropic / Gemini / vLLM
  │
  ▼
Response
  │
  ├── Token usage
  ├── Cost calculation
  ├── Latency
  └── Provider status
  │
  ├────────→ Kafka → Analytics
  │
  └────────→ Client

17. Key Interview Design Decisions

Why adapters?

To isolate provider-specific APIs and allow adding a new provider without modifying routing logic.

Why Redis for rate limiting?

Rate limiting is on the hot path and requires low-latency atomic operations across multiple gateway instances.

Why Kafka for usage?

Usage events can be extremely high volume. Kafka decouples request processing from analytics, billing and monitoring.

Why PostgreSQL?

Tenant, model and routing configurations require transactions and strong consistency.

Why Prometheus?

Health and latency metrics are naturally time-series data and need efficient aggregation.

How do we avoid one provider outage affecting the entire system?

Health checks
+
Circuit breaker
+
Retries
+
Fallback
+
Multi-provider routing

How do we minimize cost?

Cost-aware router
+
Model capability classification
+
Tenant budgets
+
Token estimation
+
Usage tracking

The core idea is to make the gateway provider-agnostic: clients talk to one API, while the gateway dynamically decides which model, which provider, and whether to retry/fallback, based on cost + quality + latency + health + tenant policy.

The mentorship is tailored for interview preparation, backend engineering, distributed systems, AI infrastructure, and building a strong systems-thinking foundation.

📩 Contact: programmingappliedai@gmail.com

1. Requirements

Functional

• Store and search 10B+ embeddings with configurable dimensions such as 768/1536.

• Support real-time insert, update and delete of vectors.

• Support approximate nearest-neighbor search using HNSW / IVF-PQ.

• Support metadata filtering such as tenant_id, category, timestamp, etc.

• Support top-K similarity search with optional filters.

• Provide replication, backup and disaster recovery.

Non-functional

• Search latency: <100 ms p95 for normal workloads.

• High availability: 99.99%+.

• Horizontally scalable by adding nodes.

• No data loss for acknowledged writes; target RPO ≈ 0 with synchronous/replicated WAL depending on deployment.

• Support tenant isolation and predictable performance under noisy-neighbor workloads.

2. High-Level Architecture

                        ┌──────────────────┐
                        │   Client / Apps  │
                        └────────┬─────────┘
                                 │
                                 ▼
                       ┌───────────────────┐
                       │   API Gateway     │
                       │ Auth / Rate Limit │
                       └─────────┬─────────┘
                                 │
                 ┌───────────────┴────────────────┐
                 ▼                                ▼
        ┌─────────────────┐              ┌─────────────────┐
        │ Query Coordinator│              │  Write Service  │
        └────────┬────────┘              └────────┬────────┘
                 │                                │
                 ▼                                ▼
        ┌─────────────────┐              ┌─────────────────┐
        │ Query Planner   │              │ WAL / Log       │
        │ + Filter Engine │              │ Kafka           │
        └────────┬────────┘              └────────┬────────┘
                 │                                │
                 ▼                                ▼
        ┌──────────────────────────────────────────────────┐
        │              Shard Router / Metadata             │
        └───────────────┬──────────────────────────────────┘
                        │
             ┌──────────┼──────────┐
             ▼          ▼          ▼
          Shard 1    Shard 2     Shard N
          ┌─────┐    ┌─────┐     ┌─────┐
          │Index│    │Index│     │Index│
          │HNSW │    │IVF-PQ│    │HNSW │
          ├─────┤    ├─────┤     ├─────┤
          │Vector│   │Vector│    │Vector│
          │Store │   │Store │    │Store │
          ├─────┤    ├─────┤     ├─────┤
          │Meta  │    │Meta  │    │Meta  │
          └─────┘    └─────┘     └─────┘
             │          │           │
             └──────────┼───────────┘
                        ▼
                Object Storage
               Backups / Snapshots

3. Data Model

A vector record can look like:

Vector {
    vector_id
    tenant_id
    embedding
    dimension
    metadata
    created_at
    updated_at
    version
    deleted
}

Example metadata:

{
  "tenant_id": "tenant_123",
  "category": "electronics",
  "language": "en",
  "timestamp": 1755000000
}

Separate vector and metadata storage is useful because metadata filtering and vector similarity have very different access patterns.

4. Sharding

At 10B vectors, a single machine is impossible.

Assume:

10B vectors
1536 dimensions
FP32 = 4 bytes

Raw vector storage:

10B × 1536 × 4
≈ 61.4 TB

And this excludes:

• HNSW index

• metadata

• replication

• WAL

• operating-system overhead

• temporary indexing memory

With 3× replication, raw vector data alone becomes roughly:

61.4 TB × 3 ≈ 184 TB

Therefore we horizontally shard the dataset.

Sharding strategy

Use consistent hashing / hash-based partitioning:

hash(tenant_id + vector_id) → shard

For very large tenants, use:

hash(vector_id) → shard

rather than putting an entire tenant on one shard.

A logical partition could be:

Partition P1 → Shard 17
Partition P2 → Shard 42
Partition P3 → Shard 83
...

Use virtual shards so that physical nodes can be added or removed without massive data movement.

5. Query Flow

For:

Top K similar vectors
where category = "phone"

the flow is:

Client
  ↓
API Gateway
  ↓
Query Coordinator
  ↓
Metadata / Filter Planner
  ↓
Determine relevant shards
  ↓
Parallel ANN search
  ↓
Shard 1 → top K
Shard 2 → top K
Shard 3 → top K
...
  ↓
Merge candidates
  ↓
Optional exact distance calculation
  ↓
Global Top K
  ↓
Client

If searching 100 shards and requesting K=10, each shard may return:

local_top_k = 50

Then the coordinator merges:

100 × 50 = 5000 candidates

and performs exact distance computation on those candidates.

This gives better recall than directly returning each shard’s top 10.

6. Indexing — HNSW

HNSW creates a graph of vectors.

        A -------- B
       /            \
      C ---- D ------ E
       \             /
        F -----------G

Each vector is connected to nearby vectors.

Search starts from an entry point and traverses the graph toward increasingly similar vectors.

Advantages

• Excellent recall.

• Very low query latency.

• Good for relatively dynamic workloads.

Disadvantages

• High memory consumption.

• Expensive updates.

• Large graph at 10B vectors.

Therefore, we generally don’t maintain one gigantic HNSW graph.

Instead:

Shard
 ├── Segment 1 → HNSW
 ├── Segment 2 → HNSW
 ├── Segment 3 → HNSW
 └── ...

7. Indexing — IVF-PQ

For extremely large datasets, IVF-PQ is attractive because it substantially reduces memory/storage requirements.

IVF

First cluster vectors:

             Vector Space

        C1        C2

              C3

    C4                 C5

During search:

Query
  ↓
Find nearest clusters
  ↓
Search only those clusters

Instead of searching 10B vectors, we may search only vectors belonging to the closest few clusters.

PQ

Product Quantization compresses vectors into compact codes.

For example:

1536-dimensional vector
        ↓
Split into sub-vectors
        ↓
Quantize each sub-vector
        ↓
Compact code

This significantly reduces memory requirements.

HNSW vs IVF-PQ

| Feature   | HNSW             | IVF-PQ           |
| --------- | ---------------- | ---------------- |
| Recall    | Very high        | High, tunable    |
| Latency   | Very low         | Low              |
| Memory    | High             | Much lower       |
| Updates   | Easier           | More complex     |
| 10B scale | Expensive        | More suitable    |
| Best use  | Hot/dynamic data | Massive datasets |

A practical architecture can use HNSW for recent/hot segments and IVF-PQ for large historical segments.

8. Real-Time Updates

This is one of the hardest parts.

Building a huge ANN index after every write is impossible.

Use a mutable + immutable segment architecture.

                    Incoming writes
                          │
                          ▼
                     WAL / Kafka
                          │
                          ▼
                  Mutable Memtable
                          │
                 ┌────────┴────────┐
                 ▼                 ▼
             HNSW index       Metadata index
                 │
                 ▼
           Periodic flush
                 │
                 ▼
          Immutable Segment
                 │
                 ▼
         Background compaction

Write path

Client
  ↓
Write API
  ↓
WAL
  ↓
Replicas
  ↓
Memtable
  ↓
Realtime HNSW

Once a segment reaches a threshold:

Memtable → Immutable segment

Background workers then build/optimize IVF-PQ indexes.

This gives low write latency without continuously rebuilding the entire index.

9. Updates and Deletes

Never physically modify a huge immutable segment immediately.

For an update:

vector_id = 123
version = 7

write:

version = 8

and mark version 7 obsolete.

For delete:

tombstone(vector_id)

During query:

candidate
   ↓
metadata/version check
   ↓
if deleted/old → discard

Later, compaction removes obsolete vectors and tombstones.

This is similar to an LSM-tree style architecture.

10. Metadata Storage

Do not store large metadata blobs inside the ANN index.

Use a separate metadata store:

Vector ID
   │
   ├── ANN index
   │
   └── Metadata Store
          ├── tenant_id
          ├── category
          ├── timestamp
          ├── language
          └── attributes

Depending on requirements:

• RocksDB → local metadata/index state.

• PostgreSQL → transactional metadata with moderate scale.

• Cassandra/ScyllaDB → massive distributed metadata.

• Object storage → immutable historical metadata/segments.

For 10B-scale infrastructure, I would favor a distributed KV store for metadata that needs horizontal scaling.

11. Filtering

Filtering is tricky because:

similarity search
+
metadata filtering

must be performed efficiently.

Example:

Find 20 nearest vectors
WHERE
    tenant_id = 123
    AND category = "car"
    AND language = "en"

Approach 1 — Pre-filter

First identify matching IDs:

Metadata Index
      ↓
1M matching vectors
      ↓
ANN search

Good when filters are highly selective.

Approach 2 — Post-filter

ANN search
   ↓
1000 candidates
   ↓
metadata filter
   ↓
20 results

Good when filter selectivity is low.

Better approach

Use an adaptive planner:

Estimate filter selectivity
        ↓
Highly selective?
   /           \
 Yes            No
 ↓               ↓
Pre-filter     ANN first

For frequently used fields, maintain bitmap/inverted indexes.

12. Replication

Each shard has multiple replicas:

Shard 17

Replica A → Node 1
Replica B → Node 8
Replica C → Node 21

Use 3 replicas across different failure domains.

        Shard Leader
        /     |     \
       /      |      \
 Replica 1  Replica 2  Replica 3
 Zone A      Zone B      Zone C

Writes

Client
  ↓
Leader
  ↓
WAL
  ↓
Replicas
  ↓
Quorum ACK

For stronger durability:

W + R > N

For example:

N = 3
W = 2
R = 2

A leader failure triggers leader election.

13. Read Scaling

ANN queries are generally read-heavy.

Therefore:

               Query Coordinator
                 /    |    \
                /     |     \
             Replica Replica Replica

Queries can be distributed across replicas.

Use:

• replica-aware routing

• load-based routing

• cache-aware routing

• tenant-aware routing

Avoid sending every query to the leader.

14. Hot/Cold Architecture

10B embeddings may not have uniform access patterns.

Use:

HOT
 ↓
RAM + HNSW
 ↓
Warm
 ↓
SSD + IVF
 ↓
COLD
 ↓
Object Storage + IVF-PQ

For example:

Recent 100M vectors → HNSW
Older 9.9B vectors → IVF-PQ

Search both and merge results.

This significantly reduces RAM requirements.

15. Disaster Recovery

Use multiple layers.

WAL

Every mutation goes into durable WAL:

Write
 ↓
WAL
 ↓
Vector Store

Kafka/object storage can retain the mutation stream.

Snapshots

Periodically create:

Shard snapshot
+
ANN index
+
Metadata
+
WAL checkpoint

Store snapshots in durable object storage.

Example:

Object Storage

bucket/
 ├── shard-001/
 │    ├── snapshot-100
 │    ├── snapshot-200
 │    └── WAL-offset
 ├── shard-002/
 └── ...

Recovery

Latest snapshot
      ↓
Restore
      ↓
Replay WAL
      ↓
Rebuild missing indexes
      ↓
Replica synchronization
      ↓
Shard becomes healthy

For regional disaster recovery:

Region A
  ↓ async replication
Region B

For critical workloads, keep a warm standby cluster.

16. Failure Handling

Node failure

Node dies
   ↓
Shard coordinator detects failure
   ↓
Route traffic to replica
   ↓
Create replacement replica
   ↓
Stream missing data

Replica failure

Rebuild from:

healthy replica
+
WAL

Corrupted index

Don’t treat the ANN index as the source of truth.

The source of truth should be:

WAL + durable vector segments

The ANN index is a derived structure.

Therefore:

Corrupted HNSW
      ↓
Delete index
      ↓
Rebuild from vector segments

17. Microservices

API Gateway
   │
   ├── Auth / Rate Limiting
   │
   ▼
Query Service
   │
   ├── Query Planner
   ├── Filter Engine
   └── Shard Router
          │
          ▼
      Vector Nodes
       ├── ANN Engine
       ├── Vector Store
       ├── Metadata Store
       └── WAL Consumer

Write Service
   │
   ├── Validation
   ├── WAL Producer
   └── Partition Router

Index Service
   ├── HNSW Builder
   ├── IVF Builder
   ├── PQ Compressor
   └── Compaction

Metadata Service
   └── Distributed KV Store

Snapshot Service
   ├── Checkpoint
   └── Object Storage

Cluster Manager
   ├── Membership
   ├── Leader Election
   ├── Rebalancing
   └── Health Checks

18. Important APIs

Insert

POST /v1/vectors

{
  "id": "vec_123",
  "tenant_id": "tenant_1",
  "embedding": [0.12, 0.43, "..."],
  "metadata": {
    "category": "car",
    "language": "en"
  }
}

Response:

{
  "id": "vec_123",
  "version": 7,
  "status": "accepted"
}

Search

POST /v1/search

{
  "tenant_id": "tenant_1",
  "query_vector": [0.12, 0.43, "..."],
  "top_k": 20,
  "filter": {
    "category": "car",
    "language": "en"
  }
}

Response:

{
  "results": [
    {
      "id": "vec_892",
      "score": 0.94,
      "metadata": {
        "category": "car"
      }
    }
  ]
}

Delete

DELETE /v1/vectors/{id}

Response:

{
  "id": "vec_123",
  "status": "deleted"
}

19. Database / Storage Choices

Vector data
     → SSD/Object Storage

ANN index
     → Local NVMe SSD + RAM

Metadata
     → Cassandra/ScyllaDB or distributed KV

WAL
     → Kafka

Snapshots
     → S3/GCS/Object Storage

Cluster metadata
     → etcd

The important architectural principle is:

Don’t make the ANN index your source of truth.

Treat vectors + WAL + durable segments as the source of truth and make HNSW/IVF-PQ rebuildable derived indexes.

20. Final Architecture

                         Clients
                            │
                            ▼
                     ┌─────────────┐
                     │ API Gateway │
                     └──────┬──────┘
                            │
                 ┌──────────┴──────────┐
                 ▼                     ▼
          ┌─────────────┐       ┌─────────────┐
          │ Query Svc   │       │ Write Svc   │
          └──────┬──────┘       └──────┬──────┘
                 │                     │
                 ▼                     ▼
          ┌─────────────┐          ┌────────┐
          │Query Planner│          │  WAL   │
          └──────┬──────┘          │ Kafka  │
                 │                 └───┬────┘
                 │                     │
                 └──────────┬──────────┘
                            ▼
                    ┌───────────────┐
                    │ Shard Router  │
                    └───────┬───────┘
                            │
          ┌─────────────────┼─────────────────┐
          ▼                 ▼                 ▼
       Shard 1           Shard 2           Shard N
     ┌─────────┐       ┌─────────┐       ┌─────────┐
     │ HNSW    │       │ IVF-PQ  │       │ HNSW    │
     │ Index   │       │ Index   │       │ Index   │
     ├─────────┤       ├─────────┤       ├─────────┤
     │Vectors  │       │Vectors  │       │Vectors  │
     ├─────────┤       ├─────────┤       ├─────────┤
     │Metadata │       │Metadata │       │Metadata │
     └────┬────┘       └────┬────┘       └────┬────┘
          │                 │                 │
          └─────────────────┼─────────────────┘
                            ▼
                     Object Storage
                    Snapshots / Backup

Key design decisions

1. Virtual sharding enables horizontal scaling to 10B+ vectors.

2. HNSW is used for hot/recent data where low latency and high recall matter.

3. IVF-PQ is used for massive cold datasets to reduce memory/storage.

4. Mutable HNSW + immutable segments enables real-time updates without rebuilding huge indexes.

5. Metadata indexes + adaptive filtering handle vector + structured queries efficiently.

6. 3-way replication across AZs provides high availability.

7. WAL + snapshots + replay provides disaster recovery.

8. ANN indexes are derived data, allowing them to be rebuilt after corruption or failure.

9. Query fan-out + local Top-K + global merge enables distributed ANN search.

10. Hot/warm/cold tiers keep the frequently accessed vectors in RAM while cheaply storing billions of older vectors.

If you’re looking for paid 1:1 mentorship with a strong focus on Low-Level Design (LLD)—especially Multithreading and Concurrency—along with High-Level Design (HLD), Data Structures & Algorithms (DSA), System Design, LLM/Generative AI systems, and research paper discussions, feel free to reach out.

The mentorship is tailored for interview preparation, backend engineering, distributed systems, AI infrastructure, and building a strong systems-thinking foundation.

📩 Contact: programmingappliedai@gmail.com

1. Functional Requirements — 5 points

1. Task Creation

   • User should be able to submit a high-level task.

   • Example: “Build a Python service that analyzes customer transactions.”

2. Task Planning

   • Planner Agent decomposes the task into smaller executable subtasks.

   • Creates dependencies between subtasks.

3. Multi-Agent Collaboration

   • Route subtasks to appropriate agents such as Researcher, Coder, Reviewer, etc.

   • Agents should be able to consume outputs generated by other agents.

4. Workflow Execution

   • Execute independent tasks in parallel and dependent tasks sequentially.

   • Support retries, timeouts, cancellation, and failure handling.

5. Monitoring & Results

   • Track task/agent status, execution history, token usage, latency, failures, and final output.

   • User should be able to query workflow progress and retrieve results.

2. Non-Functional Requirements — 5 points

1. Scalability

   • Support thousands/millions of workflows with dynamically scalable agent workers.

2. Availability

   • Target 99.9%+ availability.

   • Agent/workflow failures should not cause loss of workflow state.

3. Low Latency

   • Task scheduling/routing should ideally add <50–100 ms overhead.

   • Independent agent tasks should execute concurrently.

4. Reliability

   • At-least-once task execution with idempotent workers.

   • Retry transient failures using exponential backoff.

5. Observability & Security

   • Distributed tracing, metrics, logs, token/cost tracking.

   • Tenant-level isolation and authorization for shared memory.

3. High-Level Architecture

                         ┌──────────────────┐
                         │      Client      │
                         │ Web / API / SDK  │
                         └────────┬─────────┘
                                  │
                                  ▼
                         ┌──────────────────┐
                         │    API Gateway   │
                         └────────┬─────────┘
                                  │
                                  ▼
                    ┌─────────────────────────┐
                    │   Workflow Service      │
                    │                         │
                    │ Create / Pause / Resume │
                    └───────────┬─────────────┘
                                │
                                ▼
                    ┌─────────────────────────┐
                    │   Planner Agent         │
                    │                         │
                    │ Task → DAG of subtasks │
                    └───────────┬─────────────┘
                                │
                                ▼
                  ┌──────────────────────────────┐
                  │      Task Scheduler          │
                  │                              │
                  │ DAG + dependency management  │
                  └────────────┬─────────────────┘
                               │
                  ┌────────────┼─────────────┐
                  ▼            ▼             ▼
             ┌────────┐   ┌────────┐   ┌────────┐
             │Research│   │ Coder  │   │Reviewer│
             │ Agent  │   │ Agent  │   │ Agent  │
             └───┬────┘   └───┬────┘   └───┬────┘
                 │            │             │
                 └────────────┼─────────────┘
                              ▼
                    ┌────────────────────┐
                    │   Shared Memory    │
                    │                    │
                    │ Context / Artifacts│
                    │ Agent Outputs      │
                    └────────────────────┘

                              │
                              ▼
                    ┌────────────────────┐
                    │   Event Bus        │
                    │ Kafka / SQS        │
                    └────────────────────┘

                              │
                              ▼
                    ┌────────────────────┐
                    │ Observability      │
                    │ Metrics / Logs /   │
                    │ Traces / Cost      │
                    └────────────────────┘

4. Core Workflow

Suppose the user asks:

“Build a REST API for analyzing transactions.”

Planner creates:

                Root Task
                   │
          ┌────────┴────────┐
          ▼                 ▼
   Research APIs       Research DB
          │                 │
          └────────┬────────┘
                   ▼
              Write Code
                   │
                   ▼
             Run Tests
                   │
                   ▼
                Review
                   │
             ┌─────┴─────┐
             │           │
           PASS         FAIL
             │           │
             ▼           ▼
           Done       Fix Code
                         │
                         └──────→ Review

The workflow is represented as a DAG.

Each node contains:

Task {
    task_id
    workflow_id
    agent_type
    input
    dependencies[]
    status
    retry_count
    timeout
}

5. Task Scheduling

The Task Scheduler maintains the workflow DAG.

Task lifecycle

PENDING
   ↓
READY
   ↓
RUNNING
   ↓
SUCCESS

Failure:

RUNNING
   ↓
FAILED
   ↓
RETRY
   ↓
READY

After maximum retries:

FAILED → DEAD_LETTER

Scheduling algorithm

For every completed task:

1. Mark task SUCCESS
2. Find dependent tasks
3. Check whether all dependencies are SUCCESS
4. If yes → mark dependent task READY
5. Push task to appropriate agent queue

Independent tasks can therefore execute concurrently.

6. Task Routing

Use an Agent Registry.

Agent Registry

researcher → researcher_queue
coder      → coder_queue
reviewer   → reviewer_queue
planner    → planner_queue

Scheduler determines:

task.agent_type = CODER

and publishes:

Kafka:
    topic = agent.coder.tasks

Coder workers consume from this topic.

For dynamic routing, maintain agent metadata:

Agent {
    agent_id
    type
    capabilities[]
    current_load
    max_concurrency
    health
}

Example:

"Search latest Kafka documentation"
            ↓
Researcher
            ↓
Researcher-3

Routing can consider:

• capability

• current load

• priority

• model availability

• cost

• region

7. Shared Memory

Agents shouldn’t directly depend on each other’s local memory.

Use a centralized Shared Memory Service.

                     Shared Memory
                          │
        ┌─────────────────┼─────────────────┐
        │                 │                 │
    Task Context       Artifacts        Agent Outputs
        │                 │                 │
    Redis/Postgres    Object Storage    Vector DB

Types of memory

Short-term memory

Current workflow context.

Redis

Durable state

Workflow/task state.

PostgreSQL

Large artifacts

Code, documents, datasets.

S3 / Object Storage

Semantic memory

Research findings, previous knowledge.

Vector DB

Agents access memory using:

GET /workflows/{workflow_id}/context

and:

POST /workflows/{workflow_id}/memory

8. Retry Strategy

Not every failure should be retried.

Retryable

LLM timeout
Network failure
Rate limit
Temporary DB failure
Temporary service unavailable

Non-retryable

Invalid prompt
Invalid tool arguments
Permission denied
Invalid workflow

Use exponential backoff:

retry 1 → 1 sec
retry 2 → 2 sec
retry 3 → 4 sec
retry 4 → 8 sec

with jitter.

Each task has:

max_retries = 3

After that:

Dead Letter Queue

9. Deadlock Prevention

Since workflows are DAG-based, the system should reject cyclic dependencies.

Example invalid workflow:

A → B
B → C
C → A

When creating the DAG, perform cycle detection using:

DFS / Topological Sort

If:

number_of_processed_nodes != total_nodes

then a cycle exists.

Return:

400 INVALID_WORKFLOW

Runtime deadlock prevention

Also use:

• Task timeout

• Workflow timeout

• Lease expiration

• Heartbeats

• Maximum dependency depth

• Maximum task execution time

• Cancellation propagation

Example:

Worker takes task
      ↓
Lease = 5 minutes
      ↓
Worker sends heartbeat
      ↓
Worker crashes
      ↓
Lease expires
      ↓
Scheduler requeues task

10. APIs

Create Workflow

POST /v1/workflows

Request

{
  "task": "Build a REST API for analyzing customer transactions",
  "priority": "HIGH",
  "max_retries": 3,
  "metadata": {
    "language": "python"
  }
}

Response

{
  "workflow_id": "wf_12345",
  "status": "PLANNING",
  "created_at": "2026-08-10T17:00:00Z"
}

Get Workflow Status

GET /v1/workflows/{workflow_id}

Response

{
  "workflow_id": "wf_12345",
  "status": "RUNNING",
  "progress": 65,
  "tasks": [
    {
      "task_id": "t1",
      "agent": "researcher",
      "status": "SUCCESS"
    },
    {
      "task_id": "t2",
      "agent": "coder",
      "status": "RUNNING"
    },
    {
      "task_id": "t3",
      "agent": "reviewer",
      "status": "PENDING"
    }
  ]
}

Get Workflow Result

GET /v1/workflows/{workflow_id}/result

Response

{
  "workflow_id": "wf_12345",
  "status": "COMPLETED",
  "result": {
    "code_artifact": "s3://bucket/code.zip",
    "review": "Approved",
    "summary": "REST API implemented successfully"
  }
}

Cancel Workflow

POST /v1/workflows/{workflow_id}/cancel

Response

{
  "workflow_id": "wf_12345",
  "status": "CANCELLED"
}

Get Shared Memory

GET /v1/workflows/{workflow_id}/memory

Response

{
  "workflow_id": "wf_12345",
  "memory": [
    {
      "task_id": "t1",
      "type": "RESEARCH_RESULT",
      "content": "PostgreSQL is selected as the primary database"
    },
    {
      "task_id": "t2",
      "type": "CODE_ARTIFACT",
      "uri": "s3://bucket/code.zip"
    }
  ]
}

11. Microservices Architecture

1. API Gateway

Responsibilities:

Authentication
Authorization
Rate limiting
Request validation
Routing

2. Workflow Service

Responsible for:

Create workflow
Pause/resume
Cancel workflow
Workflow state

3. Planner Service

Converts:

User Goal
   ↓
Task DAG

Example:

Research → Code → Test → Review

4. Scheduler Service

Responsible for:

DAG execution
Dependency resolution
Task prioritization
Retries
Timeouts
Task leases

5. Agent Router

Routes tasks based on:

Agent type
Capability
Load
Cost
Model
Availability

6. Agent Workers

Separate worker pools:

Researcher Workers
Coder Workers
Reviewer Workers
Planner Workers

Each worker can independently scale.

7. Memory Service

Provides:

Read context
Write context
Semantic search
Artifact references
Versioning

8. Artifact Service

Stores:

Source code
Documents
Research reports
Images
Large outputs

Object storage such as S3 is appropriate.

9. Event Bus

Kafka/SQS handles:

Task created
Task completed
Task failed
Retry task
Workflow completed
Agent events

10. Observability Service

Collect:

Metrics
Logs
Distributed traces
LLM token usage
LLM cost
Agent latency
Task retry count
Workflow success rate

12. Database Choice

| Data                | Database                 | Why                                    |
| ------------------- | ------------------------ | -------------------------------------- |
| Workflow/Task state | PostgreSQL               | Transactions + relational DAG metadata |
| Hot workflow state  | Redis                    | Low-latency reads/writes               |
| Agent queues/events | Kafka                    | Durable asynchronous processing        |
| Large artifacts     | S3/Object Storage        | Cheap, scalable blob storage           |
| Semantic memory     | Vector DB                | Similarity search                      |
| Logs                | ClickHouse/Elasticsearch | High-volume analytics/search           |

PostgreSQL Schema

workflows
---------
workflow_id PK
tenant_id
status
priority
input
created_at
updated_at


tasks
-----
task_id PK
workflow_id FK
agent_type
status
input
output
retry_count
max_retries
timeout
created_at
updated_at


task_dependencies
-----------------
task_id
depends_on_task_id
PRIMARY KEY(task_id, depends_on_task_id)


agent_executions
----------------
execution_id PK
task_id
agent_id
started_at
completed_at
status
error
token_usage
cost

13. Important Design Decision: DB + Kafka

Don’t use Kafka as the source of truth for workflow state.

Use:

PostgreSQL = Source of Truth
Kafka      = Event/Task Transport
Redis      = Cache

For example:

Task completed
     │
     ├── UPDATE PostgreSQL
     │
     └── Publish TaskCompleted event → Kafka

For reliable DB → Kafka publishing, use the Transactional Outbox Pattern:

                    PostgreSQL
                 ┌─────────────┐
                 │ task update │
                 │ outbox event│
                 └──────┬──────┘
                        │
                  Outbox Worker
                        │
                        ▼
                      Kafka

This prevents the situation where the DB update succeeds but the Kafka event is lost.

14. End-to-End Flow

User
 │
 ▼
API Gateway
 │
 ▼
Workflow Service
 │
 ▼
Planner Agent
 │
 │ Creates DAG
 ▼
Task Scheduler
 │
 ├──────────────┐
 ▼              ▼
Researcher     Researcher
 │              │
 └──────┬───────┘
        ▼
      Coder
        │
        ▼
      Tests
        │
        ▼
     Reviewer
        │
    ┌───┴───┐
    │       │
  PASS     FAIL
    │       │
    ▼       ▼
  Done    Coder
            │
            └────→ Reviewer

Every agent interaction is recorded in:

PostgreSQL → workflow/task state
Redis      → hot context
Kafka      → events/tasks
S3        → artifacts
Vector DB  → semantic memory

And monitored through:

OpenTelemetry
      ↓
Metrics + Logs + Traces
      ↓
Grafana / Alerting

Key interview point: The heart of this system is a persistent DAG-based workflow engine + asynchronous task queues + shared memory layer. The agents themselves remain stateless workers, which makes them independently scalable and allows the scheduler to handle retries, failures, timeouts, and recovery centrally.

Multiple applications (chatbots, support assistants, coding assistants, search copilots, etc.) send repeated or similar prompts to LLM providers. Since LLM inference is expensive and introduces latency, we need a Distributed Prompt Cache that:

• Reduces LLM API costs

• Reduces response latency

• Supports multiple applications and tenants

• Handles exact and semantic prompt matching

• Preserves privacy and security

• Scales horizontally

2. Functional Requirements

Core Features

• Cache LLM responses for repeated prompts

• Support exact-match lookup

• Support semantic similarity lookup

• Configurable TTL policies

• Cache invalidation support

• Multi-tenant isolation

• Cache warming/pre-population

• Distributed deployment

Non-Functional Requirements

• High availability (99.9%+)

• Low latency (<20 ms cache lookup)

• Horizontally scalable

• Secure tenant isolation

• Cost-efficient storage

3. High-Level Architecture

                +----------------+
                | Applications   |
                | (Multi-tenant) |
                +-------+--------+
                        |
                        v
           +-------------------------+
           | Prompt Cache Gateway    |
           +-----------+-------------+
                       |
         +-------------+--------------+
         |                            |
         v                            v

+----------------+        +---------------------+
| Exact Cache    |        | Semantic Cache      |
| (Redis)        |        | (Vector DB)         |
+--------+-------+        +----------+----------+
         |                           |
         +------------+--------------+
                      |
                      v
            +------------------+
            | Cache Decision   |
            | Service          |
            +--------+---------+
                     |
          Cache Miss |
                     v
             +---------------+
             | LLM Provider  |
             | (GPT/Claude)  |
             +-------+-------+
                     |
                     v
             +---------------+
             | Cache Writer  |
             +---------------+

4. Request Flow

Exact Match Hit

User Request
      |
      v
Generate Cache Key
      |
      v
Redis Lookup
      |
      +---- HIT ----> Return Response

Cache Miss

User Request
      |
      v
Exact Match Lookup
      |
      +---- MISS
               |
               v
       Semantic Search
               |
         Similarity > Threshold?
             /        \
           Yes        No
            |          |
            v          v
      Return Cached   Call LLM
       Response         |
                         v
                  Store in Cache

5. Cache Key Design

The cache key must uniquely identify prompts that should produce identical responses.

Key Structure

hash(
    tenant_id +
    model_name +
    model_version +
    system_prompt +
    user_prompt +
    temperature +
    top_p
)

Example

tenantA:gpt-4o:v2:
SHA256(normalized_prompt)

Why Include These Fields?

| Field         | Reason                                     |
| ------------- | ------------------------------------------ |
| Tenant ID     | Prevent data leakage                       |
| Model Name    | Different models produce different outputs |
| Model Version | Responses change after upgrades            |
| Temperature   | Affects output randomness                  |
| System Prompt | Changes behavior                           |
| User Prompt   | Main query                                 |

6. Prompt Normalization

Before generating cache keys:

Transformations

" What is Kubernetes? "
"What is Kubernetes?"

Normalize:

• Trim whitespace

• Lowercase (optional)

• Remove repeated spaces

• Canonicalize formatting

Example:

"What  is Kubernetes?"

↓

"what is kubernetes?"

This improves cache hit rates.

7. Semantic Similarity Matching

Exact matching misses logically equivalent prompts.

Examples

Prompt A:

What is Kubernetes?

Prompt B:

Explain Kubernetes.

Both should reuse the same response.

Semantic Cache Architecture

Step 1: Generate Embedding

Prompt
   |
Embedding Model
   |
Vector (1536 dimensions)

Step 2: Store in Vector DB

Possible technologies:

• Pinecone

• Weaviate

• Milvus

• pgvector

• OpenSearch Vector Search

Store:

{
  "tenant":"A",
  "embedding":[...],
  "response":"...",
  "ttl":"..."
}

Lookup

New Prompt
      |
Generate Embedding
      |
Vector Search
      |
Top K Results
      |
Similarity Score

Decision Rule

similarity >= 0.92

→ Cache Hit

Else

→ LLM Call

Similarity Metrics

• Cosine similarity

• Euclidean distance

• Dot product

Cosine similarity is most common.

8. Cache Storage Design

L1 Cache

Redis

Purpose:

• Ultra-fast exact lookup

• Millisecond latency

Stores:

key -> response

L2 Cache

Vector Database

Purpose:

• Semantic retrieval

Stores:

embedding -> response

9. TTL Strategy

Not all prompts age equally.

Dynamic TTL

| Prompt Type  | TTL     |
| ------------ | ------- |
| Math         | 30 days |
| Coding Help  | 7 days  |
| News         | 1 hour  |
| Weather      | 15 mins |
| Stock Prices | 5 mins  |

TTL Formula

TTL =
baseTTL *
accessFrequencyMultiplier *
confidenceMultiplier

Frequently reused prompts stay longer.

10. Cache Invalidation

One of the hardest problems.

Trigger 1: Model Upgrade

GPT-4o-v1
      ↓
GPT-4o-v2

Invalidate entries tied to old version.

Trigger 2: Prompt Template Change

System Prompt Changed

Invalidate matching namespace.

Trigger 3: Knowledge Refresh

Example:

Company policy updated

Invalidate related cache entries.

Trigger 4: Manual Purge

Admin API:

DELETE /cache/{tenant}
DELETE /cache/key/{id}

11. Privacy Isolation

Critical in enterprise environments.

Tenant Isolation

Every cache entry contains:

tenant_id

Lookup only within same tenant.

tenantA ≠ tenantB

No cross-tenant reuse.

Namespace Isolation

tenantA/app1
tenantA/app2
tenantB/app1

Separate cache namespaces.

Encryption

At Rest:

• AES-256

In Transit:

• TLS

Sensitive Data Protection

Before caching:

Prompt
   |
PII Detector
   |
Sensitive?

If sensitive:

Do Not Cache

Examples:

• Credit card numbers

• SSN

• Passwords

• Medical records

12. Cache Warming

Pre-populate cache before traffic arrives.

Method 1: Historical Analysis

Use top prompts from logs.

Top 10,000 prompts
      |
Generate Responses
      |
Populate Cache

Method 2: Product Launch Warmup

Example:

New HR Bot

Preload:

• Leave policy

• Benefits

• Payroll FAQ

Method 3: Scheduled Refresh

Nightly job:

Refresh frequently used entries

before expiry.

13. Scalability Design

Redis Cluster

Redis Shard 1
Redis Shard 2
Redis Shard 3

Partition by:

hash(cache_key)

Vector DB Cluster

Embedding Space
      |
Partition/Sharding
      |
Multiple Nodes

Supports billions of vectors.

14. APIs

Lookup

POST /cache/lookup

Request:

{
  "tenant":"A",
  "prompt":"What is Kubernetes?"
}

Response:

{
  "hit": true,
  "response":"..."
}

Insert

POST /cache/store

Invalidate

DELETE /cache/key/{id}

15. Metrics & Monitoring

Track:

Cache Metrics

Exact Hit Rate
Semantic Hit Rate
Miss Rate

Cost Metrics

Tokens Saved
LLM Calls Avoided
Dollar Savings

Performance Metrics

P95 Latency
P99 Latency
Redis Latency
Vector Search Latency

16. Design Decisions Summary

| Component         | Choice                                 |
| ----------------- | -------------------------------------- |
| L1 Cache          | Redis Cluster                          |
| L2 Semantic Cache | Vector DB (Milvus/Pinecone/pgvector)   |
| Cache Key         | Hash(tenant + model + prompt config)   |
| Similarity Metric | Cosine Similarity                      |
| Threshold         | 0.90–0.95                              |
| TTL               | Dynamic by content type                |
| Invalidation      | Versioning + Event-driven purge        |
| Privacy           | Tenant namespaces + PII filtering      |
| Warming           | Historical prompts + scheduled refresh |

Final Design

A two-level distributed prompt cache consists of:

1. L1 Redis exact-match cache for sub-millisecond retrieval.

2. L2 Vector semantic cache for similar prompts.

3. Tenant-aware cache keys for privacy.

4. Dynamic TTL and invalidation policies to maintain freshness.

5. Cache warming pipelines to maximize hit rates.

6. Horizontal scaling via Redis Cluster and distributed vector databases.

This architecture can typically reduce 30–70% of LLM calls, significantly lowering inference costs and improving response latency across multiple applications.

If you’re looking for paid 1:1 mentorship with a strong focus on Low-Level Design (LLD)—especially Multithreading and Concurrency—along with High-Level Design (HLD), Data Structures & Algorithms (DSA), System Design, LLM/Generative AI systems, and research paper discussions, feel free to reach out.

The mentorship is tailored for interview preparation, backend engineering, distributed systems, AI infrastructure, and building a strong systems-thinking foundation.

📩 Contact: programmingappliedai@gmail.com

Functional Requirements (FR)

• Users should be able to interact with the AI agent using voice in real time.

• System should convert speech to text (ASR), generate responses using an LLM, and convert responses back to speech (TTS).

• System should support full-duplex conversations where users can interrupt the AI while it is speaking (barge-in).

• System should maintain conversational context and session history across multiple turns.

• System should support multilingual conversations and voice personalization.

Non-Functional Requirements (NFR)

• Support up to 10 million concurrent users globally.

• End-to-end voice response latency should be less than 300 ms.

• Achieve 99.99% availability with no single point of failure.

• Scale horizontally across regions with automatic failover.

• Optimize GPU utilization to maximize throughput and minimize inference costs.

Capacity Estimation

Assumptions:

• 10M concurrent users.

• Average audio packet every 100 ms.

• Average audio chunk size = 3 KB.

Incoming traffic:

10M × 10 chunks/sec
=
100M audio chunks/sec

Bandwidth:

100M × 3 KB
=
300 GB/sec

This requires a globally distributed architecture with edge processing.

Core APIs

Start Session

Request

POST /v1/session/start

{
  "userId": "u123",
  "language": "en",
  "voice": "female_1"
}

Response

{
  "sessionId": "sess_789",
  "websocketUrl": "wss://voice.company.com/session/sess_789"
}

Streaming Audio API

WebSocket

wss://voice.company.com/session/{sessionId}

Client continuously sends audio chunks.

{
  "sequence": 1,
  "audioChunk": "<binary>"
}

Session End API

POST /v1/session/end

{
  "sessionId": "sess_789"
}

Health API

GET /v1/health

High-Level Components / Microservices

• API Gateway

• WebSocket Gateway

• Session Management Service

• Streaming ASR Service

• Conversation Orchestrator

• Context/Memory Service

• LLM Inference Service

• TTS Service

• Barge-In Detection Service

• GPU Scheduler

• Model Serving Platform

• Redis Session Cache

• Kafka/Pulsar Event Bus

• Monitoring & Analytics Service

Database Schema

Session Table

CREATE TABLE sessions (
    session_id VARCHAR(128) PRIMARY KEY,
    user_id VARCHAR(128),
    status VARCHAR(32),
    started_at TIMESTAMP,
    last_activity TIMESTAMP
);

Conversation History

CREATE TABLE conversations (
    id BIGSERIAL PRIMARY KEY,
    session_id VARCHAR(128),
    role VARCHAR(16),
    message TEXT,
    created_at TIMESTAMP
);

User Voice Preferences

CREATE TABLE voice_profiles (
    user_id VARCHAR(128) PRIMARY KEY,
    language VARCHAR(32),
    preferred_voice VARCHAR(64)
);

Microservices Interaction

Step 1: User Starts Conversation

User opens application.

Request reaches API Gateway.

API Gateway forwards request to Session Service.

Session Service:

• Creates session.

• Allocates region.

• Stores metadata in Redis.

Returns WebSocket endpoint.

Step 2: User Starts Speaking

Audio stream arrives through WebSocket Gateway.

Gateway forwards chunks to Streaming ASR Service.

Example:

User:
"Book me a flight to London"

Audio is streamed continuously.

Step 3: Streaming ASR

Instead of waiting for complete speech:

Traditional ASR:

User speaks
     ↓
Speech ends
     ↓
ASR starts

High latency.

Streaming ASR:

Audio Chunk 1
Audio Chunk 2
Audio Chunk 3

ASR continuously emits partial transcripts.

Example:

"Book"
"Book me"
"Book me a flight"
"Book me a flight to London"

This saves hundreds of milliseconds.

Step 4: Conversation Orchestrator

Receives partial transcript.

Maintains conversation state.

Fetches:

• User profile

• Conversation memory

• Previous messages

Builds prompt.

Step 5: LLM Inference Service

Prompt sent to LLM.

Example:

User:
Book me a flight to London

Context:
User prefers economy seats.

LLM begins generating tokens immediately.

Streaming output:

Sure
Sure, I
Sure, I can
Sure, I can help

No need to wait for complete answer.

Step 6: TTS Service

As soon as first tokens arrive:

Sure,

TTS starts synthesis.

Produces audio stream immediately.

Pipeline:

LLM Tokens
    ↓
TTS
    ↓
Audio Chunks
    ↓
User

This overlap is critical for achieving <300 ms latency.

End-to-End Streaming Pipeline

User Speech
      ↓
Streaming ASR
      ↓
Partial Transcript
      ↓
LLM Streaming
      ↓
Partial Tokens
      ↓
Streaming TTS
      ↓
Audio Response

All stages run simultaneously.

Barge-In (Interruption Handling)

Problem

AI is speaking:

AI:
"Sure, I can help you book..."

User interrupts:

"No, I meant Paris."

System must stop speaking immediately.

Barge-In Flow

User audio detected.

Barge-In Service receives voice activity event.

User Voice Detected
       ↓
Stop Current TTS
       ↓
Cancel Current LLM Generation
       ↓
Start New ASR Stream

Conversation becomes:

AI:
Sure I can help...

User:
No, I meant Paris.

AI:
Understood. You want Paris.

Session Management

Session Service maintains:

• Session ID

• User preferences

• Current conversation state

• Active model

• Current TTS stream

• Current ASR stream

Hot session state stored in Redis.

Session
      ↓
Redis

Access latency:

< 5 ms

Long-term history stored in distributed database.

Autoscaling Strategy

ASR Autoscaling

Scale based on:

Active Audio Streams

Metric:

Streams per GPU

LLM Autoscaling

Scale based on:

Waiting Requests
Token Throughput
GPU Queue Length

TTS Autoscaling

Scale based on:

Concurrent Audio Synthesis Requests

Kubernetes HPA scales services independently.

GPU Utilization Optimization

Running one request per GPU wastes resources.

Bad:

GPU
 └─ Request A

Utilization:

15%

Use Continuous Batching.

Good:

GPU
 ├─ Req A
 ├─ Req B
 ├─ Req C
 ├─ Req D
 └─ Req E

Utilization:

80-95%

GPU Scheduler

Responsibilities:

• Assign requests to GPUs.

• Perform dynamic batching.

• Route to least loaded GPU.

• Handle model placement.

• Manage KV cache memory.

Latency Breakdown (<300 ms Target)

| Component       | Latency |
| --------------- | ------- |
| Network         | 20 ms   |
| Streaming ASR   | 50 ms   |
| Prompt Building | 10 ms   |
| LLM First Token | 80 ms   |
| Streaming TTS   | 80 ms   |
| Buffering       | 20 ms   |

Total:

≈ 260 ms

High-Level Architecture

Client
   |
WebSocket Gateway
   |
Session Service
   |
Conversation Orchestrator
   |
   +----------------+
   |                |
Streaming ASR   Memory Service
   |                |
   +-------+--------+
           |
           v
      LLM Service
           |
           v
       TTS Service
           |
           v
      Audio Stream
           |
         Client

Key Design Decisions

• Streaming ASR instead of batch ASR.

• Streaming token generation from LLM.

• Streaming TTS synthesis.

• Redis-backed session state.

• Immediate barge-in cancellation.

• Dynamic GPU batching.

• Regional deployment with edge WebSocket gateways.

• Stateless services with Kubernetes autoscaling.

• Event-driven communication via Kafka/Pulsar.

• Continuous GPU utilization monitoring and scheduling.

• Users should be able to submit natural language questions to the RAG system.

• System should retrieve the most relevant documents/chunks from the knowledge base.

• System should generate answers using retrieved context and an LLM.

• System should cache query embeddings to avoid repeated embedding generation.

• System should cache retrieval results to avoid repeated vector searches.

• System should cache final LLM responses for repeated or semantically similar queries.

• System should support semantic cache lookup based on embedding similarity.

• System should allow ingestion, update, and deletion of knowledge base documents.

• System should invalidate stale cache entries when underlying documents change.

• System should provide source citations/metadata used for answer generation.

• System should support multi-tenant isolation (different organizations/knowledge bases).

• System should expose cache hit/miss metrics for monitoring and analytics.

Non-Functional Requirements (NFR)

• Support 100M+ queries per day.

• Handle peak traffic of 10K–50K QPS.

• Achieve P95 query latency below 500 ms for cached requests.

• Achieve P99 latency below 2 seconds for uncached requests.

• Maintain 99.99% service availability.

• Scale horizontally without downtime.

• Ensure fault tolerance with no single point of failure.

• Support eventual consistency for distributed cache invalidation.

• Minimize LLM and vector search costs through high cache hit ratios.

• Maintain cache freshness after KB updates.

• Support multi-region deployments with local cache access.

• Ensure tenant-level security and data isolation.

• Enable observability through metrics, logs, and distributed tracing.

• Support rolling deployments and zero-downtime upgrades.

• Provide disaster recovery and backup mechanisms.

Core APIs

Query API

POST /v1/query

Request

{
  "query": "What is GPT-5?",
  "tenantId": "org123",
  "userId": "user456"
}

Response

{
  "answer": "GPT-5 is...",
  "sources": ["doc1", "doc2"],
  "cacheHit": "response_cache",
  "latencyMs": 45
}

Document Ingestion API

POST /v1/documents

Request

{
  "documentId": "doc123",
  "content": "document content",
  "metadata": {
    "category": "AI"
  }
}

Response

{
  "status": "success"
}

Document Update API

PUT /v1/documents/{documentId}

Request

{
  "content": "updated content",
  "version": 5
}

Response

{
  "status": "updated"
}

Cache Invalidation API

POST /v1/cache/invalidate

Request

{
  "documentId": "doc123"
}

Response

{
  "status": "invalidated"
}

Database Schema

Documents

CREATE TABLE documents (
    document_id VARCHAR(64) PRIMARY KEY,
    content TEXT,
    version BIGINT,
    metadata JSONB,
    updated_at TIMESTAMP
);

Embeddings

CREATE TABLE embeddings (
    embedding_id BIGSERIAL PRIMARY KEY,
    document_id VARCHAR(64),
    embedding VECTOR(1536),
    version BIGINT
);

Response Cache Metadata

CREATE TABLE response_cache (
    cache_key VARCHAR(255) PRIMARY KEY,
    answer TEXT,
    kb_version BIGINT,
    created_at TIMESTAMP,
    expires_at TIMESTAMP
);

Retrieval Cache Metadata

CREATE TABLE retrieval_cache (
    query_hash VARCHAR(255) PRIMARY KEY,
    document_ids JSONB,
    kb_version BIGINT,
    created_at TIMESTAMP
);

Microservices

• API Gateway

• Query Orchestrator Service

• Embedding Service

• Embedding Cache Service

• Retrieval Cache Service

• Vector Search Service

• Response Cache Service

• LLM Service

• Knowledge Base Service

• Cache Invalidation Service

• Metadata Service

• Monitoring & Analytics Service

• Kafka/Pulsar Event Bus

High-Level Request Flow

Client
   |
API Gateway
   |
Query Orchestrator
   |
   +--> Response Cache
   |        |
   |        +--> Hit --> Return Answer
   |
   +--> Embedding Cache
   |        |
   |        +--> Miss --> Embedding Service
   |
   +--> Retrieval Cache
   |        |
   |        +--> Miss --> Vector Search
   |
   +--> LLM Service
   |
Return Answer

Microservices Interaction (Request Flow)

1. Client → API Gateway

• User sends a query such as “What is GPT-5?”

• API Gateway performs authentication, rate limiting, request validation, and routing.

• Request is forwarded to the Query Orchestrator Service.

2. Query Orchestrator → Response Cache Service

• Query Orchestrator first checks whether the final answer is already available in the Response Cache.

• Cache lookup can happen in two ways:

  • Exact match lookup using query hash.

  • Semantic lookup using query embedding similarity.

• If a valid cache entry exists, the response is returned immediately.

• This is the fastest path and avoids all downstream processing.

Example

"What is GPT-5?"

already exists in cache.

Response is returned in ~10-50 ms.

3. Query Orchestrator → Embedding Cache Service

If response cache misses:

• System needs query embedding.

• Query Orchestrator checks Embedding Cache.

Cache Hit

• Cached embedding vector is returned.

Cache Miss

• Request goes to Embedding Service.

• Embedding model generates vector.

• Generated vector is stored in Embedding Cache for future use.

Example:

"What is GPT-5?"

↓

[0.12, 0.45, 0.98 ...]

4. Query Orchestrator → Retrieval Cache Service

Once query embedding is available:

• System checks Retrieval Cache.

• Retrieval Cache stores previously retrieved document IDs for similar queries.

Cache Hit

Query Embedding
       ↓
[Doc1, Doc7, Doc9]

Documents are returned directly.

Cache Miss

System performs vector search.

5. Query Orchestrator → Vector Search Service

• Vector Search Service receives query embedding.

• Searches Vector Database using Approximate Nearest Neighbor (ANN) algorithms such as HNSW or IVF.

• Retrieves Top-K relevant chunks/documents.

Example:

Query:
"What is GPT-5?"

Retrieved:

Doc12
Doc19
Doc42

Results are stored in Retrieval Cache.

6. Query Orchestrator → Knowledge Base Service

• Query Orchestrator requests full content for retrieved document IDs.

• KB Service fetches chunk text and metadata.

Example:

Doc12 → GPT-5 architecture
Doc19 → GPT-5 capabilities
Doc42 → GPT-5 release information

These chunks form the context.

7. Query Orchestrator → LLM Service

• Query and retrieved context are sent to the LLM Service.

• LLM Service constructs prompt.

Example:

System Prompt

Retrieved Context:
[Doc12]
[Doc19]
[Doc42]

User Query:
What is GPT-5?

• LLM generates final answer.

8. Query Orchestrator → Response Cache Service

Before returning answer:

• Generated response is stored in Response Cache.

• Metadata is stored:

  • Query hash

  • Embedding hash

  • Retrieved document IDs

  • KB version

  • TTL

This allows future requests to avoid LLM generation.

9. Query Orchestrator → Client

Final response is returned.

{
  "answer": "GPT-5 is...",
  "sources": ["Doc12", "Doc19", "Doc42"],
  "cacheHit": false
}

Document Update Flow

When knowledge base content changes:

1. Admin → KB Service

Document update request arrives.

Doc19 Updated

2. KB Service → Vector Search Service

• New document embedding is generated.

• Old vector is replaced in Vector DB.

3. KB Service → Kafka/Event Bus

Publish event:

DocumentUpdated
{
   docId: 19,
   version: 22
}

4. Event Bus → Cache Invalidation Service

Cache Invalidation Service consumes update event.

5. Cache Invalidation Service → Retrieval Cache

Invalidate retrieval entries containing Doc19.

QueryA → [Doc19, Doc22]
QueryB → [Doc7, Doc19]

Both entries removed.

6. Cache Invalidation Service → Response Cache

Invalidate responses generated using Doc19.

ResponseX
ResponseY
ResponseZ

All removed.

7. Fresh Requests Use New Data

Future queries:

Query
  ↓
Vector Search
  ↓
Updated Doc19
  ↓
LLM

Users receive fresh answers.

End-to-End Summary

User Query
    ↓
API Gateway
    ↓
Query Orchestrator
    ↓
Response Cache ?
    ↓ miss
Embedding Cache ?
    ↓ miss
Embedding Service
    ↓
Retrieval Cache ?
    ↓ miss
Vector Search
    ↓
Knowledge Base
    ↓
LLM Service
    ↓
Response Cache Update
    ↓
User

The Query Orchestrator Service is the central coordinator. All other services (cache, embedding, vector search, KB, LLM) are specialized services that it calls in sequence to answer the query efficiently.

1. Schedule device backups.

2. Trigger automatic backups as per schedule.

3. View backup status and history.

4. Restore device data from selected backup.

5. Send backup/restore success or failure notifications.

NFR:

1. High availability of backup and restore services.

2. Secure data storage and transmission through encryption.

3. Scalable to support multiple devices concurrently.

4. Minimal impact on device performance during backup.

5. Maintain backup data integrity and reliability.

API’s with Request and Response

1. Create Backup Schedule

API: POST /api/v1/backup/schedules

Request

{
  "deviceId": "DEV001",
  "frequency": "DAILY",
  "time": "02:00"
}

Response

{
  "scheduleId": "SCH001",
  "status": "CREATED",
  "message": "Backup schedule created successfully"
}

2. Trigger Backup

API: POST /api/v1/backups

Request

{
  "deviceId": "DEV001",
  "backupType": "FULL"
}

Response

{
  "backupId": "BKP001",
  "status": "IN_PROGRESS",
  "message": "Backup initiated successfully"
}

3. Get Backup Status

API: GET /api/v1/backups/{backupId}

Request

{
  "backupId": "BKP001"
}

Response

{
  "backupId": "BKP001",
  "deviceId": "DEV001",
  "status": "COMPLETED",
  "backupTime": "2026-07-08T02:00:00Z"
}

4. Get Available Backups for Restore

API: GET /api/v1/devices/{deviceId}/backups

Request

{
  "deviceId": "DEV001"
}

Response

{
  "backups": [
    {
      "backupId": "BKP001",
      "createdAt": "2026-07-08T02:00:00Z"
    },
    {
      "backupId": "BKP002",
      "createdAt": "2026-07-07T02:00:00Z"
    }
  ]
}

5. Restore Backup

API: POST /api/v1/restores

Request

{
  "deviceId": "DEV001",
  "backupId": "BKP001"
}

Response

{
  "restoreId": "RST001",
  "status": "IN_PROGRESS",
  "message": "Restore initiated successfully"
}

6. Get Restore Status

API: GET /api/v1/restores/{restoreId}

Request

{
  "restoreId": "RST001"
}

Response

{
  "restoreId": "RST001",
  "deviceId": "DEV001",
  "status": "COMPLETED",
  "restoredAt": "2026-07-08T05:00:00Z"
}

Databases and Schema

Database Choice

• PostgreSQL – Backup schedules, backup metadata, restore jobs, notifications, audit logs

• Redis – Caching and job scheduling

• Object Storage (S3 compatible) – Backup file storage

• Elasticsearch/OpenSearch (Optional) – Log search and analytics

Tables

Device

• device_id (PK, UUID)

• device_name (VARCHAR)

• device_type (VARCHAR)

• owner_id (VARCHAR)

• status (VARCHAR)

• created_at (TIMESTAMP)

Backup_Schedule

• schedule_id (PK, UUID)

• device_id (FK, UUID)

• frequency (VARCHAR)

• backup_time (TIME)

• is_active (BOOLEAN)

• created_at (TIMESTAMP)

• updated_at (TIMESTAMP)

Backup_Job

• backup_id (PK, UUID)

• device_id (FK, UUID)

• schedule_id (FK, UUID)

• backup_type (VARCHAR)

• status (VARCHAR)

• file_size (BIGINT)

• storage_path (VARCHAR)

• started_at (TIMESTAMP)

• completed_at (TIMESTAMP)

Backup_Version

• version_id (PK, UUID)

• backup_id (FK, UUID)

• version_number (INT)

• checksum (VARCHAR)

• retention_days (INT)

• created_at (TIMESTAMP)

Restore_Job

• restore_id (PK, UUID)

• device_id (FK, UUID)

• backup_id (FK, UUID)

• status (VARCHAR)

• initiated_by (VARCHAR)

• started_at (TIMESTAMP)

• completed_at (TIMESTAMP)

Notification

• notification_id (PK, UUID)

• device_id (FK, UUID)

• event_type (VARCHAR)

• recipient (VARCHAR)

• status (VARCHAR)

• sent_at (TIMESTAMP)

Audit_Log

• audit_id (PK, UUID)

• entity_type (VARCHAR)

• entity_id (UUID)

• action (VARCHAR)

• performed_by (VARCHAR)

• timestamp (TIMESTAMP)

• remarks (TEXT)

Relationships

• Device → Backup_Schedule (1:N)

• Device → Backup_Job (1:N)

• Backup_Job → Backup_Version (1:N)

• Device → Restore_Job (1:N)

• Backup_Job → Restore_Job (1:N)

• Device → Notification (1:N)

• All entities → Audit_Log (1:N)

Microservices Architecture – Device Backup Scheduler & Restore

1. API Gateway

   • Single entry point for all client requests

   • Authentication, authorization, routing, rate limiting

2. Device Management Service

   • Manages device registration and device metadata

   • Provides device details to backup and restore services

3. Backup Scheduler Service

   • Creates and manages backup schedules

   • Triggers backup jobs based on configured frequency

4. Backup Service

   • Executes backup operations

   • Stores backup metadata and uploads backup files to object storage

5. Restore Service

   • Retrieves backup data

   • Initiates and monitors restore operations

6. Notification Service

   • Sends email/SMS/in-app notifications for backup and restore events

7. Audit & Logging Service

   • Records user actions and system events

   • Provides audit trail for compliance and troubleshooting

8. Storage Service

   • Manages backup file storage and retrieval from object storage

9. Auth Service

   • User authentication and authorization

   • Issues and validates JWT/OAuth tokens

Service Interaction Flow

Backup Scheduling

1. Client → API Gateway

2. API Gateway → Auth Service (token validation)

3. API Gateway → Backup Scheduler Service

4. Backup Scheduler Service → Device Management Service (validate device)

5. Backup Scheduler Service → PostgreSQL (save schedule)

Backup Execution

1. Scheduler triggers Backup Service

2. Backup Service → Device Management Service (device details)

3. Backup Service → Storage Service (store backup file)

4. Backup Service → PostgreSQL (save metadata)

5. Backup Service → Notification Service (backup status)

6. Notification Service → User

Restore Execution

1. Client → API Gateway

2. API Gateway → Restore Service

3. Restore Service → Backup Service (backup metadata)

4. Restore Service → Storage Service (retrieve backup file)

5. Restore Service → Device Management Service (device validation)

6. Restore Service → PostgreSQL (update restore status)

7. Restore Service → Notification Service (restore status)

Audit Flow

1. Every service publishes events

2. Audit & Logging Service consumes events

3. Audit records stored in Audit_Log table

Communication Pattern

• Synchronous: REST APIs (API Gateway ↔ Services)

• Asynchronous: Kafka/RabbitMQ events for Backup Trigger, Restore Trigger, Notifications, Audit Logging

• Database per service pattern

• Object Storage for backup files

• Redis for caching and scheduled job execution

High-Level Flow
Client → API Gateway → Auth Service → Backup Scheduler Service → Backup Service → Storage Service → Restore Service → Notification Service → Audit Service → Databases/Object Storage/Event Bus.

1. Deploy containers on available compute nodes using Docker/OCI compliant images.

2. Migrate running containers across nodes or clusters while preserving application state and network connectivity.

3. Manage container lifecycle operations including start, stop, restart, update, and scaling.

4. Create and manage virtual networks, service discovery, and cross-network communication between containers.

5. Continuously monitor container and node health, collect metrics, detect failures, and trigger automatic recovery actions.

Non-Functional Requirements

1. Ensure high availability with a target uptime of at least 99.99%.

2. Scale to support up to 100,000 nodes and 1 million running containers.

3. Achieve container deployment and scheduling latency of less than 5 seconds.

4. Provide fault tolerance such that node, network, or service failures do not cause application downtime.

5. Enforce enterprise-grade security through RBAC, TLS-encrypted communication, image signing, verification, and audit logging.

Core APIs

Deploy Container

POST /v1/containers

Request

{
  "image":"nginx:latest",
  "cpu":"2",
  "memory":"4GB",
  "network_id":"net-123",
  "replicas":3
}

Response

{
  "container_id":"ctr-123",
  "status":"DEPLOYING"
}

Get Container

GET /v1/containers/{containerId}

Response

{
  "container_id":"ctr-123",
  "status":"RUNNING",
  "node_id":"node-45",
  "ip":"10.0.1.20"
}

Migrate Container

POST /v1/containers/{containerId}/migrate

Request

{
  "target_node":"node-99"
}

Response

{
  "migration_id":"mig-111",
  "status":"IN_PROGRESS"
}

Scale Container

POST /v1/containers/{containerId}/scale

Request

{
  "replicas":20
}

Response

{
  "status":"SCALING"
}

Create Network

POST /v1/networks

Request

{
  "name":"payment-network",
  "cidr":"10.10.0.0/16"
}

Response

{
  "network_id":"net-123"
}

Database Choice

| Component          | DB            |
| ------------------ | ------------- |
| Container Metadata | PostgreSQL    |
| Cluster State      | etcd          |
| Metrics            | Cassandra     |
| Logs               | Elasticsearch |
| Cache              | Redis         |

Database Schemas

Container

{
  "container_id":"ctr-123",
  "image":"nginx:latest",
  "status":"RUNNING",
  "node_id":"node-45",
  "network_id":"net-123",
  "cpu":"2",
  "memory":"4GB",
  "created_at":"timestamp"
}

Node

{
  "node_id":"node-45",
  "hostname":"worker-1",
  "ip":"192.168.1.10",
  "cpu_total":"64",
  "cpu_available":"20",
  "memory_total":"256GB",
  "memory_available":"120GB",
  "status":"HEALTHY"
}

Cluster

{
  "cluster_id":"cluster-1",
  "name":"us-east-cluster",
  "region":"us-east",
  "status":"ACTIVE"
}

Network

{
  "network_id":"net-123",
  "name":"payment-network",
  "cidr":"10.10.0.0/16",
  "gateway":"10.10.0.1"
}

Migration Job

{
  "migration_id":"mig-111",
  "container_id":"ctr-123",
  "source_node":"node-45",
  "target_node":"node-99",
  "status":"IN_PROGRESS",
  "created_at":"timestamp"
}

High Level Components

                    +------------------+
                    |      User        |
                    +---------+--------+
                              |
                              v
                    +------------------+
                    |    API Gateway   |
                    +---------+--------+
                              |
          -----------------------------------------
          |              |            |           |
          v              v            v           v

+----------------+ +--------------+ +---------------+
| Deployment     | | Migration    | | Network       |
| Service        | | Service      | | Service       |
+-------+--------+ +------+-------+ +-------+-------+
        |                 |                 |
        |                 |                 |
        v                 v                 v

+--------------------------------------------------+
|              Scheduler Service                   |
+--------------------------------------------------+
                        |
                        v

+--------------------------------------------------+
|              Cluster Manager                     |
+--------------------------------------------------+
                        |
        -----------------------------------
        |                |               |
        v                v               v

+------------+ +-------------+ +-------------+
| Node Agent | | Node Agent  | | Node Agent  |
| Node-1     | | Node-2      | | Node-N      |
+------------+ +-------------+ +-------------+

Container Deployment Flow

User
 |
 | Deploy Container
 v

API Gateway
 |
 v

Deployment Service
 |
 v

Scheduler
 |
 | Find best node
 v

Cluster Manager
 |
 v

Node Agent
 |
 | Pull Image
 | Create Container
 v

Container Runtime
 |
 v

Container Running

Container Migration Flow

User
 |
 | Migrate Container
 v

Migration Service
 |
 v

Scheduler
 |
 | Select Target Node
 v

Checkpoint Service
 |
 | Save Container State
 v

Target Node Agent
 |
 | Restore State
 v

Network Service
 |
 | Update Routing
 v

Container Running on New Node

Microservices

1. API Gateway

• Authentication
  

• Rate limiting
  

• Routing
  

2. Deployment Service

• Container deployment
  

• Scaling requests
  

3. Scheduler Service

• Node selection
  

• Resource allocation
  

4. Migration Service

• Container relocation
  

• State transfer
  

5. Network Service

• Overlay networking
  

• Service discovery
  

6. Cluster Manager

• Cluster state management
  

• Node registration
  

7. Health Monitoring Service

• Metrics collection
  

• Alerting
  

8. Auto-Healing Service

• Restart failed containers
  

• Reschedule workloads
  

9. Image Registry Service

• Image storage
  

• Image versioning
  

10. Audit & Security Service

• RBAC
  

• Secrets management
  

• Compliance logs
  

Microservice Interaction

                 +----------------+
                 | API Gateway    |
                 +-------+--------+
                         |
     ------------------------------------------------
     |              |              |                |
     v              v              v                v

Deployment   Migration      Network         Monitoring
 Service      Service       Service           Service
     |              |             |                |
     ------------------------------------------------
                         |
                         v

                  Scheduler
                         |
                         v

                 Cluster Manager
                         |
         ---------------------------------
         |               |               |
         v               v               v

      Node Agent     Node Agent      Node Agent
         |               |               |
         ---------------------------------
                         |
                         v

                Container Runtime
             (containerd / Docker)

Bottlenecks & Scaling

Scheduler Bottleneck

• Use sharded schedulers.
  

• Leader election via etcd.
  

Metrics Explosion

• Store metrics in Cassandra.
  

• Aggregate before persistence.
  

Container Migration

• Use incremental checkpoints.
  

• Transfer only changed memory pages.
  

Multi-Region Networking

• Overlay network (VXLAN/WireGuard).
  

• Global service discovery.
  

Cluster State

• Store in etcd with Raft consensus.
  

• Multiple replicas for HA.
  

This design is very close to how modern orchestration platforms such as Kubernetes, Nomad, and Docker Swarm manage large-scale container deployments across clusters and networks.

📩 Contact: programmingappliedai@gmail.com

Functional Requirements (FR)

1. Asset Tracking

• System should track assets (packages, vehicles, devices, orders) in real-time.

• Support GPS coordinate updates.

2. Location History

• Store and retrieve complete location history of tracked entities.

• Allow querying historical movement.

3. Geofence Management

• Create geofences around specific locations.

• Trigger events when tracked entities enter or exit geofences.

4. Alerts & Notifications

• Generate alerts for:

  • Geofence breach

  • Device offline

  • Route deviation

  • Low battery

5. Tracking Dashboard

• Display current location, status, speed, route, and historical trails.

• Support filtering and search.

Non-Functional Requirements (NFR)

1. High Availability

• 99.99% uptime.

2. Scalability

• Support 10M+ tracked devices.

• Support 100K location updates/sec.

3. Low Latency

• Location updates visible within 2-3 seconds.

4. Durability

• No loss of tracking events.

5. Security

• OAuth/JWT authentication.

• Encrypted data in transit and at rest.

APIs

1. Register Tracking Device

Request

POST /api/v1/devices

{
  "deviceId": "DEV123",
  "assetType": "VEHICLE",
  "assetId": "CAR123"
}

Response

{
  "deviceId": "DEV123",
  "status": "REGISTERED"
}

2. Update Location

Request

POST /api/v1/location/update

{
  "deviceId": "DEV123",
  "latitude": 12.9716,
  "longitude": 77.5946,
  "speed": 45,
  "timestamp": 1745000000
}

Response

{
  "status": "SUCCESS"
}

3. Get Current Location

Request

GET /api/v1/location/current/{deviceId}

Response

{
  "deviceId": "DEV123",
  "latitude": 12.9716,
  "longitude": 77.5946,
  "speed": 45,
  "timestamp": 1745000000
}

4. Get Location History

Request

GET /api/v1/location/history

{
  "deviceId": "DEV123",
  "startTime": 1745000000,
  "endTime": 1746000000
}

Response

{
  "locations": [
    {
      "latitude": 12.9716,
      "longitude": 77.5946,
      "timestamp": 1745000000
    }
  ]
}

5. Create Geofence

Request

POST /api/v1/geofence

{
  "geofenceId": "GF123",
  "name": "Warehouse",
  "centerLatitude": 12.9716,
  "centerLongitude": 77.5946,
  "radiusMeters": 500
}

Response

{
  "status": "CREATED"
}

6. Get Alerts

Request

GET /api/v1/alerts/{deviceId}

Response

{
  "alerts": [
    {
      "alertType": "GEOFENCE_EXIT",
      "timestamp": 1745000000
    }
  ]
}

Database Design

Device Table

{
  "deviceId": "DEV123",
  "assetId": "CAR123",
  "assetType": "VEHICLE",
  "status": "ACTIVE",
  "registeredAt": "2026-01-01T10:00:00Z"
}

Partition Key:

deviceId

Database:

PostgreSQL

Current Location Table

{
  "deviceId": "DEV123",
  "latitude": 12.9716,
  "longitude": 77.5946,
  "speed": 45,
  "lastUpdatedAt": "2026-01-01T10:10:00Z"
}

Partition Key:

deviceId

Database:

Redis

Location History Table

{
  "deviceId": "DEV123",
  "timestamp": 1745000000,
  "latitude": 12.9716,
  "longitude": 77.5946,
  "speed": 45
}

Partition Key:

deviceId

Sort Key:

timestamp

Database:

Cassandra

Geofence Table

{
  "geofenceId": "GF123",
  "name": "Warehouse",
  "centerLatitude": 12.9716,
  "centerLongitude": 77.5946,
  "radiusMeters": 500,
  "createdAt": "2026-01-01T10:00:00Z"
}

Database:

PostgreSQL

Alert Table

{
  "alertId": "ALT123",
  "deviceId": "DEV123",
  "alertType": "GEOFENCE_EXIT",
  "message": "Vehicle left warehouse zone",
  "createdAt": "2026-01-01T10:15:00Z"
}

Database:

PostgreSQL

High Level Architecture

                +----------------+
                | Mobile Device  |
                | GPS Tracker    |
                +--------+-------+
                         |
                         v
                +----------------+
                | API Gateway    |
                +--------+-------+
                         |
                         v
                +----------------+
                | Tracking       |
                | Service        |
                +--------+-------+
                         |
          +--------------+--------------+
          |                             |
          v                             v

 +----------------+         +-------------------+
 | Redis Current  |         | Kafka             |
 | Location Cache |         | Location Stream   |
 +----------------+         +---------+---------+
                                      |
                                      |
             +------------------------+------------------+
             |                                           |
             v                                           v

   +-------------------+                     +------------------+
   | Location History  |                     | Geofence Engine  |
   | Service           |                     | Service          |
   +--------+----------+                     +--------+---------+
            |                                         |
            v                                         v

      +------------+                          +--------------+
      | Cassandra  |                          | Alert Service|
      +------------+                          +------+-------+
                                                      |
                                                      v
                                            +------------------+
                                            | Notification     |
                                            | Service          |
                                            +------------------+

Microservices

1. Device Service

Responsibilities:

• Device registration
  

• Device lifecycle management
  

• Device metadata storage
  

Database:

PostgreSQL

2. Tracking Service

Responsibilities:

• Receive GPS updates
  

• Validate payload
  

• Publish events to Kafka
  

Database:

Redis

3. Location History Service

Responsibilities:

• Persist location updates
  

• Historical queries
  

Database:

Cassandra

4. Geofence Service

Responsibilities:

• Geofence creation
  

• Entry/exit detection
  

Database:

PostgreSQL

5. Alert Service

Responsibilities:

• Generate alerts
  

• Store alert history
  

Database:

PostgreSQL

6. Notification Service

Responsibilities:

• SMS notifications
  

• Email notifications
  

• Push notifications
  

Database:

None (Stateless)

End-to-End Flow

1. GPS Device sends location update.

2. API Gateway forwards request to Tracking Service.

3. Tracking Service:
   - Validates request
   - Updates Redis current location
   - Publishes LocationEvent to Kafka

4. Location History Service consumes event.
   - Stores event in Cassandra

5. Geofence Service consumes event.
   - Checks geofence rules

6. If breach detected:
   - Generate Alert Event

7. Alert Service stores alert.

8. Notification Service sends SMS/Email/Push.

9. Dashboard fetches:
   - Current location from Redis
   - Historical route from Cassandra
   - Alerts from PostgreSQL

Scale Estimation

10 Million Devices

Average Update Frequency = 30 seconds

Writes/sec
= 10,000,000 / 30
≈ 333,000 writes/sec

Daily Location Records
≈ 28 Billion records/day

Storage Choice:
- Redis -> Current State
- Kafka -> Streaming
- Cassandra -> Time Series History
- PostgreSQL -> Metadata & Alerts

This design is suitable for a logistics, fleet-management, package-tracking, or IoT tracking platform operating at internet scale.

📩 Contact: programmingappliedai@gmail.com

Example:

Search Query: "red"
Expected Output: List of all file IDs/names containing the word "red".

Traffic:

• ~10 lakh requests/day (~11.5 requests/sec average)

• Assume:

  • Exact keyword matching only

  • No typo tolerance

  • Words contain only a-z

  • Files can be added/updated periodically

Functional Requirements (FR)

FR1. File Ingestion

System should accept and process files for indexing.

FR2. Word Extraction & Tokenization

System should parse files and extract normalized words (a-z only).

FR3. Inverted Index Creation

System should maintain an inverted index mapping:

word → list of file_ids

Example:

red → [file1, file8, file20]

FR4. Search API

User should be able to search for a word and retrieve all matching files.

FR5. Incremental Updates

System should support:

• New file addition
  

• File modification
  

• File deletion
  
  without rebuilding the complete index.
  

Non-Functional Requirements (NFR)

NFR1. Low Latency

Search response time should be under 100ms.

NFR2. High Availability

Search service should remain available during node failures.

NFR3. Scalability

System should scale horizontally as:

• Number of files increases
  

• Query traffic increases
  

NFR4. Durability

Indexed metadata and file mappings should not be lost.

NFR5. Efficient Storage

Index storage should be optimized using compression and deduplication.

High Level Architecture

                +------------------+
                |   Client/User    |
                +--------+---------+
                         |
                         v
               +---------+----------+
               |   Search API Layer |
               +---------+----------+
                         |
              +----------+-----------+
              |                      |
              v                      v
     +--------+-------+      +-------+--------+
     |   Query Cache  |      | Search Service |
     |    (Redis)     |      +-------+--------+
     +----------------+              |
                                     v
                          +----------+----------+
                          |   Inverted Index    |
                          |  (ElasticSearch /  |
                          |    Lucene Index)   |
                          +----------+----------+
                                     |
                  +------------------+------------------+
                  |                                     |
                  v                                     v
        +---------+---------+             +-------------+----------+
        | Index Builder     |             | Metadata DB            |
        | Tokenizer Engine  |             | (File Metadata Store)  |
        +---------+---------+             +------------------------+
                  |
                  v
         +--------+--------+
         | Distributed File |
         | Storage (S3/HDFS)|
         +------------------+

Core Idea — Inverted Index

Traditional search over 1 million files is expensive:

For every query:
   Scan all files

Complexity:

O(number_of_files)

Instead, build an inverted index.

Inverted Index Structure

{
   "red"  : [1, 7, 10, 55],
   "blue" : [2, 3, 10],
   "car"  : [5, 6]
}

Search becomes:

Lookup(word)

Complexity:

O(1) or O(logN)

depending on storage engine.

Components

1. File Storage Service

Stores original files.

Possible storage:

• HDFS
  

• S3
  

• Blob storage
  

Each file gets:

{
  "file_id": "f123",
  "path": "/docs/a.txt",
  "uploaded_at": 1712345678
}

2. File Ingestion Service

Responsible for:

• Accepting files
  

• Sending indexing jobs to queue
  

Flow:

Upload File
    ↓
Generate file_id
    ↓
Push indexing job to Kafka/SQS

3. Tokenizer / Parser Service

Reads file contents and extracts words.

Steps:

• Lowercase conversion
  

• Remove punctuation
  

• Split into words
  

Example:

"The Red Car"

becomes:

["the", "red", "car"]

4. Index Builder

Builds inverted index.

Example:

file1 -> ["red", "car"]

Index:
red -> file1
car -> file1

5. Search Service

Handles user queries.

Flow:

1. Receive query "red"
   

2. Check Redis cache
   

3. Lookup inverted index
   

4. Return matching files
   

Database Design

A. File Metadata Table

{
  "file_id": "string",
  "file_name": "string",
  "storage_path": "string",
  "uploaded_at": "timestamp",
  "version": "int",
  "status": "ACTIVE | DELETED"
}

Database:

• PostgreSQL
  
  or
  

• MySQL
  

B. Inverted Index Storage

Logical Schema

{
  "word": "red",
  "document_frequency": 1200,
  "posting_list": [
    "file1",
    "file8",
    "file20"
  ]
}

Preferred technologies:

• ElasticSearch
  

• Apache Lucene
  

• OpenSearch
  

Search Flow

User Searches "red"
        |
        v
Search API
        |
        v
Redis Cache Check
   |          |
Hit          Miss
 |             |
Return      Query Index
                 |
                 v
        Get Posting List
                 |
                 v
         Fetch Metadata
                 |
                 v
            Return Files

File Indexing Flow

File Upload
    |
    v
Ingestion Service
    |
    v
Message Queue (Kafka)
    |
    v
Indexer Worker
    |
    v
Tokenizer
    |
    v
Update Inverted Index

Scaling Strategy

Read Scaling

Search traffic is read-heavy.

Use:

• Multiple search replicas
  

• Load balancer
  

• Redis cache
  

Index Partitioning (Sharding)

Partition words alphabetically:

a-f  → shard1
g-l  → shard2
m-r  → shard3
s-z  → shard4

or use:

• Consistent hashing
  

Caching Strategy

Popular searches cached in Redis.

Example:

key   = "red"
value = [file1, file2, file8]
TTL   = 10 mins

Reduces repeated index lookups.

Estimation

Requests

10 lakh/day
≈ 11.5 req/sec average
Peak ≈ 100-200 req/sec

Very manageable with:

• Redis
  

• ElasticSearch cluster
  

Fault Tolerance

Replication

Maintain:

• Replica index shards
  

• Multi-node search cluster
  

Queue Durability

Kafka persists indexing jobs.

If indexer crashes:

• Job replay possible
  

Optimizations

1. Compression

Store posting lists using:

• Delta encoding
  

• Bitmaps
  

2. Skip Lists

Improve search traversal for large posting lists.

3. Batch Indexing

Instead of updating index per file:

• Batch updates every few seconds.
  

Bottlenecks

Hot Keywords

Words like:

the, is, and

can have huge posting lists.

Solution:

• Stop-word filtering
  

• Compressed bitmaps
  

Technologies

| Component     | Technology            |
| ------------- | --------------------- |
| API Layer     | Go / Java Spring Boot |
| Queue         | Kafka                 |
| Cache         | Redis                 |
| Search Engine | ElasticSearch         |
| Metadata DB   | PostgreSQL            |
| Storage       | S3 / HDFS             |

Final Design Summary

The system uses an inverted index to efficiently map:

word → files

Core benefits:

• Fast lookup
  

• Horizontal scalability
  

• Incremental indexing
  

• Low latency search
  

The architecture separates:

• File storage
  

• Index building
  

• Search serving
  

allowing the system to scale independently for ingestion and querying.

📩 Contact: programmingappliedai@gmail.com

Functional Requirements (FR)

1. Penalty/Tax Rule Management
   System should allow admins to define and manage dynamic rules for penalties/taxes based on:

   • Minimum rides completed

   • Cancellation rate

   • Average rider/driver ratings

   • Inactive days

   • Peak-hour participation

   • Fraud signals

2. Periodic Evaluation Engine
   System should periodically evaluate riders/drivers (daily/weekly/monthly) against configured rules and determine applicable penalties or taxes.

3. Penalty Calculation & Ledger Management
   System should calculate penalty amounts accurately and maintain a financial ledger/history for every deduction, adjustment, refund, or waiver.

4. Notification & Transparency
   Users should receive notifications explaining:

   • Why the penalty was applied

   • Rule violated

   • Amount deducted

   • Appeal process

   • Deadline for compliance

5. Appeal & Dispute Handling
   Riders/drivers should be able to raise disputes against penalties, upload proofs, and track appeal status through the platform.

Non-Functional Requirements (NFR)

1. Scalability
   System should support millions of riders/drivers and evaluate penalties for large datasets efficiently during monthly batch processing.

2. High Availability
   Critical services like rule evaluation, deductions, and notifications should maintain high availability (99.9%+) to avoid financial inconsistencies.

3. Consistency & Accuracy
   Financial calculations and deductions must be strongly consistent to prevent duplicate or incorrect penalties.

4. Extensibility
   System should support adding new penalty rules and policy types without requiring major architectural changes.

5. Auditability & Compliance
   Every rule evaluation, deduction, override, and admin action should be fully auditable for regulatory and legal compliance.

APIs for Penalty/Tax Enforcement System

1. Create Penalty Rule

API

POST /api/v1/rules

Request

{
  "ruleName": "Minimum Monthly Ride Rule",
  "entityType": "DRIVER",
  "condition": {
    "metric": "MONTHLY_RIDES",
    "operator": "LESS_THAN",
    "value": 100
  },
  "penalty": {
    "type": "FIXED",
    "amount": 500
  },
  "evaluationFrequency": "MONTHLY",
  "isActive": true
}

Response

{
  "ruleId": "RULE_12345",
  "status": "CREATED",
  "createdAt": "2026-05-26T10:00:00Z"
}

2. Evaluate User Eligibility for Penalty

API

POST /api/v1/evaluations/run

Request

{
  "entityType": "DRIVER",
  "entityId": "DRIVER_789",
  "evaluationPeriod": "2026-05"
}

Response

{
  "entityId": "DRIVER_789",
  "evaluationPeriod": "2026-05",
  "violations": [
    {
      "ruleId": "RULE_12345",
      "metricValue": 72,
      "expectedValue": 100,
      "penaltyAmount": 500
    }
  ],
  "totalPenalty": 500,
  "status": "PENALTY_APPLIED"
}

3. Fetch Penalty History

API

GET /api/v1/penalties/{entityId}

Response

{
  "entityId": "DRIVER_789",
  "penalties": [
    {
      "penaltyId": "PEN_1001",
      "ruleId": "RULE_12345",
      "amount": 500,
      "status": "DEDUCTED",
      "createdAt": "2026-05-26T12:00:00Z"
    },
    {
      "penaltyId": "PEN_1002",
      "ruleId": "RULE_99999",
      "amount": 200,
      "status": "WAIVED",
      "createdAt": "2026-04-26T12:00:00Z"
    }
  ]
}

4. Deduct Penalty from Wallet

API

POST /api/v1/wallets/deduct

Request

{
  "entityId": "DRIVER_789",
  "penaltyId": "PEN_1001",
  "amount": 500,
  "currency": "INR"
}

Response

{
  "transactionId": "TXN_56789",
  "walletBalance": 3200,
  "status": "SUCCESS"
}

5. Raise Penalty Dispute

API

POST /api/v1/disputes

Request

{
  "penaltyId": "PEN_1001",
  "entityId": "DRIVER_789",
  "reason": "Ride count calculation incorrect",
  "attachments": [
    "s3://proofs/screenshot1.png"
  ]
}

Response

{
  "disputeId": "DISP_4567",
  "status": "UNDER_REVIEW",
  "createdAt": "2026-05-26T13:00:00Z"
}

6. Get Evaluation Summary Dashboard

API

GET /api/v1/reports/summary?period=2026-05

Response

{
  "period": "2026-05",
  "totalDriversEvaluated": 1000000,
  "totalPenaltiesApplied": 125000,
  "totalAmountCollected": 45000000,
  "topViolationReasons": [
    {
      "rule": "MIN_MONTHLY_RIDES",
      "count": 80000
    },
    {
      "rule": "LOW_RATING",
      "count": 45000
    }
  ]
}

Databases and Schema

```json
{
  "users": {
    "user_id": "BIGINT",
    "user_type": "ENUM(DRIVER, RIDER)",
    "name": "VARCHAR",
    "avg_rating": "DECIMAL(2,1)",
    "total_rides": "BIGINT",
    "wallet_balance": "DECIMAL",
    "status": "ENUM(ACTIVE, BLOCKED)",
    "created_at": "TIMESTAMP"
  },

  "penalty_rules": {
    "rule_id": "BIGINT",
    "rule_name": "VARCHAR",
    "entity_type": "ENUM(DRIVER, RIDER)",
    "metric": "VARCHAR",
    "operator": "ENUM(LESS_THAN, GREATER_THAN)",
    "threshold_value": "DECIMAL",
    "penalty_type": "ENUM(FIXED, PERCENTAGE)",
    "penalty_amount": "DECIMAL",
    "evaluation_frequency": "ENUM(DAILY, WEEKLY, MONTHLY)",
    "is_active": "BOOLEAN",
    "created_by": "BIGINT",
    "created_at": "TIMESTAMP"
  },

  "user_metrics": {
    "metric_id": "BIGINT",
    "user_id": "BIGINT",
    "month": "VARCHAR",
    "completed_rides": "INT",
    "cancelled_rides": "INT",
    "avg_rating": "DECIMAL(2,1)",
    "online_hours": "INT",
    "last_updated_at": "TIMESTAMP"
  },

  "penalties": {
    "penalty_id": "BIGINT",
    "user_id": "BIGINT",
    "rule_id": "BIGINT",
    "metric_value": "DECIMAL",
    "expected_value": "DECIMAL",
    "penalty_amount": "DECIMAL",
    "status": "ENUM(APPLIED, DEDUCTED, WAIVED)",
    "evaluation_period": "VARCHAR",
    "created_at": "TIMESTAMP"
  },

  "wallet_transactions": {
    "transaction_id": "BIGINT",
    "user_id": "BIGINT",
    "penalty_id": "BIGINT",
    "transaction_type": "ENUM(DEBIT, CREDIT)",
    "amount": "DECIMAL",
    "balance_after_txn": "DECIMAL",
    "transaction_status": "ENUM(SUCCESS, FAILED)",
    "created_at": "TIMESTAMP"
  },

  "disputes": {
    "dispute_id": "BIGINT",
    "penalty_id": "BIGINT",
    "user_id": "BIGINT",
    "reason": "TEXT",
    "attachment_url": "VARCHAR",
    "status": "ENUM(UNDER_REVIEW, APPROVED, REJECTED)",
    "resolved_by": "BIGINT",
    "resolution_notes": "TEXT",
    "created_at": "TIMESTAMP",
    "resolved_at": "TIMESTAMP"
  },

  "audit_logs": {
    "audit_id": "BIGINT",
    "entity_type": "VARCHAR",
    "entity_id": "BIGINT",
    "action": "VARCHAR",
    "performed_by": "BIGINT",
    "old_value": "JSON",
    "new_value": "JSON",
    "created_at": "TIMESTAMP"
  }
}
```

HLD

```text
========================================================
HIGH LEVEL DESIGN (HLD)
========================================================

                    +----------------------+
                    |      Admin Panel     |
                    +----------+-----------+
                               |
                               v
                    +----------------------+
                    |   Rule Management    |
                    |      Service         |
                    +----------+-----------+
                               |
                               v
                    +----------------------+
                    |   Penalty Rules DB   |
                    +----------------------+



+------------+       +----------------------+       +------------------+
| Ride       | ----> |  Event Ingestion     | ----> | Kafka/Event Bus  |
| Service    |       |  Service             |       +--------+---------+
+------------+       +----------------------+                |
                                                             |
+------------+                                               |
| Rating     | ----------------------------------------------+
| Service    |                                               |
+------------+                                               |
                                                             v

                                                +----------------------+
                                                | Metrics Aggregator   |
                                                | Service              |
                                                +----------+-----------+
                                                           |
                                                           v
                                                +----------------------+
                                                | User Metrics DB      |
                                                +----------+-----------+
                                                           |
                                                           v

                                                +----------------------+
                                                | Penalty Evaluation   |
                                                | Engine               |
                                                +----------+-----------+
                                                           |
                                      +--------------------+------------------+
                                      |                                       |
                                      v                                       v

                        +--------------------------+         +------------------------+
                        | Penalty Ledger Service   |         | Notification Service   |
                        +------------+-------------+         +------------+-----------+
                                     |                                     |
                                     v                                     v

                        +--------------------------+         +------------------------+
                        | Penalty DB               |         | Email/SMS/Push         |
                        +------------+-------------+         +------------------------+
                                     |
                                     v

                        +--------------------------+
                        | Wallet Service           |
                        +------------+-------------+
                                     |
                                     v

                        +--------------------------+
                        | Wallet Transactions DB   |
                        +--------------------------+



========================================================
FLOW DIAGRAM - MONTHLY PENALTY EVALUATION
========================================================

1. Ride Completed
        |
        v
2. Ride Service emits event to Kafka
        |
        v
3. Metrics Aggregator consumes events
        |
        v
4. Update monthly metrics in User Metrics DB
        |
        v
5. Monthly Scheduler triggers Evaluation Engine
        |
        v
6. Evaluation Engine fetches:
       - Active Rules
       - User Metrics
        |
        v
7. Rule Validation
       Example:
       completed_rides < 100
        |
        +---- NO ----> No action
        |
        +---- YES ---> Generate penalty
                             |
                             v
8. Store penalty in Penalty DB
                             |
                             v
9. Call Wallet Service
                             |
                             v
10. Deduct amount from wallet
                             |
                             v
11. Store wallet transaction
                             |
                             v
12. Send notification to user



========================================================
FLOW DIAGRAM - DISPUTE HANDLING
========================================================

1. User raises dispute
        |
        v
2. Dispute Service stores dispute
        |
        v
3. Admin reviews dispute
        |
        +---- APPROVED ----+
        |                  |
        |                  v
        |          Refund penalty
        |                  |
        |                  v
        |          Update wallet balance
        |
        +---- REJECTED ----+
                           |
                           v
                    Keep penalty active
                           |
                           v
                    Notify user



========================================================
CORE MICROSERVICES
========================================================

1. Rule Management Service
   - Create/update/delete rules
   - Rule versioning

2. Metrics Aggregator Service
   - Consume ride/rating events
   - Aggregate monthly metrics

3. Penalty Evaluation Engine
   - Execute rule evaluation jobs
   - Generate penalties

4. Penalty Ledger Service
   - Maintain immutable penalty history

5. Wallet Service
   - Deduct/refund money
   - Ensure transactional consistency

6. Notification Service
   - SMS/Push/Email alerts

7. Dispute Service
   - Dispute lifecycle management

8. Audit Service
   - Maintain compliance logs



========================================================
SCALABILITY CONSIDERATIONS
========================================================

1. Kafka partitions by user_id

2. Batch evaluation jobs partitioned by region/user buckets

3. Redis cache for active rules

4. Read replicas for analytics queries

5. Monthly partitioned tables for penalties and transactions

6. Idempotent wallet deduction APIs

7. Retry queues + DLQ for failed deductions
```

========================================================
MICROSERVICES AND INTERACTIONS
========================================================

1. API GATEWAY
--------------------------------------------------------
Responsibilities:
- Entry point for all clients
- Authentication & authorization
- Rate limiting
- Request routing

Interacts With:
- Rule Management Service
- Penalty Service
- Dispute Service
- Wallet Service
- Notification Service



2. RULE MANAGEMENT SERVICE
--------------------------------------------------------
Responsibilities:
- Create/update/delete penalty rules
- Enable/disable rules
- Rule versioning
- Rule validation

Database:
- penalty_rules DB

Publishes Events:
- RULE_CREATED
- RULE_UPDATED
- RULE_DELETED

Interacts With:
- Admin Panel
- Penalty Evaluation Engine
- Redis Cache



3. EVENT INGESTION SERVICE
--------------------------------------------------------
Responsibilities:
- Consume ride/rating/cancellation events
- Standardize incoming events
- Push events to Kafka

Consumes From:
- Ride Service
- Rating Service

Publishes To:
- Kafka Topics



4. METRICS AGGREGATOR SERVICE
--------------------------------------------------------
Responsibilities:
- Consume Kafka events
- Aggregate monthly metrics
- Compute:
   - completed rides
   - cancellation %
   - avg ratings
   - active hours

Database:
- user_metrics DB

Consumes:
- ride_completed events
- rating_updated events
- cancellation events

Publishes:
- METRICS_UPDATED



5. PENALTY EVALUATION ENGINE
--------------------------------------------------------
Responsibilities:
- Run periodic evaluation jobs
- Fetch active rules
- Evaluate user metrics
- Generate penalties

Consumes:
- METRICS_UPDATED
- Scheduler triggers

Reads From:
- penalty_rules DB
- user_metrics DB

Publishes:
- PENALTY_CREATED

Interacts With:
- Penalty Ledger Service
- Wallet Service
- Notification Service



6. PENALTY LEDGER SERVICE
--------------------------------------------------------
Responsibilities:
- Store immutable penalty history
- Maintain audit trail
- Penalty state management

Database:
- penalties DB

Consumes:
- PENALTY_CREATED

Publishes:
- PENALTY_APPLIED
- PENALTY_WAIVED

Interacts With:
- Wallet Service
- Dispute Service
- Audit Service



7. WALLET SERVICE
--------------------------------------------------------
Responsibilities:
- Deduct penalty amount
- Refund amount on dispute approval
- Maintain financial consistency
- Handle retries/idempotency

Database:
- wallet_transactions DB

Consumes:
- PENALTY_APPLIED
- REFUND_REQUESTED

Publishes:
- PAYMENT_SUCCESS
- PAYMENT_FAILED

Interacts With:
- Penalty Service
- Notification Service



8. NOTIFICATION SERVICE
--------------------------------------------------------
Responsibilities:
- Send push/email/SMS notifications
- Notify about:
   - penalties
   - refunds
   - disputes
   - warnings

Consumes:
- PENALTY_CREATED
- PAYMENT_SUCCESS
- DISPUTE_RESOLVED

External Integrations:
- Email provider
- SMS gateway
- Push notification provider



9. DISPUTE SERVICE
--------------------------------------------------------
Responsibilities:
- Create/manage disputes
- Upload proof documents
- Track dispute lifecycle

Database:
- disputes DB

Consumes:
- User dispute requests

Publishes:
- DISPUTE_CREATED
- DISPUTE_APPROVED
- DISPUTE_REJECTED

Interacts With:
- Penalty Ledger Service
- Wallet Service
- Notification Service



10. AUDIT SERVICE
--------------------------------------------------------
Responsibilities:
- Track all admin/system actions
- Maintain compliance logs
- Store immutable audit records

Database:
- audit_logs DB

Consumes:
- RULE_UPDATED
- PENALTY_APPLIED
- DISPUTE_RESOLVED
- WALLET_DEDUCTED



========================================================
SERVICE INTERACTION FLOW
========================================================

Ride Service
     |
     v
Event Ingestion Service
     |
     v
Kafka/Event Bus
     |
     v
Metrics Aggregator Service
     |
     v
User Metrics DB
     |
     v
Penalty Evaluation Engine
     |
     +--------------------+
     |                    |
     v                    v
Penalty Ledger      Notification Service
Service
     |
     v
Wallet Service
     |
     v
Wallet Transactions DB
     |
     v
Notification Service



========================================================
ASYNC COMMUNICATION
========================================================

Kafka Topics:
- ride_completed
- ride_cancelled
- rating_updated
- metrics_updated
- penalty_created
- penalty_applied
- payment_success
- dispute_created
- dispute_resolved



========================================================
WHY MICROSERVICE ARCHITECTURE?
========================================================

1. Independent scaling of evaluation engine

2. Wallet service isolated for strong consistency

3. Rule engine can evolve independently

4. Async event-driven processing reduces coupling

5. Easier fault isolation

6. Supports millions of monthly evaluations

📩 Contact: programmingappliedai@gmail.com

Example:

• Detect high CPU spikes on routers

• Predict link failures

• Detect packet drops or abnormal latency

• Restart interfaces automatically

• Shift traffic to healthy routes

2. Functional Requirements (FR)

1. The system should continuously collect router telemetry such as CPU utilization, memory usage, packet drops, interface statistics, link status, temperature, and logs/SNMP traps at configurable intervals.

2. The system should detect failures and anomalies like router downtime, link degradation, interface flapping, traffic anomalies, high latency, and packet loss within a few seconds.

3. The system should automatically execute preventive or corrective actions such as restarting interfaces, switching routes, triggering failover, throttling traffic, or rebooting unhealthy devices based on predefined policies.

4. The system should generate alerts and incidents through channels like email, Slack, PagerDuty, and SMS for critical network issues and remediation events.

5. The system should store historical telemetry and fault data to support trend analysis, root cause analysis, capacity planning, and ML/rule-based failure prediction.

3. Non-Functional Requirements (NFR)

NFR1. High Availability

Monitoring system should remain operational even during node failures.

Target: 99.99% availability.

NFR2. Low Detection Latency

Faults should be detected within:

• 2–5 seconds for critical failures

• <1 second for heartbeat failures

NFR3. Massive Scalability

Support:

• Millions of telemetry events/sec

• 100K+ routers

• Multi-region deployments

NFR4. Fault Tolerance

No data loss during:

• Broker failures

• Service crashes

• Network partitions

NFR5. Extensibility

System should support:

• New device types

• New telemetry protocols

• Custom remediation plugins

without major redesign.

4. High Level Architecture

                 +----------------+
                 |   Admin UI     |
                 +--------+-------+
                          |
                          v
                 +----------------+
                 | API Gateway    |
                 +--------+-------+
                          |
      ------------------------------------------------
      |              |               |               |
      v              v               v               v

+------------+ +-------------+ +-------------+ +--------------+
| Device Mgmt| | Alert Svc   | | Policy Svc  | | Analytics Svc|
+------------+ +-------------+ +-------------+ +--------------+

                          |
                          v

                +------------------+
                | Kafka / Pulsar   |
                +--------+---------+
                         |
     ---------------------------------------------------
     |                 |               |               |
     v                 v               v               v

+------------+ +--------------+ +-------------+ +--------------+
| Collectors | | Detection Svc| | ML Engine   | | Remediation  |
+------------+ +--------------+ +-------------+ +--------------+

                         |
                         v

                 +---------------+
                 | Routers       |
                 +---------------+

5. Core Components

A. Telemetry Collectors

Responsible for:

• SNMP polling
  

• Streaming telemetry
  

• Syslog ingestion
  

• NetFlow/sFlow collection
  

Protocols:

• SNMP
  

• gRPC telemetry
  

• NETCONF
  

• SSH
  

• Syslog
  

B. Event Streaming Layer

Use:

• Apache Kafka
  
  or
  

• Apache Pulsar
  

Purpose:

• Buffer telemetry
  

• Decouple services
  

• Enable replay
  

• Handle burst traffic
  

Topics:

telemetry.cpu
telemetry.memory
telemetry.interface
alerts
remediation.tasks

C. Detection Engine

Performs:

• Threshold detection
  

• Pattern detection
  

• Correlation
  

• Stateful analysis
  

Examples:

• CPU > 90% for 5 mins
  

• Packet loss > 20%
  

• Interface flapping > 5 times/min
  

Techniques:

• Rule engine
  

• CEP (Complex Event Processing)
  

D. ML Prediction Engine

Predicts:

• Link failure probability
  

• Hardware degradation
  

• Congestion patterns
  

Models:

• Time-series forecasting
  

• Isolation Forest
  

• LSTM
  

• ARIMA
  

E. Remediation Service

Executes automated actions:

• Route failover
  

• Interface reset
  

• Restart BGP sessions
  

• Traffic rerouting
  

Must support:

• Rollback
  

• Audit logs
  

• Approval workflows
  

F. Alerting Service

Integrations:

• PagerDuty
  

• Slack
  

• Email/SMS systems
  

Supports:

• Deduplication
  

• Escalation
  

• Suppression
  

6. APIs

A. Register Router API

Request

POST /api/v1/routers

{
  "routerId": "RTR-1001",
  "hostname": "blr-edge-1",
  "ip": "10.0.0.5",
  "location": "Bangalore-DC",
  "vendor": "Cisco",
  "model": "ASR-9000"
}

Response

{
  "status": "REGISTERED",
  "routerId": "RTR-1001"
}

B. Push Telemetry API

POST /api/v1/telemetry

Request

{
  "routerId": "RTR-1001",
  "timestamp": 1710001231,
  "metrics": {
    "cpu": 91,
    "memory": 72,
    "packetLoss": 15,
    "latency": 240
  }
}

Response

{
  "status": "INGESTED"
}

C. Create Remediation Policy

POST /api/v1/policies

Request

{
  "policyName": "HighCPURecovery",
  "condition": "cpu > 90 for 5m",
  "action": "restart_interface",
  "severity": "HIGH"
}

Response

{
  "policyId": "POL-101",
  "status": "ACTIVE"
}

D. Get Active Alerts

GET /api/v1/alerts?severity=CRITICAL

Response

[
  {
    "alertId": "ALT-1001",
    "routerId": "RTR-1001",
    "message": "Packet loss exceeded threshold",
    "severity": "CRITICAL",
    "timestamp": 1710009999
  }
]

E. Execute Manual Remediation

POST /api/v1/remediation/execute

Request

{
  "routerId": "RTR-1001",
  "action": "restart_bgp"
}

Response

{
  "executionId": "REM-1009",
  "status": "IN_PROGRESS"
}

7. Database Design

A. Router Metadata Table

CREATE TABLE routers (
    router_id VARCHAR(50) PRIMARY KEY,
    hostname VARCHAR(255),
    ip_address VARCHAR(50),
    vendor VARCHAR(50),
    model VARCHAR(50),
    location VARCHAR(255),
    status VARCHAR(20),
    created_at TIMESTAMP
);

B. Telemetry Table (Time-Series DB)

CREATE TABLE telemetry_metrics (
    router_id VARCHAR(50),
    metric_name VARCHAR(50),
    metric_value DOUBLE,
    ts TIMESTAMP,
    PRIMARY KEY(router_id, ts, metric_name)
);

Recommended DB:

• InfluxDB
  

• OpenTSDB
  

C. Alert Table

CREATE TABLE alerts (
    alert_id VARCHAR(50) PRIMARY KEY,
    router_id VARCHAR(50),
    severity VARCHAR(20),
    message TEXT,
    status VARCHAR(20),
    created_at TIMESTAMP
);

D. Policy Table

CREATE TABLE remediation_policies (
    policy_id VARCHAR(50) PRIMARY KEY,
    policy_name VARCHAR(255),
    condition_expression TEXT,
    action VARCHAR(100),
    severity VARCHAR(20),
    enabled BOOLEAN
);

E. Remediation Execution Table

CREATE TABLE remediation_execution (
    execution_id VARCHAR(50) PRIMARY KEY,
    router_id VARCHAR(50),
    action VARCHAR(100),
    status VARCHAR(50),
    started_at TIMESTAMP,
    completed_at TIMESTAMP,
    rollback_status VARCHAR(50)
);

8. Storage Choices

| Data Type         | Database             |
| ----------------- | -------------------- |
| Router Metadata   | PostgreSQL           |
| Telemetry Metrics | InfluxDB / Cassandra |
| Logs              | Elasticsearch        |
| Alerts            | PostgreSQL           |
| Event Streaming   | Kafka                |
| ML Features       | S3/HDFS              |

9. Microservices

| Service                     | Responsibility                |
| --------------------------- | ----------------------------- |
| API Gateway                 | Authentication, routing       |
| Device Service              | Router metadata management    |
| Telemetry Ingestion Service | Accept telemetry              |
| Detection Engine            | Fault detection               |
| Policy Engine               | Evaluate remediation policies |
| Remediation Service         | Execute preventive actions    |
| Alert Service               | Notifications                 |
| Analytics Service           | Dashboards and trends         |
| ML Prediction Service       | Predict future failures       |

10. Microservices Interaction Flow

Scenario:

CPU spike detected on router.

1. Router sends telemetry
2. Collector publishes to Kafka
3. Detection Engine consumes telemetry
4. Rule matched: CPU > 90%
5. Alert generated
6. Policy Engine checks remediation rules
7. Remediation Service restarts interface
8. Result published back to Kafka
9. Alert Service sends PagerDuty alert
10. Analytics Service updates dashboard

11. Sequence Flow Diagram

Router
   |
   | Telemetry
   v
Collector Service
   |
   | Publish Event
   v
Kafka
   |
   | Consume
   v
Detection Engine
   |
   | Fault Detected
   v
Policy Engine
   |
   | Execute Action
   v
Remediation Service
   |
   | SSH/NETCONF
   v
Router

Detection Engine
   |
   | Create Alert
   v
Alert Service
   |
   | Notify
   v
PagerDuty / Slack / Email

12. Scaling Strategy

Horizontal Scaling

• Multiple collectors
  

• Partitioned Kafka topics
  

• Stateless detection workers
  

Kafka Partitioning

Partition by:

hash(routerId)

Ensures:

• Ordering per router
  

• Balanced processing
  

Telemetry Optimization

Use:

• Compression
  

• Batch ingestion
  

• Sampling
  

• Aggregation windows
  

13. Reliability & Fault Tolerance

Retry Mechanisms

• Exponential backoff
  

• Dead-letter queues
  

Exactly Once Processing

Use:

• Kafka transactions
  

• Idempotent consumers
  

Multi-Region Deployment

Deploy:

• Active-active clusters
  

• Geo-replication
  

14. Security Considerations

• mTLS between routers and collectors
  

• RBAC for remediation APIs
  

• Encrypted telemetry
  

• Audit logs for all actions
  

• Secret management using:
  

  • HashiCorp Vault
    

15. Bottlenecks & Optimizations

| Bottleneck            | Optimization        |
| --------------------- | ------------------- |
| High telemetry volume | Kafka batching      |
| Detection lag         | Stream processing   |
| Alert storms          | Alert deduplication |
| Expensive queries     | Pre-aggregation     |
| Large storage         | Tiered retention    |

16. Interview Deep Dive Questions

1. How will you detect interface flapping efficiently?
   

2. Why choose Kafka over RabbitMQ?
   

3. How would you avoid alert storms?
   

4. How will rollback work during failed remediation?
   

5. How do you guarantee ordering per router?
   

6. How would ML-based prediction integrate?
   

7. How would the system behave during Kafka outages?
   

8. How do you prevent false positives?
   

9. How would you support millions of routers?
   

10. How would you design multi-region failover?
    

Consider this code:

class Counter {
    int count = 0;

    void increment() {
        count++;
    }
}

public class Test {
    public static void main(String[] args) throws Exception {
        Counter counter = new Counter();

        Thread t1 = new Thread(() -> {
            for (int i = 0; i < 100000; i++) {
                counter.increment();
            }
        });

        Thread t2 = new Thread(() -> {
            for (int i = 0; i < 100000; i++) {
                counter.increment();
            }
        });

        t1.start();
        t2.start();

        t1.join();
        t2.join();

        System.out.println(counter.count);
    }
}

At first glance, the answer should obviously be:

200000

Two threads.

Each increments 100000 times.

So why does the output often become:

173421
189302
196774

and almost never exactly 200000?

Let’s break it down.

The Real Problem: count++ Is NOT Atomic

Most developers assume:

count++;

is a single operation.

It is not.

Under the hood, it becomes three separate CPU operations:

1. Read count from memory
2. Increment locally
3. Write updated value back

Equivalent pseudo-operations:

temp = count;
temp = temp + 1;
count = temp;

Now imagine two threads executing simultaneously.

The Race Condition

Suppose:

count = 5

Now both threads execute count++.

Thread Interleaving

StepThread 1Thread 2count1Read 552Read 553Increment to 654Increment to 655Write 666Write 66

Expected result:

7

Actual result:

6

One increment is lost.

This is called a:

⚠️ Lost Update Problem

And because this happens thousands of times across both threads, the final result becomes unpredictable.

Why Does This Happen?

Because:

• Threads share the same heap memory
  

• CPU scheduling is nondeterministic
  

• Context switching can occur anytime
  

• count++ is not thread-safe
  

This entire category of bugs falls under:

⚠️ Race Conditions

A race condition occurs when:

Multiple threads access and modify shared state concurrently without proper synchronization.

How to Fix It

There are multiple ways.

Each has different performance and scalability tradeoffs.

1. Using synchronized

class Counter {
    int count = 0;

    synchronized void increment() {
        count++;
    }
}

Now only one thread can execute increment() at a time.

How It Works

Java uses an intrinsic monitor lock.

Before entering:

increment()

thread acquires the monitor.

Other threads wait.

Pros

✔ Easy to understand

✔ Guarantees correctness

✔ Built into JVM

Cons

❌ Blocking

❌ Context switching overhead

❌ Poor scalability under heavy contention

2. Using ReentrantLock

import java.util.concurrent.locks.ReentrantLock;

class Counter {
    int count = 0;

    ReentrantLock lock = new ReentrantLock();

    void increment() {
        lock.lock();

        try {
            count++;
        } finally {
            lock.unlock();
        }
    }
}

Why Use It?

More flexible than synchronized.

Supports:

• Fair locking
  

• Try lock
  

• Interruptible lock acquisition
  

• Timed waits
  

Pros

✔ More control

✔ Better advanced concurrency handling

Cons

❌ Manual unlock required

❌ Still blocking

3. Using AtomicInteger (Best for Counters)

import java.util.concurrent.atomic.AtomicInteger;

class Counter {
    AtomicInteger count = new AtomicInteger();

    void increment() {
        count.incrementAndGet();
    }
}

Why This Is Better

AtomicInteger uses:

⚡ Compare-And-Swap (CAS)

instead of locking.

What CAS Does

CPU instruction:

if current_value == expected_value
    update
else
    retry

No thread blocking.

No monitor locking.

Extremely efficient.

Internally

while(true) {
   int existing = value;
   int next = existing + 1;

   if(CAS(existing, next))
       break;
}

Why Modern Systems Prefer Atomic Operations

High-performance systems like:

• Kafka
  

• Netty
  

• Aerospike
  

• Cassandra
  

• Redis internals
  

heavily rely on:

• Lock-free algorithms
  

• CAS operations
  

• Atomic primitives
  

because locks become bottlenecks at scale.

Performance Comparison

ApproachThread SafeBlockingScalablePlain int❌❌❌synchronized✅✅MediumReentrantLock✅✅MediumAtomicInteger✅❌High

But AtomicInteger Is Not Always Enough

For extremely high contention systems:

100+ threads
millions of increments/sec

even CAS retries become expensive.

That’s why Java introduced:

⚡ LongAdder

Used internally in:

• ConcurrentHashMap
  

• Metrics systems
  

• High throughput counters
  

It reduces contention using striped counters.

Real-World Backend Engineering Insight

This tiny example explains why distributed systems are hard.

Now imagine:

• Millions of concurrent users
  

• Multiple JVMs
  

• Distributed databases
  

• Replication
  

• Network retries
  

The exact same “lost update” problem appears everywhere:

• Inventory systems
  

• Banking transactions
  

• Payment systems
  

• Distributed counters
  

• Leaderboards
  

Concurrency bugs scale from a single integer → to entire distributed architectures.

Final Takeaway

The biggest lesson is:

Concurrency problems are rarely visible in code syntax.

The bug is hidden inside:

count++;

What looks like one operation is actually multiple CPU-level operations racing against each other.

Understanding this deeply is foundational for:

• Multithreading
  

• JVM internals
  

• High-performance backend engineering
  

• Distributed systems design
  

Because at scale:

Correctness is harder than performance.

#Java #Concurrency #Multithreading #BackendEngineering #DistributedSystems #JVM #SystemDesign

📩 Contact: programmingappliedai@gmail.com

1. Problem Framing

Design a Podcast Tracking Service where users can:

• Follow podcasts

• Track listening progress

• Get updates on new episodes

• Resume playback across devices

✅ 2. Functional Requirements (FR)

1. User can search & discover podcasts

2. User can follow / unfollow podcasts

3. User can view episodes of a podcast

4. User can mark episode as played / in-progress

5. Track listening progress (timestamp-based)

6. Resume playback from last position

7. Notify users on new episode release

8. Show user feed (new episodes from followed podcasts)

9. Support multi-device sync

10. Maintain listening history

⚙️ 3. Non-Functional Requirements (NFR)

1. High write throughput (progress updates are frequent)

2. Low latency reads (resume playback must be instant)

3. High availability (user data must not be lost)

4. Scalability (millions of users, podcasts)

5. Eventual consistency acceptable (for feeds/notifications)

6. Durability of progress tracking

7. Idempotent APIs (progress updates retry-safe)

🧱 4. Core Entities

Key Entities:

• User

• Podcast

• Episode

• UserSubscription

• ListeningProgress

• ListeningHistory

🗄️ 5. Database Design

🔹 1. User

user_id (PK)
name
email
created_at

🔹 2. Podcast

podcast_id (PK)
title
description
publisher
category
created_at

🔹 3. Episode

episode_id (PK)
podcast_id (FK)
title
duration
publish_time
audio_url

🔹 4. UserSubscription

user_id (PK, FK)
podcast_id (PK, FK)
subscribed_at

👉 Composite PK for fast lookup

🔹 5. ListeningProgress (HOT PATH)

user_id (PK)
episode_id (PK)
progress_seconds
last_updated
status (IN_PROGRESS / COMPLETED)

👉 Stored in NoSQL (Cassandra/DynamoDB)

Reason: high write frequency

🔹 6. ListeningHistory

user_id
episode_id
completed_at

👉 Append-only → good for analytics

🗃️ 6. Database Choices

| Use Case               | DB                            |
| ---------------------- | ----------------------------- |
| User, Podcast metadata | MySQL / PostgreSQL            |
| Episodes               | MySQL (or Elastic for search) |
| Listening Progress     | Cassandra / DynamoDB          |
| Feed / Notifications   | Redis + Kafka                 |
| Search                 | Elasticsearch                 |

🔌 7. API Design

🔹 1. Follow Podcast

POST /users/{userId}/follow

Request:
{
  “podcastId”: “p123”
}

🔹 2. Get User Subscriptions

GET /users/{userId}/subscriptions

🔹 3. Get Episodes of Podcast

GET /podcasts/{podcastId}/episodes

🔹 4. Update Listening Progress (CRITICAL API)

POST /progress

Request:
{
  “userId”: “u1”,
  “episodeId”: “e1”,
  “progressSeconds”: 120
}

👉 Must be:

• Idempotent
  

• High throughput
  

🔹 5. Get Resume Playback

GET /progress?userId=u1&episodeId=e1

🔹 6. Get User Feed

GET /users/{userId}/feed

🧩 8. High-Level Architecture

                ┌──────────────┐
                │ API Gateway  │
                └──────┬───────┘
                       │
        ┌──────────────┼──────────────┐
        │              │              │
 User Service   Podcast Service   Progress Service
        │              │              │
        │              │              │
   MySQL DB       MySQL DB       Cassandra
                                      │
                                  Kafka
                                      │
                              Notification Service
                                      │
                                   Redis

🔄 9. Microservices Breakdown

🔹 1. User Service

• Manage users
  

• Auth (optional)
  

🔹 2. Podcast Service

• Podcast metadata
  

• Episode management
  

🔹 3. Subscription Service

• Follow/unfollow
  

• Stores mapping
  

🔹 4. Progress Service (HOT SERVICE)

• Handles frequent writes
  

• Writes to Cassandra
  

• Publishes events to Kafka
  

🔹 5. Feed Service

• Aggregates new episodes
  

• Pull vs Push model
  

🔹 6. Notification Service

• Consumes Kafka events
  

• Sends push/email notifications
  

🔁 10. Flow: Update Listening Progress

Client → API Gateway → Progress Service
        → Write to Cassandra
        → Publish event to Kafka
                ↓
        Notification / Analytics consumers

🔁 11. Flow: New Episode Release

Podcast Service
    → Publish “New Episode Event” to Kafka
            ↓
    Feed Service
        → Fan-out to followers (push model)
            ↓
    Notification Service

⚖️ 12. Key Design Decisions

🔹 Why Cassandra for Progress?

• High write throughput
  

• Partition key = user_id → efficient
  

🔹 Push vs Pull Feed

ApproachTradeoffPullSimple but slowPushFast reads, heavy writes

👉 Use hybrid model

🔹 Idempotency for Progress API

• Use (user_id, episode_id) as key
  

• Always overwrite latest progress
  

🔗 13. Decoupling Strategy (Important for Interview)

1. Use Kafka for async communication
   

2. Services don’t call each other directly
   

3. Event-driven architecture:
   

   • Progress updates → analytics
     

   • New episodes → feed generation
     

🚀 14. Scaling Considerations

• Partition Cassandra by user_id
  

• Cache feed in Redis
  

• Use CDN for audio files
  

• Batch updates for analytics
  

🧠 15. Follow-Up Questions (They Might Ask)

❓ How to handle rewind/skip edge cases?

→ Always store latest timestamp

❓ What if user listens offline?

→ Sync on reconnect (client-side buffering)

❓ How to avoid write overload?

→ Throttle updates (every 5–10 sec instead of every second)

❓ Multi-device consistency?

→ Last write wins (timestamp-based conflict resolution)

• Deep dive on Kafka topic design & partitioning
  

• Cassandra schema optimization (LSM trees, compaction)
  

• Or Feed system design (fan-out on write vs read)

  
  

1. Kafka Design (Topics, Partitions, Throughput)

You’re using Kafka as the backbone for decoupling, so design it intentionally—not vaguely.

🧵 Key Events

1. Progress Updates

topic: listening-progress-events

Event:

{
  userId,
  episodeId,
  progressSeconds,
  timestamp
}

2. New Episode Release

topic: new-episode-events

Event:

{
  podcastId,
  episodeId,
  publishTime
}

⚙️ Partitioning Strategy

🔹 Progress Events

👉 Partition by userId

Why?

• Guarantees ordering per user
  

• Avoids race conditions in progress updates
  

🔹 New Episode Events

👉 Partition by podcastId

Why?

• All followers of a podcast handled consistently
  

• Enables batching for fan-out
  

📊 Throughput Estimation (Back-of-envelope)

Let’s assume:

• 10M daily active users
  

• Each sends progress every 10 sec
  

👉 Writes/sec:

10M * (1 update / 10 sec) = 1M events/sec

👉 You must:

• Use multiple partitions (1000+)
  

• Use batching producers
  

• Enable compression
  

⚠️ Common Mistake

Don’t do:

• One partition → bottleneck
  

• Random partition → breaks ordering
  

🧱 2. Cassandra Schema Deep Dive

This is where most answers stay shallow—you shouldn’t.

🔹 Table: Listening Progress

PRIMARY KEY ((user_id), episode_id)

👉 Partition key: user_id

👉 Clustering key: episode_id

❗ Problem

What if a user listens to 1000+ episodes?

→ Partition becomes large

✅ Better Design

Option 1: Bucketing

PRIMARY KEY ((user_id, bucket), episode_id)

Where:

bucket = hash(episode_id) % N

👉 Prevents hot partitions

Option 2: Time-based Bucketing (better for history)

PRIMARY KEY ((user_id, month), episode_id)

🔹 Write Pattern

• Always upsert
  

• Last write wins
  

🔹 Read Pattern

• Fetch by (user_id, episode_id) → fast O(1)
  

⚠️ Compaction Strategy

Use:

• Leveled Compaction → for read-heavy
  

• Size-Tiered → for write-heavy (better here)
  

⚡ 3. Feed System Design (THE differentiator)

This is where interviewers test depth.

🧩 Problem

User follows 100 podcasts.

Each podcast releases episodes.

👉 How do you build the user feed?

🆚 Approach 1: Fan-out on Read

Flow:

User opens app →
Fetch all followed podcasts →
Fetch latest episodes →
Merge + sort

❌ Problem:

• Slow
  

• High read amplification
  

🆚 Approach 2: Fan-out on Write

Flow:

New Episode →
Push to all followers’ feeds

❌ Problem:

• If podcast has 10M followers → explosion
  

✅ Hybrid Approach (Best Answer)

Strategy:

• Small podcasts → fan-out on write
  

• Large podcasts → fan-out on read
  

🔹 Feed Table (Redis / Cassandra)

user_id → list of episode_ids (sorted by time)

🔄 Flow: New Episode

Podcast Service
    → Kafka (new-episode-events)
        ↓
Feed Service
    → Fetch followers
    → If small → push to feed
    → If large → mark for pull

⚡ Optimization

• Use Redis Sorted Sets
  

• Score = timestamp
  

🔁 4. End-to-End Flow (Putting It All Together)

🎧 Flow: User Listening

Client
 → API Gateway
 → Progress Service
 → Cassandra (write)
 → Kafka (event)

Kafka Consumers:
 → Analytics Service
 → Recommendation Service

🆕 Flow: New Episode

Podcast Service
 → Kafka

Feed Service
 → Fan-out (hybrid)

Notification Service
 → Push alerts

🔗 5. Decoupling Strategy (Strong Signal)

❌ Bad Design

Progress Service → directly calls Feed Service

👉 Tight coupling

✅ Good Design

Progress Service → Kafka → Consumers

👉 Loose coupling

👉 Scalable

👉 Replayable events

🧠 6. Advanced Follow-ups (Be Ready)

❓ How do you handle duplicate Kafka events?

👉 Use:

• Idempotent consumers
  

• Dedup key = (userId, episodeId, timestamp)
  

❓ How do you handle out-of-order updates?

👉 Use:

only update if incoming.timestamp > existing.timestamp

❓ How do you reduce progress write load?

👉 Client optimization:

• Send updates every 10 sec (not every second)
  

• Send only if progress changed significantly
  

❓ How to scale Kafka consumers?

👉 Increase:

• Consumer groups
  

• Partitions
  

🧠 Final Interview Tip

When you explain this, don’t just describe components—justify every decision:

• “I chose Cassandra because write-heavy workload”
  

• “Kafka ensures decoupling and replayability”
  

• “Hybrid fan-out avoids celebrity problem”
  

📩 Contact: programmingappliedai@gmail.com

Requirements

• Read last N lines

• Should work for large files (GBs) → cannot load entire file

• Efficient (O(file_size) worst case, but optimized by reverse reading)

• Handle edge cases:

  • File smaller than N lines

  • Very long lines

  • Different line endings

🏗️ LLD Design

1. Core Components

TailService

• Main service to fetch last N lines

FileReaderStrategy

• Strategy pattern for reading:

  • ForwardReader (simple, small files)

  • ReverseReader (optimized for large files)

LineBuffer

• Maintains last N lines using deque

2. Approach Options

❌ Naive Approach

• Read entire file → keep last N lines

• Not scalable

✅ Optimal Approach (Reverse Read)

• Use RandomAccessFile

• Start from end of file

• Move backwards byte by byte

• Count \n

• Stop after N lines

⚙️ Java Implementation (Efficient)

import java.io.*;
import java.util.*;

public class TailService {

    public List<String> tail(String filePath, int n) throws IOException {
        if (n <= 0) return Collections.emptyList();

        RandomAccessFile file = new RandomAccessFile(filePath, “r”);
        long fileLength = file.length();

        List<String> result = new LinkedList<>();
        StringBuilder lineBuilder = new StringBuilder();

        int lineCount = 0;
        long pointer = fileLength - 1;

        while (pointer >= 0 && lineCount < n) {
            file.seek(pointer);
            char c = (char) file.read();

            if (c == ‘\n’) {
                if (lineBuilder.length() > 0) {
                    result.add(0, lineBuilder.reverse().toString());
                    lineBuilder.setLength(0);
                    lineCount++;
                }
            } else {
                lineBuilder.append(c);
            }

            pointer--;
        }

        // Add last line if file doesn’t end with newline
        if (lineBuilder.length() > 0 && lineCount < n) {
            result.add(0, lineBuilder.reverse().toString());
        }

        file.close();
        return result;
    }

    public static void main(String[] args) throws IOException {
        TailService service = new TailService();
        List<String> lines = service.tail(”test.txt”, 10);

        for (String line : lines) {
            System.out.println(line);
        }
    }
}

🔍 Complexity

| Aspect | Complexity              |
| ------ | ----------------------- |
| Time   | O(file_size) worst case |
| Space  | O(N * avg_line_length)  |

⚡ Improvements (Production Level)

1. Use Buffered Block Reads

Instead of reading char-by-char:

• Read chunks (e.g., 4KB)
  

• Traverse buffer backwards
  

2. Handle Encoding

• Use CharsetDecoder for UTF-8 safety
  

3. Support Streams (like tail -f)

• Add file watcher (WatchService)
  

• Continuously read appended lines
  

4. Memory Optimization

• Use ArrayDeque instead of LinkedList
  

🧱 Cleaner LLD Version

interface TailReader {
    List<String> readLastNLines(String filePath, int n) throws IOException;
}

Reverse Reader

class ReverseTailReader implements TailReader {
    @Override
    public List<String> readLastNLines(String filePath, int n) throws IOException {
        return new TailService().tail(filePath, n);
    }
}

Service Layer

class TailApplication {
    private final TailReader reader;

    public TailApplication(TailReader reader) {
        this.reader = reader;
    }

    public void execute(String filePath, int n) throws IOException {
        List<String> lines = reader.readLastNLines(filePath, n);
        lines.forEach(System.out::println);
    }
}

💡 Follow-ups (Interview Ready)

You can extend this design into:

1. tail -f (Real-time streaming)

• Use WatchService
  

• Poll file growth
  

• Maintain pointer
  

2. Distributed Logs

• Read from S3 / HDFS
  

• Use range queries
  

3. API Service

GET /tail?file=abc.log&lines=50

4. Large-scale Observability System

• Combine with Kafka + log ingestion
  

• Real-time tail via WebSockets

  
  

Unix Functionality Github Link

Design a Temporary Email Service that allows users to:

• Generate disposable email addresses

• Receive emails in a sandboxed inbox

• Auto-expire emails and addresses after a TTL

• Avoid spam/abuse

• Scale to millions of emails per day

✅ Functional Requirements (FR)

1. Users can generate temporary email addresses

2. System should receive emails sent to these addresses

3. Users can view emails via UI/API

4. Emails and addresses should expire automatically (TTL-based)

5. Support multiple emails per temp address

6. Provide meaningful / human-readable email IDs (not random hashes)

⚙️ Non-Functional Requirements (NFR)

1. High throughput (millions of incoming emails/day)

2. Low latency for inbox retrieval (<200ms)

3. Highly scalable (horizontal scaling)

4. Fault tolerant (no email loss)

5. Secure sandboxing (no malware execution)

6. Abuse prevention (rate limiting, spam filtering)

🧩 Core Design Challenges

1. Email Address Generation (Important 🔥)

Cannot just be hashes → must be meaningful

Approach:

Use structured format:

<word>-<word>-<random>@tempmail.com

Examples:

• blue-tiger-92@tempmail.com
  

• swift-lake-431@tempmail.com
  

Why this works:

• Human-readable
  

• Easier debugging
  

• Still unique with suffix
  

Backend Strategy:

• Dictionary-based generator + counter/random suffix
  

• Maintain uniqueness via Redis / DB constraint
  

🏗️ High-Level Architecture

                +-------------------+
                |   DNS (MX Record) |
                +--------+----------+
                         |
                         v
                +-------------------+
                | Email Receiving   |
                | Server (SMTP)     |
                +--------+----------+
                         |
                         v
                +-------------------+
                | Email Parser      |
                +--------+----------+
                         |
          +--------------+--------------+
          |                             |
          v                             v
+-------------------+        +----------------------+
| Spam Filter       |        | Attachment Scanner   |
+--------+----------+        +----------+-----------+
         |                              |
         +--------------+---------------+
                        v
                +-------------------+
                | Message Queue     | (Kafka)
                +--------+----------+
                         |
                         v
                +-------------------+
                | Email Storage     |
                +--------+----------+
                         |
         +---------------+---------------+
         |                               |
         v                               v
+-------------------+        +-------------------+
| Metadata DB       |        | Blob Storage      |
| (Cassandra)       |        | (S3 / GCS)        |
+-------------------+        +-------------------+

                +-------------------+
                | Inbox API Service |
                +--------+----------+
                         |
                         v
                +-------------------+
                | Frontend UI       |
                +-------------------+

🔄 Key Flow

1. Email Creation

• User requests → Service generates email ID
  

• Store in DB with TTL
  

2. Email Receiving Flow

1. SMTP server receives email
   

2. Parses:
   

   • To address
     

   • Subject
     

   • Body
     

3. Push to Kafka
   

4. Consumers:
   

   • Store metadata → Cassandra
     

   • Store attachments → Blob store
     

3. Inbox Fetch

• Query by email address
  

• Fetch recent messages (sorted by timestamp)
  

🧱 Data Modeling

Email Address Table

email_id (PK)
created_at
expires_at
status (active/expired)

Email Metadata Table (Cassandra)

partition_key: email_id
clustering_key: timestamp

Fields:
- message_id
- subject
- sender
- preview
- blob_url
- created_at

⏳ TTL Strategy

• Use DB TTL (Cassandra TTL)
  

• Also maintain background cleanup job
  

🚀 Scaling Deep Dive

1. Email Ingestion Scaling

• Multiple SMTP servers behind load balancer
  

• Partition by domain or hash(email)
  

2. Kafka Partitioning

• Partition key = email_id
  

• Ensures ordering per inbox
  

3. Storage Scaling

• Cassandra → write-heavy optimized
  

• Blob store → scalable for attachments
  

4. Read Scaling

• Cache inbox in Redis
  

• Recent emails only
  

🛡️ Security & Sandboxing

Must handle:

• Malicious links
  

• Attachments
  

• Scripts
  

Approach:

• Strip scripts from HTML
  

• Convert to safe format
  

• Virus scan attachments
  

• Disable external image loading
  

🚫 Abuse Prevention

• Rate limit email creation per IP
  

• Blocklist domains
  

• CAPTCHA
  

• Detect bot patterns
  

💡 Out-of-the-Box Ideas (Strong Interview Signal)

1. Auto Forwarding Rules

• Forward temp email → real inbox
  

2. Disposable API Keys

• Use temp emails in automated testing
  

3. Session-based Emails

• Email tied to browser session
  

• No login needed
  

4. Smart Expiry

• Extend TTL if actively used
  

5. Email Categorization

• ML model for:
  

  • Spam
    

  • OTP
    

  • Promotions
    

6. Domain Pooling

• Multiple domains:

@maildrop.cc
@tempmail.dev

• Avoid blacklisting
  

🔥 Bottlenecks & Tradeoffs

ProblemSolutionHot email IDsShard by hashLarge attachmentsOffload to blobHigh read trafficRedis cacheSpam overloadPre-filter before Kafka

🧪 APIs

Create Email

POST /email
Response:
{
  “email”: “blue-tiger-92@tempmail.com”,
  “expires_at”: “...”
}

Get Inbox

GET /emails/{email_id}

Get Email Detail

GET /emails/{email_id}/{message_id}

🧠 Interview Tips

• Start with email infra (SMTP) → shows depth
  

• Emphasize Kafka for decoupling
  

• Push on:
  

  • TTL strategy
    

  • Abuse prevention
    

  • Scaling ingestion vs reads
    

📩 Contact: programmingappliedai@gmail.com

FR

• Support bulk migration of historical data from RDBMS to NoSQL

• Capture and replicate real-time data changes (CDC) from source to target

• Transform relational schema into NoSQL-compatible data models

• Enable dual writes to both databases during migration phase

• Ensure seamless incremental sync between bulk load and live updates

• Provide data validation mechanisms (counts, checksums, record-level diff)

• Support controlled traffic cutover from old DB to new datastore

• Allow rollback to source database in case of failures

• Handle write conflicts and ensure consistency during dual-write phase

• Provide monitoring, alerting, and operational controls (pause/resume/retry)

NFR

• High Availability – Migration should not cause downtime; ticket booking and browsing must remain continuously available

• Strong Consistency (Critical Flows) – Ensure strict consistency for operations like seat booking to prevent double bookings

• Scalability – System should handle large-scale data (TB–PB) and high throughput during peak traffic events

• Fault Tolerance – System should recover from failures using retries, idempotent operations, and checkpointing

• Low Latency Impact – Migration (especially dual writes and CDC) should not significantly increase application latency

• Observability – Provide detailed metrics, logging, and alerting for migration progress, lag, errors, and system health

API’s

1. Start Bulk Migration

POST /migration/start

Request:

{
  “migrationId”: “mig_123”,
  “sourceDB”: “mysql_cluster_A”,
  “targetDB”: “cassandra_cluster_B”,
  “tables”: [”events”, “tickets”, “bookings”],
  “batchSize”: 10000
}

Response:

{
  “status”: “STARTED”,
  “migrationId”: “mig_123”
}

2. Get Migration Status

GET /migration/{migrationId}/status

Response:

{
  “migrationId”: “mig_123”,
  “status”: “IN_PROGRESS”,
  “progress”: “65%”,
  “lagInMs”: 1200,
  “errors”: 10
}

3. Pause Migration

POST /migration/{migrationId}/pause

Response:

{
  “status”: “PAUSED”
}

4. Resume Migration

POST /migration/{migrationId}/resume

Response:

{
  “status”: “RESUMED”
}

5. Start CDC (Real-Time Sync)

POST /migration/{migrationId}/cdc/start

Request:

{
  “source”: “binlog”,
  “offset”: “log_456”
}

Response:

{
  “status”: “CDC_STARTED”
}

6. Get CDC Lag

GET /migration/{migrationId}/cdc/lag

Response:

{
  “lagInMs”: 800,
  “lastSyncedTimestamp”: “2026-04-10T10:00:00Z”
}

7. Validate Data

POST /migration/{migrationId}/validate

Request:

{
  “type”: “checksum”,
  “tables”: [”bookings”]
}

Response:

{
  “status”: “VALIDATION_STARTED”
}

8. Get Validation Report

GET /migration/{migrationId}/validation/report

Response:

{
  “status”: “COMPLETED”,
  “mismatchedRecords”: 25,
  “details”: “checksum mismatch in bookings”
}

9. Trigger Cutover

POST /migration/{migrationId}/cutover

Request:

{
  “mode”: “phased”,
  “trafficPercentage”: 50
}

Response:

{
  “status”: “CUTOVER_IN_PROGRESS”
}

10. Rollback Migration

POST /migration/{migrationId}/rollback

Response:

{
  “status”: “ROLLED_BACK”
}

11. Dual Write Toggle

POST /migration/{migrationId}/dual-write

Request:

{
  “enabled”: true
}

Response:

{
  “status”: “DUAL_WRITE_ENABLED”
}

Databases and Schema

Databases Used in Migration Platform

1. Source Database (RDBMS)

• Example: MySQL / PostgreSQL

• Purpose: Existing transactional system (Ticket booking, inventory)

2. Target Database (NoSQL)

• Example: Cassandra / DynamoDB

• Purpose: Scalable, high-throughput datastore post-migration

3. Migration Metadata DB

• Example: PostgreSQL / MySQL

• Purpose: Track migration jobs, states, checkpoints

4. Validation / Audit Store

• Example: S3 / Blob store / Cassandra

• Purpose: Store validation reports, mismatches

5. Cache (Optional)

• Example: Redis

• Purpose: Speed up reads during dual-read phase

Schema Design

1. Source DB Schema (Relational - Simplified)

Events Table

Events(
  event_id BIGINT PRIMARY KEY,
  name VARCHAR,
  venue VARCHAR,
  date TIMESTAMP
)

Tickets Table

Tickets(
  ticket_id BIGINT PRIMARY KEY,
  event_id BIGINT,
  seat_number VARCHAR,
  price DECIMAL,
  status VARCHAR,  -- AVAILABLE / BOOKED
  FOREIGN KEY (event_id) REFERENCES Events(event_id)
)

Bookings Table

Bookings(
  booking_id BIGINT PRIMARY KEY,
  user_id BIGINT,
  ticket_id BIGINT,
  status VARCHAR, -- CONFIRMED / CANCELLED
  created_at TIMESTAMP
)

👉 Highly normalized → joins required

2. Target DB Schema (NoSQL - Cassandra Style)

Design Principle

• Query-driven
  

• Denormalized
  

• Avoid joins
  

Table: Event_Tickets (Primary Read Path)

Event_Tickets(
  event_id,
  seat_number,
  ticket_id,
  price,
  status,
  user_id,
  booking_status,
  PRIMARY KEY (event_id, seat_number)
)

👉 Fetch all seats for an event quickly

👉 Handles seat availability + booking in one place

Table: User_Bookings

User_Bookings(
  user_id,
  booking_id,
  event_id,
  ticket_id,
  status,
  created_at,
  PRIMARY KEY (user_id, booking_id)
)

👉 Query bookings per user

Table: Event_Metadata

Event_Metadata(
  event_id PRIMARY KEY,
  name,
  venue,
  date
)

3. Migration Metadata DB Schema

Migration Jobs

Migration_Jobs(
  migration_id VARCHAR PRIMARY KEY,
  status VARCHAR, -- STARTED / IN_PROGRESS / COMPLETED
  start_time TIMESTAMP,
  end_time TIMESTAMP
)

Table Progress Tracking

Migration_Progress(
  migration_id,
  table_name,
  last_processed_pk BIGINT,
  status VARCHAR,
  PRIMARY KEY (migration_id, table_name)
)

👉 Used for checkpointing + resume

CDC Offsets

CDC_Offsets(
  migration_id,
  source,
  offset VARCHAR,
  updated_at TIMESTAMP,
  PRIMARY KEY (migration_id, source)
)

👉 Tracks binlog/Kafka offset

4. Validation / Audit Schema

Validation Results

Validation_Results(
  migration_id,
  table_name,
  total_records_source BIGINT,
  total_records_target BIGINT,
  mismatch_count BIGINT,
  status VARCHAR,
  PRIMARY KEY (migration_id, table_name)
)

Mismatch Records (Optional)

Mismatch_Records(
  migration_id,
  table_name,
  record_id,
  source_value TEXT,
  target_value TEXT
)

Key Design Insights (Interview Gold)

• Denormalization in NoSQL → avoids joins, improves latency
  

• Multiple tables per query pattern → common in Cassandra
  

• Checkpointing via Migration_Progress → ensures fault tolerance
  

• CDC offsets → critical for exactly-once / no data loss
  

• Event_Tickets table → solves seat consistency + fast lookup
  

Microservices in DB Migration Platform

1. Migration Orchestrator Service

Responsibility:

• Central controller of migration lifecycle

• Starts/stops jobs, manages phases (bulk → CDC → cutover)

Key tasks:

• Trigger bulk migration

• Start/stop CDC

• Initiate cutover / rollback

• Maintain state machine

2. Bulk Migration Service

Responsibility:

• Migrate historical data from RDBMS → NoSQL

Key tasks:

• Read data in batches

• Transform schema (normalize → denormalize)

• Write to NoSQL

• Update checkpoints

3. CDC (Change Data Capture) Service

Responsibility:

• Capture real-time changes from source DB

Key tasks:

• Read binlogs (Debezium/Kafka)

• Convert events → NoSQL format

• Push to stream (Kafka)

4. Stream Processing / Replication Service

Responsibility:

• Apply CDC events to NoSQL

Key tasks:

• Consume Kafka topics

• Ensure ordering (per key)

• Idempotent writes

• Conflict resolution

5. Schema Transformation Service

Responsibility:

• Convert relational schema → NoSQL model

Key tasks:

• Flatten joins

• Build denormalized records

• Maintain mapping rules

6. Dual Write Proxy Service

Responsibility:

• Enable application to write to both DBs

Key tasks:

• Intercept write requests

• Write to RDBMS + NoSQL

• Handle partial failures (retry / compensate)

7. Validation Service

Responsibility:

• Ensure data correctness

Key tasks:

• Row count validation

• Checksums

• Record-level diff

• Generate reports

8. Cutover Service

Responsibility:

• Switch traffic from RDBMS → NoSQL

Key tasks:

• Gradual traffic shift (0% → 100%)

• Enable shadow reads

• Monitor errors

9. Rollback Service

Responsibility:

• Revert to old system if needed

Key tasks:

• Redirect traffic back

• Disable dual writes

• Ensure consistency

10. Monitoring & Observability Service

Responsibility:

• System visibility

Key tasks:

• Track lag (CDC)

• Migration progress

• Error rates

• Alerts

Interaction Flow (End-to-End)

Phase 1: Bulk Migration

1. Orchestrator → triggers Bulk Migration Service

2. Bulk Service:

   • Reads from RDBMS

   • Calls Schema Transformer

   • Writes to NoSQL

3. Updates checkpoints in Metadata DB

Phase 2: Real-Time Sync (CDC)

1. CDC Service:

   • Reads binlogs → pushes to Kafka

2. Stream Processor:

   • Consumes events

   • Transforms + writes to NoSQL

3. Offsets stored for recovery

Phase 3: Dual Writes

1. Application → Dual Write Proxy

2. Proxy:

   • Writes to RDBMS

   • Writes to NoSQL

3. Handles failures with retries / idempotency

Phase 4: Validation

1. Validation Service:

   • Compares source vs target

   • Generates reports

2. Orchestrator decides readiness

Phase 5: Cutover

1. Cutover Service:

   • Gradually shifts reads to NoSQL

   • Then writes fully to NoSQL

2. Shadow reads verify correctness

Phase 6: Rollback (if needed)

1. Rollback Service:

   • Redirects traffic back to RDBMS

   • Stops CDC / dual writes

High-Level Interaction Diagram (Textual)

Client
  ↓
Dual Write Proxy
  ↓              ↘
RDBMS         NoSQL

RDBMS → CDC Service → Kafka → Stream Processor → NoSQL

Bulk Migration Service → NoSQL

Validation Service ↔ RDBMS & NoSQL

Orchestrator → Controls all services

Key Design Insights (What interviewer looks for)

• Decoupling via Kafka → avoids tight coupling between CDC and writes
  

• Idempotent consumers → ensures no duplicate writes
  

• Partitioning by event_id / ticket_id → preserves ordering
  

• Dual write + CDC overlap → ensures zero data loss
  

• Shadow reads before cutover → reduces risk
  

If you want to go next level (staff-level discussion), I can explain:

• Exactly-once semantics in CDC pipeline
  

• Handling seat booking consistency in Cassandra (LWT vs Redis locks)
  

• Or failure scenarios (Kafka lag, partial writes, replay strategies) 🚀
  

flow diagram

Here’s a clean, interview-ready flow diagram you can draw on a whiteboard 👇

DB Migration Platform – Flow Diagram

                  +----------------------+
                  |   Migration          |
                  |   Orchestrator       |
                  +----------+-----------+
                             |
        -----------------------------------------------
        |                     |                      |
        ↓                     ↓                      ↓

+----------------+   +----------------+    +----------------+
| Bulk Migration |   |   CDC Service  |    |  Validation    |
|   Service      |   | (Binlog Reader)|    |   Service      |
+--------+-------+   +--------+-------+    +--------+-------+
         |                    |                     |
         ↓                    ↓                     |
   +------------+     +---------------+            |
   |  RDBMS     |---->|    Kafka      |------------+
   +------------+     +-------+-------+
                               |
                               ↓
                      +----------------------+
                      | Stream Processor     |
                      | (Replication Layer)  |
                      +----------+-----------+
                                 |
                                 ↓
                         +--------------+
                         |   NoSQL DB   |
                         +--------------+

------------------------------------------------------------

                 (Application Traffic Flow)

                     +------------------+
Client  -----------> | Dual Write Proxy |
                     +--------+---------+
                              |
                  -------------------------
                  |                       |
                  ↓                       ↓
             +---------+           +------------+
             |  RDBMS  |           |  NoSQL DB  |
             +---------+           +------------+

------------------------------------------------------------

                 (Cutover Phase)

Client  -------------------------------------> NoSQL DB
                     (RDBMS removed gradually)

How to Explain This in Interview (Quick Script)

• Top layer → Orchestrator controls everything
  

• Left path → Bulk migration moves historical data
  

• Middle path → CDC + Kafka handles real-time updates
  

• Right path → Validation ensures correctness
  

• Bottom → Dual writes keep both DBs in sync
  

• Final step → Cutover shifts traffic fully to NoSQL
  

Questions:
Write a query to fetch all order IDs with their latest state in the last 1 hour
Exclude orders that are already delivered
How would this query behave at scale?
What indexes would you add?

This is a classic event-sourced state transition problem — very relevant for high-scale systems like food delivery or ride-sharing.

Let’s go step by step.

1. Query: Latest state in last 1 hour (excluding delivered)

Assume table:

order_state_transitions (
    order_id BIGINT,
    state VARCHAR,
    timestamp TIMESTAMP
)

✅ SQL Query

SELECT order_id, state
FROM (
    SELECT 
        order_id,
        state,
        timestamp,
        ROW_NUMBER() OVER (PARTITION BY order_id ORDER BY timestamp DESC) as rn
    FROM order_state_transitions
    WHERE timestamp >= NOW() - INTERVAL ‘1 hour’
) t
WHERE rn = 1
AND state != ‘Delivered’;

🔍 Explanation

• WHERE timestamp >= NOW() - 1 hour
  
  → limits dataset early (critical for performance)
  

• ROW_NUMBER() OVER (PARTITION BY order_id ORDER BY timestamp DESC)
  
  → finds latest state per order
  

• rn = 1
  
  → keeps only latest record
  

• state != 'Delivered'
  
  → filters completed orders
  

2. Behavior at Scale 🚀

At scale (millions of events/min):

❌ Problems

1. Window function is expensive
   

   • Needs sorting per order_id
     

   • Memory + CPU heavy
     

2. Large scan even for 1 hour
   

   • If system handles high throughput → still huge data
     

3. Hot partitions
   

   • Recent data is heavily accessed
     

4. Repeated computation
   

   • Every query recomputes latest state
     

⚠️ Real-world impact

• High latency (100ms → seconds)
  

• Increased DB CPU
  

• Not suitable for real-time dashboards or APIs
  

3. Optimizations (What you’d do in production)

✅ 1. Maintain a current state table (denormalized)

orders_current_state (
    order_id BIGINT PRIMARY KEY,
    current_state VARCHAR,
    last_updated TIMESTAMP
)

Update this on every state change.

👉 Query becomes:

SELECT order_id, current_state
FROM orders_current_state
WHERE last_updated >= NOW() - INTERVAL ‘1 hour’
AND current_state != ‘Delivered’;

✔ O(1) lookup

✔ No window function

✔ Much faster

✅ 2. Stream processing (Kafka-style)

Instead of querying raw DB:

• Consume events
  

• Maintain latest state in:
  

  • Redis / Cassandra / materialized view
    

✅ 3. Partitioning

Partition table by time:

PARTITION BY RANGE (timestamp)

→ Only scan last 1-hour partition

4. Indexing Strategy 📊

If you must run the original query, indexes are critical.

✅ Primary Index

CREATE INDEX idx_order_time 
ON order_state_transitions (order_id, timestamp DESC);

✔ Helps window function ordering

✔ Faster partitioning by order_id

✅ Time-based Index

CREATE INDEX idx_timestamp 
ON order_state_transitions (timestamp);

✔ Helps filter last 1 hour quickly

✅ Covering Index (best case)

CREATE INDEX idx_covering 
ON order_state_transitions (timestamp, order_id, state);

✔ Avoids table lookup (index-only scan)

🚀 Ideal combo

• (timestamp) → prune data
  

• (order_id, timestamp DESC) → latest per order
  

5. Final System Design Insight 🧠

In real systems (Uber/Zomato style):

👉 You never compute latest state from raw events on demand

Instead:

• Write → append-only event log
  

• Maintain → materialized latest state
  

• Query → read from optimized store
  

TL;DR

• Use window function for correctness
  

• Avoid it in production at scale
  

• Maintain current state table
  

• Index on (timestamp) and (order_id, timestamp DESC)
  

• Consider streaming + caching layer
  

1. Kafka + Consumer Architecture 🚀

🧩 Core Idea

Treat order state changes as an event stream, not just DB writes.

🔄 Flow

1. Producer (Order Service)

Every state change emits an event:

{
  “order_id”: 123,
  “state”: “Reached Customer”,
  “timestamp”: “2026-03-21T10:00:00Z”
}

2. Kafka Topic

topic: order_state_transitions
partition key: order_id

👉 Why order_id as key?

• Ensures ordering per order
  

• All events for an order go to same partition
  

3. Consumers

✅ Consumer 1: Current State Updater

• Reads events
  

• Updates materialized view
  

orders_current_state
(order_id, current_state, last_updated)

✅ Consumer 2: Analytics / Monitoring

• SLA tracking
  

• Delay detection (e.g., stuck at “Reached Merchant”)
  

✅ Consumer 3: Notification Service

• Trigger push notifications:
  

  • “Order Out for Delivery”
    

  • “Delivered”
    

⚙️ Processing Logic

for each event:
    if event.timestamp > stored_timestamp:
        update current_state

👉 Protects against out-of-order events

⚠️ Challenges

ProblemSolutionOut-of-order eventsUse timestamp/version checkDuplicate eventsIdempotent updatesConsumer lagScale partitionsReprocessingKafka replay capability

✅ Why this architecture?

• No heavy DB queries
  

• Real-time updates
  

• Scales horizontally
  

• Enables multiple downstream use cases
  

2. Redis vs Cassandra Tradeoffs ⚖️

This is where interviews get interesting.

🟥 Redis (In-memory, ultra-fast)

✅ Pros

• Sub-millisecond reads
  

• Perfect for latest state lookup
  

• Supports TTL (auto-expire old orders)
  

• Simple key-value model
  

key: order:123 → “Reached Customer”

❌ Cons

• Memory expensive
  

• Limited durability (unless tuned)
  

• Not ideal for long-term storage
  

🟦 Cassandra (Distributed, write-heavy DB)

✅ Pros

• Handles massive write throughput
  

• Persistent storage
  

• Highly scalable
  

• Good for time-series/event data
  

PRIMARY KEY (order_id, timestamp)

❌ Cons

• Higher read latency than Redis
  

• More complex data modeling
  

• Eventually consistent
  

🧠 When to use what?

Use CaseBest ChoiceReal-time API (get current state)✅ RedisHistorical queries / audit✅ CassandraHigh write ingestion✅ CassandraUltra-low latency reads✅ Redis

💡 Best Practice (Hybrid Approach)

👉 Use both:

• Kafka → Consumers →
  

  • Redis → latest state (fast reads)
    

  • Cassandra → full history (durability)
    

🔥 Architecture Pattern

Kafka → Consumer →
    ├── Redis (current state)
    └── Cassandra (event log)

3. Full HLD Interview Answer 🏗️

Here’s how you’d present this in an interview.

🧾 Problem Statement

Design a system to:

• Track order state transitions
  

• Fetch latest state for active orders
  

• Handle high throughput (millions of events)
  

✅ Functional Requirements

1. Track order lifecycle states
   

2. Fetch latest state of an order
   

3. Fetch active orders (not delivered)
   

4. Support real-time updates
   

5. Maintain historical transitions
   

⚙️ Non-Functional Requirements

1. Low latency (<50ms reads)
   

2. High throughput (writes/sec in millions)
   

3. High availability
   

4. Fault tolerance
   

5. Scalability