It was around 2 AM when my phone rang.

I was half asleep, so for a few seconds I just looked at the screen, hoping whoever was calling would call back in the morning.

Then I saw the team name. I picked it up.

“Production mein issue hai. Payment requests fail ho rahi hain.”

Bas. Neend gayi.

I opened my laptop and joined the call. Everyone was already trying to figure things out. Someone was checking logs, someone was looking at the database, and someone was asking about the latest deployment. And honestly, my first thought wasn’t some amazing debugging strategy.

It was simply, “Yaar, ab kya hua?”

That’s the thing about production issues. When something breaks on your local machine, you have time. You can add logs, put breakpoints, restart the application, change things and try again.

Production is different.

There are real users on the other side. Someone is trying to make a payment. Someone is waiting for an order. Someone is staring at an error message. And while you’re trying to understand the problem, people are waiting for an answer.

That’s where I learned one of the most important things about production debugging:

Don’t panic. And don’t start fixing things before you understand what is actually broken.

My First Instinct Used to Be Wrong

Earlier, whenever I saw a production issue, I would immediately open the code.

API failing? Open the controller.

Then the service.

Then the repository.

Then the database query.

Ten minutes later, I would have twenty tabs open and still wouldn’t know what was happening.

The problem wasn’t that I couldn’t debug.

I was just starting from the wrong place.

Instead of asking “Where is the bug?”, I should have been asking “What exactly is happening?”

So that night, I started with the simplest thing: the numbers.

How many requests were failing?

When did they start failing?

Were all payments failing?

Or only some?

That immediately gave us an interesting clue.

Not every payment was failing.

Some were going through perfectly.

Now the problem wasn’t “payment is down.”

Something was failing under a particular condition.

And suddenly, the problem became much smaller.

Don’t Trust the First Explanation

During production incidents, someone will almost always say something like:

“Database down hai.”

“API broken hai.”

“Payment service is down.”

I’ve learned not to take these statements as facts.

Usually, they’re just someone’s observation.

If a payment fails, it doesn’t necessarily mean the payment service is broken. Maybe the database is fine. Maybe the API is fine. Maybe a third-party service is returning something unexpected.

So now, whenever I hear “X is broken”, my first reaction is:

“Okay, let’s verify that.”

That one habit can save a lot of time.

Start With the Timeline

The next thing I usually check is the timeline.

When did the problem actually start? And more importantly, what changed just before it started?

Was there a deployment? A configuration change? A database migration? A traffic spike? A change in another service?

Production systems have a lot of moving parts, and sometimes your code hasn’t changed at all.

Maybe another service was deployed.

Maybe traffic suddenly increased.

Maybe a scheduled job started behaving differently.

Maybe an external API started responding slowly.

That night, we noticed that the failures had started shortly after a deployment. Interesting. But I didn’t want to immediately say, “Deployment caused it.”

It was a clue, not a conclusion.

The Production Debugging Flow

This is roughly how I approach a production issue now. I start broad, collect evidence, and slowly narrow the problem down until I reach the actual failure point.

Follow One Failing Request

At this point, instead of looking at thousands of logs, I picked one failing request. I wanted to follow its entire journey.

The request reached our API. The business logic ran. The database responded normally. Then our service called another service. And that’s where things started looking strange.

This is where I usually step back and look at the bigger picture. A production issue rarely gives you the answer from one place.

I look at the timeline, logs, metrics and request trace together, then compare the evidence before deciding what the actual root cause is.

The downstream service wasn’t completely down. It was responding, but in one particular scenario, the response wasn’t what our application expected.

That changed everything.

We weren’t dealing with a completely broken application.

We had a specific failure happening between two services.

This is something production teaches you pretty quickly: your application doesn’t live alone. It depends on databases, queues, other services, external APIs and infrastructure.

Sometimes your code is perfectly fine.

Something around it isn’t.

Find One That Works

One of the simplest things I do during debugging is compare a failing request with a successful request.

Same API?

Same type of user?

Same data?

Same region?

Same request?

Same downstream response?

Usually, there is a difference somewhere.

Maybe one particular data combination is causing the issue.

Maybe only older records fail.

Maybe only a particular user flow triggers it.

The moment you find that difference, the problem becomes much easier.

Instead of saying:

“The payment system is broken.”

You can say:

“Payments fail when this specific condition occurs.”

That’s a much better problem to have.

Logs Should Tell You a Story

I used to think more logs meant easier debugging.

Not really.

I’ve seen systems where a single request generates hundreds of log lines. Everything is logged, but when production breaks, finding the useful information becomes harder.

What you actually want is a story.

Which request was this?

What operation was happening?

Which service handled it?

Which dependency was called?

What happened after that?

This is where correlation IDs are extremely useful.

If a request travels through multiple services, you should be able to follow that request across the system.

At 2 AM, you don’t want more information.

You want the right information.

Don’t Change Everything at Once

This is probably the easiest mistake to make when everyone is waiting for you.

You increase a timeout.

Restart the service.

Change some configuration.

Modify the query.

Deploy again.

And suddenly everything works.

Great.

But what actually fixed it?

You don’t know.

Maybe the restart helped. Maybe the configuration change helped. Maybe the external service recovered at the same time.

That’s why I try to make one meaningful change at a time.

Change something.

Observe.

Measure.

Then move forward.

Otherwise, you might fix the problem without understanding it.

And the next time it happens, you’re back at the beginning.

Sometimes Rollback Is the Right Fix

There is sometimes a strange ego around deployments.

Engineers don’t want to rollback because it feels like admitting that the deployment was wrong.

But production doesn’t care about your ego.

If a recent deployment is strongly connected to the incident and rollback is safe, rollback.

The goal during an incident is not to prove that your code is perfect.

The goal is to get users back to a stable system.

You can investigate the real root cause afterward.

Production first. Ego later.

The Hardest Part Is Staying Calm

Honestly, the technical problem isn’t always the hardest part.

The pressure is.

Someone asks, “Any update?”

You say you’re checking.

Five minutes later:

“Any update?”

Another team is waiting. Someone wants to know the impact. Meanwhile, you’re staring at the logs thinking, “I still don’t know what’s happening.”

I’ve been there.

And I’ve learned that not knowing the answer doesn’t mean you need to start moving faster.

Sometimes you need to stop for thirty seconds.

Ask yourself:

What do I know?

What don’t I know?

What have I already ruled out?

What’s the next thing I can check?

That’s enough.

You don’t need to solve the whole incident in your head.

You just need to find the next useful clue.

The Problem Usually Becomes Smaller

At 2 AM, the problem looked huge.

“Payment is failing in production.”

But after checking the metrics, looking at the timeline, following one request and comparing it with a successful request, the problem became much smaller.

It wasn’t the entire payment system.

It wasn’t the database.

It wasn’t even the whole API.

It was a specific interaction between services under a specific condition.

That’s how I think about debugging now.

You don’t magically find the answer.

You keep removing possibilities.

One by one.

Until the giant production problem becomes one service, one request, one dependency and one bug.

And that’s when debugging becomes much easier.

Fixing the Bug Isn’t the End

Once everything starts working again, it’s tempting to close the laptop and forget the whole thing.

But that’s actually when the important questions start.

Why did this happen?

Why didn’t we catch it earlier?

Why didn’t our monitoring detect it?

Could we have handled the failure better?

Do we need better validation?

Better logging?

A timeout?

A circuit breaker?

A test?

The goal isn’t just to fix today’s incident.

It’s to make sure the same problem doesn’t wake you up at 2 AM again.

What I Finally Learned

I used to think a good engineer was someone who could look at an error and immediately know the answer.

I don’t think that anymore.

The engineers I respect most aren’t the ones who know everything.

They’re the ones who stay calm when they don’t know.

They don’t guess.

They don’t randomly change five things.

They don’t assume the first error message is the root cause.

They simply keep asking better questions.

Did the request reach the service?

Did the database respond?

Did the downstream service respond?

When did the problem start?

What changed?

Why did this request fail while another one worked?

Slowly, the huge problem becomes smaller.

And eventually, you find the answer.

So the next time your phone rings at 2 AM and someone says, “Production mein issue hai,” don’t immediately start changing things.

Take a breath.

Understand what is actually happening.

Follow one failing request.

Find one that worked.

Compare them.

Reduce the possibilities.

Then fix the thing that is actually broken.

Because production debugging isn’t about being the smartest person in the room.

It’s about staying calm enough to figure out what the system is trying to tell you.

And honestly, that’s one of the most valuable engineering skills I’ve learned.

Before You Go

One more thing.

If you’re serious about becoming a better software engineer, I’m building something that might be useful for you.

Instead of putting everything into one huge learning platform, I’ve created focused learning paths for different stages of an engineer’s journey.

Master AI Engineering
Learn how to build AI-powered applications, understand AI systems, work with LLMs, agents, RAG, and take AI from an idea to something that actually works.

Master AI Engineering: https://algorithmxlr8.io/ai/engineering/home

Master DevOps Engineering
Understand the engineering behind deployments, containers, CI/CD, cloud infrastructure, monitoring, scalability, and running applications reliably in production.

Master DevOps Engineering: https://algorithmxlr8.io/devops/engineering/home

Master Software Engineer Interviews
DSA, LLD, HLD, System Design and the kind of problems you’re actually expected to solve in software engineering interviews.

Master Software Engineer Interviews:

https://algorithmxlr8.io/

The goal isn’t just to help you clear an interview.

It’s to help you become the engineer who actually understands what’s happening behind the code, how systems work, and how to build things properly.

If that sounds like the kind of engineer you want to become, come join me on the journey.

Explore:

https://algorithmxlr8.io/

I still remember the first time I opened a genuinely large codebase.

I had worked on plenty of projects before. I knew Java. I knew Spring Boot. I knew APIs, databases, queues, Docker, AWS, all the usual things.

So when someone said, “You’ll need to understand this repository before making the change,” I thought:

Okay. Give me a couple of hours.

Then I opened the repository.

There were hundreds of folders.

controller

service

repository

config

client

consumer

producer

handler

factory

strategy

processor

scheduler

utils

And then there were folders whose names made absolutely no sense.

I opened one class.

It called another class.

That class called another service.

That service called a client.

The client published an event.

The event was consumed somewhere else.

That consumer triggered another service.

That service updated the database.

And somewhere in the middle, there was a Redis cache.

I closed the file.

Opened another one.

Same story.

After an hour, I had read a lot of code.

But I understood nothing.

And that’s when I learned one of the most important lessons of working with large software systems:

You don’t understand a large codebase by reading more code. You understand it by building the right mental model.

That completely changed how I approach unfamiliar repositories.

The Biggest Mistake: Starting With the Code

When developers join a new project, the first instinct is usually:

“Let me start reading the code.”

That’s exactly what I used to do.

Open UserController.

Read 200 lines.

Open UserService.

Read 400 lines.

Open UserRepository.

Read 100 lines.

Then jump to PaymentService.

Then OrderService.

Then some random utility.

After three hours, you feel productive because you have seen thousands of lines.

But if someone asks:

“So how does an order actually get created?”

You have no idea.

That’s the trap.

A large codebase is not a book.

You don’t read it from page 1 to the last page.

A large codebase is more like a city.

If you enter a new city, you don’t walk into the first building and start studying its walls.

First, you look at the map.

You figure out:

Where is the airport?

Where is downtown?

Where are the major roads?

Where are the important neighborhoods?

Only then do you start exploring individual buildings.

A codebase works exactly the same way.

Step 1: Don’t Read the Code. Find the Business.

Before looking at classes, ask:

What does this system actually do?

Imagine you join an e-commerce company.

You are given a repository called:

commerce-platform

You could start with:

src/
 ├── controller/
 ├── service/
 ├── repository/
 ├── config/
 ├── consumer/
 ├── producer/
 └── client/

But instead, ask a completely different question:

What happens when a customer places an order?

Now you have a story.

Customer opens the application.

They add products to the cart.

They click Buy Now.

The backend receives the request.

An order is created.

Payment is initiated.

Inventory is reserved.

The customer receives confirmation.

That’s already more useful than reading 500 classes.

Because now you’re not trying to understand the entire system.

You’re trying to understand one journey through the system.

And that journey becomes your entry point.

Step 2: Find the Entry Point

Every important operation has somewhere it enters the system.

For an HTTP application, it might be:

POST /orders

Search for it.

Maybe you find:

@PostMapping(”/orders”)
public OrderResponse createOrder(
        @RequestBody CreateOrderRequest request) {
return orderService.createOrder(request);
}

Perfect.

You now have your first door into the system.

Don’t immediately understand every line.

Just follow the path.

Controller
    ↓
Service
    ↓
Repository
    ↓
Database

Maybe the real system is more complicated:

Controller
    ↓
OrderService
    ↓
InventoryService
    ↓
PaymentService
    ↓
EventPublisher
    ↓
Kafka
    ↓
OrderConsumer
    ↓
NotificationService

Now something interesting happens.

You are no longer looking at classes.

You’re looking at a story.

And stories are much easier for humans to remember.

Step 3: Trace One Request End-to-End

This is probably the most powerful technique I’ve learned.

Pick one important operation.

For example:

“Create Order.”

Then trace it from beginning to end.

Write it down.

Literally.

POST /order
↓
OrderController
        ↓
OrderService.createOrder()
        ↓
InventoryService.reserve()
        ↓
PaymentService.charge()
        ↓
OrderRepository.save()
        ↓
Kafka → OrderCreatedEvent
        ↓
NotificationConsumer
        ↓
EmailService

Suddenly the giant repository doesn’t look so scary anymore.

You have discovered one path through the forest.

Now do the same for:

Cancel Order

DELETE /orders/{id}
↓
OrderController
        ↓
OrderService.cancel()
        ↓
PaymentService.refund()
        ↓
InventoryService.release()
        ↓
OrderRepository.update()
        ↓
OrderCancelledEvent

Now you understand two important business flows.

And something magical starts happening.

The same classes appear again.

You begin recognizing them.

“Oh, I’ve already seen OrderService.”

“I’ve already seen PaymentService.”

“So this event is also consumed by NotificationService.”

That’s how a large codebase slowly becomes familiar.

Step 4: Build Your Own Map

One mistake I see engineers make is trying to remember everything in their head.

Don’t.

Create a map.

It can be something as simple as:

                     ┌──────────────┐
                     │   Client     │
                     └──────┬───────┘
                            ↓
                     ┌──────────────┐
                     │ Controller   │
                     └──────┬───────┘
                            ↓
                     ┌──────────────┐
                     │   Service    │
                     └───┬──────┬───┘
                         ↓      ↓
                  ┌─────────┐ ┌─────────┐
                  │  DB     │ │ Payment │
                  └─────────┘ └────┬────┘
                                   ↓
                                Kafka
                                   ↓
                              Consumers

It doesn’t need to be beautiful.

It needs to be useful.

Personally, I like writing three things for every unfamiliar system:

1. What comes in?

HTTP
Kafka
Scheduler
External API

2. What happens in the middle?

Validation
Business logic
Database
Cache
External services
Events

3. What goes out?

HTTP response
Kafka event
Email
Database update
External API call

Once you can answer those three questions, the system becomes much easier to navigate.

Step 5: Understand the Data Before the Classes

Here’s another mistake I made early in my career.

I spent hours understanding services.

But I didn’t understand the data.

That’s backwards.

Suppose you’re working on a payment system.

Before understanding PaymentService.java, find out:

What is a Payment?

Maybe the database looks something like:

Payment
id
orderId
userId
amount
currency
status
transactionId
createdAt
updatedAt

Now the code starts making sense.

You see:

payment.setStatus(PaymentStatus.SUCCESS);

You know why it exists.

You see:

paymentRepository.findByOrderId(orderId);

You understand what relationship is being used.

You see:

transactionId

And you immediately think:

“This probably connects our payment record with the external payment provider.”

Without understanding the data, code is just syntax.

With the data, code starts telling a story.

Step 6: Find the Database Tables

Whenever I enter a new backend project, one of the first things I want to understand is:

What are the important entities?

For example:

User
Order
Product
Payment
Inventory
Shipment

Then ask:

User
  ↓
Order
  ↓
Payment
  ↓
Shipment

Maybe:

Order
 ├── userId
 ├── paymentId
 └── shipmentId

Now you have another mental model.

You understand not only what the code does, but also what the system is storing.

And this becomes extremely useful when debugging.

Because many production bugs eventually become:

“Something went wrong between these two pieces of data.”

Step 7: Don’t Ignore Configuration

This one looks boring.

It isn’t.

I’ve seen engineers spend hours trying to understand application behavior while completely ignoring:

application.yml
application.properties
.env
Dockerfile
docker-compose.yml

Configuration often tells you more about a system than the Java classes.

Look for:

Database
Redis
Kafka
S3
AWS
External APIs
Feature flags
Timeouts
Retries
Thread pools
Connection pools

Suppose you find:

payment:
  timeout: 3000
kafka:
  retries: 3
redis:
  ttl: 3600

Now you already know something about the architecture.

The system talks to a payment provider.

It uses Kafka.

It uses Redis.

The payment call has a timeout.

Kafka retries failures.

Redis data expires after an hour.

You just learned a lot without reading hundreds of classes.

Step 8: Find the External Systems

Large systems almost never live alone.

They talk to other systems.

Your codebase might interact with:

Payment Gateway
Kafka
Redis
PostgreSQL
MongoDB
S3
Elasticsearch
Notification Service
Identity Service

Find those integrations.

Search for things like:

RestTemplate
WebClient
FeignClient
KafkaTemplate
RedisTemplate
S3Client
MongoRepository
JpaRepository

Then ask:

Why does this system depend on that system?

That’s important.

Don’t just memorize:

“We use Kafka.”

Understand:

“We use Kafka because order processing doesn’t need to wait for notification processing.”

Now you’ve understood the reason, not just the technology.

And reasons are easier to remember.

Step 9: Use Git Like a Detective

This is one of the most underrated techniques.

When you don’t understand why some weird code exists, don’t stare at it for two hours.

Check Git history.

For example:

git log

Or:

git blame PaymentService.java

You might discover:

Added retry logic after payment provider timeout issue

Suddenly that strange-looking code makes sense.

Maybe you find:

if (attempt > 2) {
    throw new PaymentException();
}

At first you think:

“Why only two retries?”

Git history might tell you:

“Because the payment provider starts rate-limiting requests after three attempts.”

Boom.

The code wasn’t random.

There was a production incident behind it.

This is why mature codebases sometimes look messy.

You’re not just reading software.

You’re reading the history of decisions.

Step 10: Read Tests Before Implementation

This changed my life.

Suppose you find:

OrderService.createOrder()

You could open the implementation.

Or you could first search:

OrderServiceTest

Tests often tell you what the business actually expects.

For example:

shouldNotCreateOrderWhenInventoryUnavailable()
shouldRefundPaymentWhenOrderCancellationFails()
shouldRetryPaymentWhenProviderTimesOut()
shouldNotDuplicateOrderForSameIdempotencyKey()

Look at those test names.

You just discovered important business rules.

And you didn’t need to read 500 lines of implementation.

Tests are often the closest thing to executable documentation.

Step 11: Learn the “Happy Path” First

Don’t start with exceptions.

Don’t start with every edge case.

Don’t start with the most complicated class.

First understand:

What happens when everything goes right?

For an order:

Request
 ↓
Validate
 ↓
Create Order
 ↓
Reserve Inventory
 ↓
Charge Payment
 ↓
Save Order
 ↓
Publish Event
 ↓
Send Confirmation

That’s your happy path.

Once you understand that, start asking:

What happens if payment fails?

Payment fails
      ↓
Retry
      ↓
Still fails
      ↓
Release inventory
      ↓
Mark order FAILED
      ↓
Return error

Then:

What happens if Kafka is down?

Then:

What happens if the request is sent twice?

Then:

What happens if the database succeeds but event publishing fails?

This is how you move from basic understanding to real engineering understanding.

Step 12: Look for the Weird Parts

Once you’ve understood the normal flow, start looking for things that make you say:

“Why the hell are they doing this?”

Those places are gold.

Maybe you find:

Thread.sleep(500);

Why?

Maybe you find:

cache.put(key, value, 300);

Why 300?

Maybe you find:

if (retryCount < 3)

Why three?

Maybe you find:

@Transactional

Why is the transaction boundary here?

Maybe you find:

eventPublisher.publish(...)

inside a database transaction.

Now you have interesting questions.

And these questions often lead you toward the most important architectural decisions in the system.

Step 13: Don’t Try to Understand Every Class

This is probably the most important thing in this entire article.

You don’t need to understand 100% of the codebase.

You need to understand the 20% that matters for your task.

Imagine you receive this ticket:

“Fix duplicate payments when the client retries the request.”

You don’t need to understand:

UserProfileService
RecommendationEngine
SearchService
NotificationService
AnalyticsService
ReportingService

You need to find:

Payment API
      ↓
Payment Service
      ↓
Payment Provider
      ↓
Database

Then ask:

Where is the request ID?

Where is idempotency handled?

Where is the payment persisted?

What happens when the same request comes twice?

Your understanding should be task-driven.

Not repository-driven.

A Real Example

Let’s say you join a company and receive this bug:

“Customers are sometimes charged twice when they click Pay multiple times.”

You open the repository.

You don’t start reading everything.

You start with:

POST /payments

You find:

PaymentController
       ↓
PaymentService
       ↓
PaymentGateway
       ↓
PaymentRepository

Then you ask:

What happens when the same request arrives twice?

You find:

public PaymentResponse pay(PaymentRequest request) {
paymentGateway.charge(
        request.getAmount()
    );
    Payment payment = new Payment();
    payment.setOrderId(request.getOrderId());
    paymentRepository.save(payment);
    return PaymentResponse.success();
}

And suddenly you see the problem.

The system charges first.

Then saves the payment.

There is no idempotency check.

So:

Request #1
    ↓
Charge ₹1000
    ↓
Save payment

Then the user retries:

Request #2
    ↓
Charge ₹1000 AGAIN
    ↓
Save another payment

The customer loses ₹2000.

You didn’t understand the entire repository.

You didn’t need to.

You understood the payment journey.

And that’s enough to solve the problem.

The 30-Minute Rule

When I enter an unfamiliar codebase, I now give myself a rule:

Don’t try to become an expert in the first 30 minutes.

The goal of the first 30 minutes is only to answer:

What does this system do?
Where does a request enter?
What are the major services?
Where is the database?
What external systems exist?
Where are the important business flows?
What part is relevant to my task?

That’s it.

If you can answer those questions, you’ve already won.

The “Breadcrumb” Technique

Here’s something I use constantly.

When you find an important method, don’t read everything around it.

Follow the breadcrumbs.

Suppose:

orderService.createOrder()

calls:

inventoryService.reserve()

Follow that.

Then:

paymentService.charge()

Follow that.

Then:

eventPublisher.publish()

Follow that.

Then:

OrderCreatedEvent

Search where it’s consumed.

Maybe you discover:

OrderCreatedEvent
        ↓
InventoryConsumer
        ↓
AnalyticsConsumer
        ↓
NotificationConsumer

One search leads to another.

That’s how you navigate a large repository.

Not randomly.

Breadcrumb by breadcrumb.

Ask “Why?” Five Times

Sometimes understanding code requires more than knowing what it does.

You need to understand why it exists.

For example:

Why are we using Redis?

Because database queries are slow.

Why are database queries slow?

Because this endpoint gets huge traffic.

Why does it get huge traffic?

Because it’s called every time the user opens the dashboard.

Why don’t we calculate the data once?

Because the data changes frequently.

Now you understand something much deeper:

Redis isn’t there because:

“Redis is cool.”

It’s there because of a specific performance requirement.

That’s the difference between knowing a system and understanding a system.

Eventually, the Codebase Starts Talking to You

This is the beautiful part.

At first:

OrderService
PaymentService
InventoryService
KafkaConsumer
RedisConfig

look like random names.

After a few days, they become familiar.

You see:

OrderCreatedEvent

and immediately think:

“Okay, something happened after an order was created.”

You see:

@KafkaListener

and think:

“This isn’t part of the synchronous request. Something is happening asynchronously.”

You see:

@Transactional

and think:

“Okay, these database operations probably need to succeed or fail together.”

You see:

@Cacheable

and think:

“This data is probably expensive to calculate or fetch.”

The code stops looking like thousands of unrelated lines.

It starts looking like a system.

My Exact Process Today

If you give me a repository I’ve never seen before, I don’t randomly open files anymore.

I usually go in this order:

1. README
      ↓
2. Project structure
      ↓
3. Application entry point
      ↓
4. Configuration
      ↓
5. Database/entities
      ↓
6. API endpoints
      ↓
7. One important business flow
      ↓
8. External integrations
      ↓
9. Events / queues
      ↓
10. Tests
      ↓
11. Git history
      ↓
12. Task-specific code

And while doing this, I keep one document open.

I call it:

“My Mental Model.”

Inside it, I keep only:

System purpose
Major components
Important APIs
Important database entities
External dependencies
Important events
Main business flows
Things I don’t understand yet

That last section is extremely important.

Because instead of pretending I understand something, I write:

TODO:
Why do we publish OrderCreated twice?
TODO:
Why is Redis used here?
TODO:
Why does payment retry three times?
TODO:
Why is this transaction boundary here?

Now I have questions to investigate.

That’s much better than having confusion floating around in my head.

The biggest shift happens when you stop asking:

“How can I understand this entire codebase?”

and start asking:

“What story is this code trying to tell me?”

Because every large codebase has a story.

A request comes in.

Something validates it.

Some business decision happens.

Data changes.

Maybe another service gets called.

Maybe an event is published.

Maybe another system reacts.

Maybe the user gets a response.

That’s the story.

Your job as an engineer isn’t to memorize every class.

Your job is to reconstruct that story.

Once you know the story, the classes become characters.

The APIs become entry points.

The database becomes the memory.

The queues become the communication system.

The external services become the outside world.

The logs become the clues.

And Git history becomes the backstory.

Suddenly, the repository that looked like a monster yesterday starts looking like something you can actually reason about.

And that’s when you know you’ve crossed the line from:

“I am reading this code.”

to:

“I understand this system.”

That is the real skill.

Not knowing every file.

Not memorizing every class.

Not reading 10,000 lines a day.

It’s knowing where to look, what to follow, what questions to ask, and how to turn thousands of lines of code into one simple story.

One Last Thing

And one more thing I want to say.

Nothing is actually that complex.

Most of the time, our mind makes it complex before the problem even becomes complex. We see a large codebase, a new technology, system design, AI, distributed systems, or an unfamiliar architecture, and immediately think, “This is too difficult.”

But it isn’t.

You just need to make your mind more flexible, lean, and adaptable.

Don’t try to understand everything at once. Don’t try to memorize everything. Just follow what you are reading, understand one thing at a time, connect it with the previous thing, and keep moving.

Eventually, the things that looked confusing in the beginning start becoming obvious.

And if you are preparing for an AI Engineer or Software Engineer role, don’t try to learn everything randomly. Start exploring the right concepts, understand how they connect, and follow a systematic learning path.

Once your learning becomes systematic, your confidence naturally starts growing.

You don’t need to become a genius.

You just need to keep learning, keep exploring, and keep making your mind more adaptable.

That’s how complicated systems slowly become simple.

And if you want to take that journey further:

If you want to master your Software Engineering interviews:

https://algorithmxlr8.io/

If you want to master your AI Engineering interviews:

https://algorithmxlr8.io/

Keep learning.

Keep exploring.

And most importantly, don’t let your mind convince you that something is difficult before you have actually tried to understand it.

I still remember when I first started getting serious about AI.

At that time, AI was everywhere. Every second person was talking about ChatGPT, LLMs, RAG, agents and the next big AI framework.

I was already working as a software engineer, so I initially thought this would be easy for me. I knew how to build APIs, work with databases, write backend services and think about scalable systems. I thought I just needed to learn how to call an LLM.

I was wrong.

The first few days were actually exciting. I started with Python because most of the AI ecosystem was built around it. I already knew programming, so Python itself wasn’t the difficult part.

But then I started seeing things like embeddings, transformers, vector databases, RAG, LangChain, LangGraph, agents, memory, tool calling and MCP. Every time I understood one thing, I discovered three more things.

That was probably the first problem I faced.

I wasn’t struggling because AI was impossible to learn. I was struggling because I didn’t know what I should learn first.

So instead of randomly jumping between YouTube videos, blogs and new AI frameworks, I decided to step back and look at AI Engineering from an engineer’s perspective.

I asked myself a simple question:

If I had to build a real AI product from scratch, what would I actually need to know?

That question changed everything.

I Started With Things I Already Knew

I didn’t start my AI journey by trying to understand every mathematical detail behind neural networks. I started from the engineering side.

I went back to Python, because I wanted to become comfortable building AI applications with it. Then I made sure I was comfortable with things that are easy to ignore when you’re excited about AI: Git, APIs, HTTP, JSON, databases, SQL, asynchronous programming, Docker and basic cloud concepts.

At first, these topics didn’t feel like AI.

I remember thinking, “Why am I learning Git and Docker again? I came here to learn AI.”

But later I understood why.

Suppose you build an AI application that works perfectly on your laptop. Now imagine someone else needs to run it. Your application needs an API. It needs configuration. It needs a database. It needs deployment. It needs logging. It needs error handling. It needs version control.

The LLM is only one part of the application.

That realization became very important for me.

I stopped thinking about AI as a separate world and started looking at it as another type of software system.

Then I Went Back to Machine Learning

After getting comfortable with the engineering foundations, I started going deeper into Machine Learning.

I didn’t want to become a research scientist. My goal was to understand what was happening underneath the tools I was using.

I learned about data, features, training, validation, models, overfitting and evaluation. Then I moved towards Deep Learning and started understanding neural networks and how models learn patterns from data.

This was interesting because I had spent years writing software where I explicitly told the computer what to do.

With traditional programming, I could write:

“If this happens, do this.”

Machine Learning felt different.

Instead of manually writing every rule, I was giving the system data and allowing a model to learn patterns.

That difference was important to understand before jumping into modern LLM applications.

Then I Found Transformers

The next big step for me was understanding Transformers.

Before that, LLMs felt almost magical.

You type a question.

The model understands it.

It generates an answer.

But when I started learning about tokens, embeddings, attention and transformers, the magic started becoming engineering.

I still didn’t understand every mathematical detail, and I didn’t need to.

What I wanted was a mental model.

I wanted to understand what happens between:

Text → Model → Output

That foundation made it much easier for me to understand modern LLMs.

Then I Started Building With LLMs

Once I understood the basics, I started using LLM APIs.

This part was fun.

You could write a small amount of code and suddenly have an application that could summarize text, answer questions, extract structured information or generate content.

But after building a few simple applications, I hit another problem.

The model didn’t know my data.

I could ask it general questions, but what if I wanted it to answer questions about my own documents?

What if those documents were private?

What if the information changed every week?

What if I had thousands of documents?

Putting everything inside the prompt obviously wasn’t going to work.

That was when I discovered RAG.

RAG Was a Big Turning Point

RAG completely changed how I thought about AI applications.

Instead of:

Question → LLM → Answer

the system could become:

Question → Retrieve relevant information → LLM → Answer

That sounds simple when you write it like that.

But when I actually started building one, I realized how many engineering decisions were hiding underneath.

How should I split a document?

How large should a chunk be?

Should I keep metadata?

How should I create embeddings?

Where should I store them?

How many documents should I retrieve?

What if the retrieved documents are wrong?

What if the answer isn’t present in the knowledge base?

What if the document changes?

Suddenly my simple RAG application wasn’t so simple anymore.

And honestly, this is where I started learning much faster.

Because I wasn’t learning RAG from a definition anymore.

I was learning it because my own application was giving me bad answers.

I Started Understanding Embeddings and Vector Databases

To make RAG work properly, I needed to understand embeddings.

At first, the idea felt strange.

How can a piece of text become numbers?

And how can those numbers help me find similar information?

I started experimenting with embeddings and vector databases.

Then I understood the basic idea: instead of searching only for exact words, we could represent information in a form that allowed us to find semantically similar content.

That opened another area of learning.

Vector search.

Similarity.

Metadata filtering.

Indexing.

Retrieval.

Hybrid search.

And once again, I realized that the AI application was not just an LLM call.

There was an entire retrieval system around it.

Then I Started Building AI APIs

At this point I wanted to move beyond notebooks and small experiments.

I wanted to build proper AI backend services.

That’s where FastAPI became useful.

I started thinking about an AI application like any other backend system.

There is an API.

There is validation.

There are services.

There is a database.

There are external dependencies.

There are failures.

There are timeouts.

There are retries.

There are logs.

There are users.

The difference is that somewhere inside the system, there is an LLM making probabilistic decisions.

That difference creates a completely new set of engineering challenges.

For example, in a normal API, I can usually predict what a function will return.

With an LLM, the output can change.

It can be correct.

It can be incomplete.

It can misunderstand the request.

It can hallucinate.

It can return something in the wrong format.

So I started thinking about validation and reliability much more seriously.

Then Agents Started Getting Interesting

After building RAG systems, I started asking another question.

What if the AI could actually do something instead of just answering?

What if I gave it a calculator?

What if I gave it a database?

What if it could call an API?

What if it could search something?

What if it could decide which tool to use?

That led me into AI Agents and tool calling.

The first time I built a simple tool-calling workflow, it felt very different from a chatbot.

The model wasn’t just generating text.

It was deciding what action should happen next.

But again, the moment I started building more complicated workflows, things became messy.

What happens when the tool fails?

What happens when the model calls the wrong tool?

What happens when the agent keeps repeating the same action?

Where do we store state?

How do we retry?

When should a human take over?

Those questions pushed me deeper into agent engineering.

LangGraph Changed My Mental Model

When I started learning LangGraph, something clicked.

Instead of thinking about an agent as some magical autonomous AI, I could think about it as a workflow.

There is state.

There are nodes.

There are decisions.

There are edges.

There can be loops.

There can be retries.

There can be human intervention.

That was much easier for me to reason about as a software engineer.

I started building workflows where one step would retrieve information, another would call a tool, another would validate the result and another would decide what should happen next.

The more I built, the more I realized that a good AI agent is not about giving the model unlimited freedom.

It’s about designing the right boundaries around it.

Then I Started Thinking About Memory

One problem became obvious after building conversational systems.

The AI forgets.

You can have a long conversation with a system, start a new session and suddenly it knows nothing.

That made me curious about AI memory.

What should an AI remember?

Should it remember everything?

Probably not.

Should it remember user preferences?

Maybe.

Should it remember previous tasks?

Sometimes.

Should memories expire?

Probably.

Where should memories be stored?

How should they be retrieved?

These questions made me realize that memory is not just “store the chat history.”

It is another engineering problem.

Then Came MCP

As I continued exploring AI agents and tools, I came across MCP.

What interested me wasn’t just the protocol itself.

It was the idea behind it.

AI systems need to interact with external tools and data.

Instead of every AI application creating completely different integrations, there could be a standardized way for models and applications to discover and use capabilities.

That made me think about AI applications differently again.

The future AI application isn’t just a chatbot sitting inside a box.

It can interact with databases, APIs, files, developer tools and other systems.

The model becomes one part of a much larger ecosystem.

Then Reality Hit: AI Can Be Wrong

This was probably one of the most important lessons I learned.

A demo can look amazing.

A production system is different.

You can build a RAG system that works perfectly on five documents.

Then you put 50,000 documents into it and everything changes.

You can build an agent that works perfectly ten times.

Then it gets a slightly different request and enters a loop.

You can write the perfect prompt.

Then the model gives you an unexpected answer.

That forced me to learn about AI evaluation.

I started thinking about questions like:

How do I know my retrieval is good?

How do I know the answer is correct?

How do I detect hallucinations?

How do I compare two versions of a prompt?

How do I know whether a new model is actually better?

Traditional unit tests aren’t enough for every AI problem.

AI systems need their own evaluation strategy.

That was another major shift in my thinking.

Then I Started Thinking About Production

Once you build enough AI applications, you eventually stop asking:

“Can I make this work?”

and start asking:

“Can I make this reliable?”

That’s a completely different question.

Now you start thinking about latency.

Token costs.

Caching.

Retries.

Timeouts.

Rate limits.

Streaming.

Logging.

Monitoring.

Tracing.

Security.

Authentication.

Deployment.

Scaling.

Model selection.

Fallbacks.

This is where my software engineering background became extremely useful.

Because AI may be new, but production engineering problems are not.

An AI application still needs good architecture.

It still needs good APIs.

It still needs proper databases.

It still needs observability.

It still needs security.

It still needs testing.

It still needs people who understand what happens when something goes wrong at 2 AM.

I Also Made Plenty of Mistakes

One thing I don’t want to hide is that I wasted a lot of time.

I jumped between frameworks.

I watched tutorials without building anything.

I sometimes learned a tool before understanding the problem it was solving.

I built small demos that looked impressive for five minutes but taught me very little.

There were also times when I thought I understood something because I could explain the definition.

Then I tried building it.

And got stuck.

That was a good lesson.

There is a huge difference between knowing:

“A vector database stores embeddings.”

and actually debugging why your retrieval system is returning irrelevant documents.

There is a difference between knowing:

“Agents can call tools.”

and debugging an agent that keeps calling the same tool again and again.

There is a difference between knowing:

“RAG reduces hallucinations.”

and actually discovering that bad retrieval can make the final answer even worse.

Building exposes those gaps very quickly.

Eventually I Stopped Learning AI as Separate Topics

This was probably the biggest change in my learning.

Earlier, I saw everything separately.

Python was one thing.

Machine Learning was another.

LLMs were another.

RAG was another.

Agents were another.

Vector databases were another.

FastAPI was another.

MCP was another.

Now I see them as pieces of one system.

A user sends a request.

The API receives it.

The application decides what needs to happen.

Maybe it retrieves information.

Maybe it calls a tool.

Maybe it uses memory.

Maybe an agent workflow is required.

The model generates a response.

The response gets validated.

The system records what happened.

Then we evaluate whether the system actually did a good job.

Suddenly all those individual topics started making sense together.

What Actually Helped Me Learn

If I had to choose one thing that helped me the most, it wouldn’t be a course or a particular framework.

It would be building things.

Whenever I learned something, I tried to build something small around it.

Learned RAG?

Build one.

Learned tool calling?

Build one.

Learned agents?

Build one.

Learned memory?

Build one.

Learned MCP?

Build one.

Learned evaluation?

Add evaluation to an existing project.

The project didn’t need to be huge.

Sometimes a small working project taught me more than spending an entire weekend watching tutorials.

The pattern became very simple:

Learn → Build → Break → Debug → Understand → Build again.

That’s how the knowledge started sticking.

I Also Started Explaining What I Learned

Another thing that helped me was writing about these concepts.

When you explain something to another engineer, you quickly discover what you don’t understand.

You think you understand RAG.

Then someone asks:

“Why this chunk size?”

And you realize you never really thought about it.

You think you understand agents.

Then someone asks:

“What happens when the tool fails?”

And suddenly you have another rabbit hole to explore.

Writing and teaching forced me to go one level deeper.

That’s also one of the reasons I started organizing these topics on AlgorithmXlr8.

I didn’t want AI Engineering to feel like a collection of random technologies.

I wanted to create a path where someone could start from the fundamentals and gradually reach real AI systems.

My Biggest Lesson

Looking back, I don’t think I learned AI Engineering by following one perfect roadmap.

I learned it by repeatedly getting stuck.

I learned Python because I needed it to build something.

I learned RAG because my LLM didn’t know my data.

I learned vector databases because I needed better retrieval.

I learned agents because I wanted the AI to take actions.

I learned LangGraph because agent workflows became difficult to manage.

I learned memory because conversations needed context.

I learned MCP because AI systems needed better access to tools.

I learned evaluation because “it seems to work” wasn’t good enough.

And I learned production engineering because eventually every demo needs to survive outside your laptop.

That’s what made AI Engineering interesting to me.

It isn’t just about AI.

It is about bringing everything together.

Software Engineering.

Data.

Machine Learning.

LLMs.

Backend Engineering.

Systems.

Infrastructure.

And a lot of experimentation.

If I Were Starting Again

If I were starting AI Engineering from zero today, I wouldn’t try to learn everything at once.

I’d first get comfortable with Python, Git, APIs, databases and basic software engineering.

Then I’d learn the foundations of Machine Learning and Deep Learning.

Then I’d understand LLMs and Transformers.

Then I’d build a RAG system.

Then I’d learn embeddings, vector databases and retrieval properly.

Then I’d build AI APIs with FastAPI.

Then I’d move into tool calling and AI Agents.

After that I’d learn LangGraph, state and workflows.

Then memory and MCP.

Then I’d spend serious time on evaluation, guardrails, reliability and production AI.

But I wouldn’t spend months learning each topic before building.

I’d learn enough.

Then build.

Then let the problems tell me what I need to learn next.

The Journey Isn’t Finished

Even after learning all of this, I don’t consider myself “done” with AI Engineering.

That’s probably the biggest difference between learning a technology and becoming an engineer.

There is always another problem.

A better retrieval strategy.

A better model.

A cheaper inference setup.

A more reliable agent.

A better evaluation method.

A new protocol.

A new architecture.

A new production problem.

And that’s okay.

I don’t think the goal is to finish AI Engineering.

The goal is to become good enough at engineering that when a new AI technology appears, you can understand where it fits, why it exists and when you should actually use it.

That’s the mindset I’m trying to build.

Don’t chase every new model.

Don’t learn every framework just because it’s trending.

Don’t build another chatbot just to say you built an AI project.

Understand the fundamentals.

Build real things.

Break them.

Debug them.

Measure them.

And keep going.

That’s how I learned AI Engineering.

And honestly, I’m still learning.

Want to Start Your AI Engineering Journey?

If you want to start learning AI Engineering and follow the same kind of path I followed, I’ve put the complete learning journey on my platform, AlgorithmXlr8.io

If you’re starting your AI Engineering journey, you can start here too.

I’ve tried to structure it around the same question I had when I started: what should I learn first, what should I learn next, and how do all these pieces actually connect?

You can start with the fundamentals and gradually move towards Machine Learning, LLMs, RAG, AI Agents, workflows, memory, MCP and the other concepts required to build real AI systems.

If you’re also trying to figure out where to start with AI Engineering, you can start here:

https://algorithmxlr8.io

I’ve learned these concepts by building, breaking, debugging and going back to the fundamentals. I’ve tried to put that same approach into the platform so you don’t have to spend months jumping randomly between different resources.

Last Sunday, I was sitting at home with my laptop.

My father walked in and asked,

Beta, what exactly do you build all day?

I smiled.

I said,

AI.

He looked confused.

Then he asked another question.

If AI is so intelligent, can it answer questions about our family’s documents?

I confidently opened ChatGPT.

I uploaded nothing.

Then I asked,

“What is my electricity bill for last month?”

The AI gave a beautiful answer.

It was also completely wrong.

My father laughed.

“So this intelligent AI doesn’t even know our own documents?”

That one sentence stayed in my head.

The Real Problem

Most people think an AI knows everything.

It doesn’t.

An LLM is like a student who studied millions of books before the exam.

But once the exam starts…

The books are taken away.

Now the student has to answer only from memory.

If you ask,

“Who won the Cricket World Cup 2011?”

Easy.

If you ask,

“What’s written on page 18 of my company’s HR policy?”

It has no idea.

Sometimes it says,

“I don’t know.”

Sometimes…

It confidently makes things up.

That’s called a hallucination. RAG was created to reduce those unsupported answers by grounding responses in external information.

Then I Thought…

What if the student was allowed to open the library before answering?

Not the whole library.

Just the few books related to the question.

Now imagine the process.

You ask,

“How many casual leaves do employees get?”

Instead of guessing…

Someone first walks into the library.

Searches the HR policy.

Finds the correct page.

Brings it back.

Only then does the student answer.

That’s exactly what RAG does.

A Simple Real-Life Example

Imagine your mother keeps every recipe in a notebook.

One day you ask,

“How do we make paneer butter masala?”

You don’t remember.

So you open the notebook.

Find the recipe.

Read it.

Then cook.

You didn’t magically know the recipe.

You retrieved it first.

That’s Retrieval.

Then you used that information to create the answer.

That’s Generation.

Together…

Retrieval-Augmented Generation.

How I Built It

I didn’t start with AI.

I started with documents.

I collected PDFs.

Company policies.

Technical documentation.

Product manuals.

Articles.

Then I broke every document into smaller pieces.

Because finding one paragraph is much easier than searching a 500-page PDF.

Next…

Every small paragraph became a numerical representation called an embedding.

Think of it like giving every paragraph its own GPS coordinates.

Similar ideas end up close together.

Different ideas stay far apart.

These embeddings were stored inside a vector database.

Now my documents became searchable by meaning instead of exact words. This semantic-search pattern is one of the core ideas behind modern RAG systems.

What Happens When Someone Asks a Question?

Suppose someone types,

“How many sick leaves can I take?”

The system doesn’t send that question directly to the LLM.

Instead…

The question is converted into an embedding.

The vector database searches for paragraphs with similar meaning.

It finds the most relevant chunks.

Those chunks are sent to the LLM.

The LLM reads them.

Then answers using only that information.

Not memory.

Evidence.

The Biggest Difference

Without RAG: Question → LLM → Guess

With RAG: Question → Search Documents → Retrieve Information → LLM → Accurate Answer

One extra step.

A huge difference.

The Funny Part

After building it…

I went back to my father.

This time he asked,

“What’s written in our electricity bill?”

The AI searched the uploaded bill.

Read the exact amount.

Answered correctly.

My father smiled.

“Now this feels intelligent.”

That was the moment I understood.

RAG doesn’t make an AI smarter.

It gives the AI access to the right information before it speaks.

Final Thought

When people hear “RAG,” they often imagine a complicated AI architecture.

But at its heart, it’s surprisingly simple.

Think of three roles:

• The Library → Your documents.

• The Librarian → Finds the right pages.

• The Teacher → Reads those pages and explains them in simple language.

That’s all a RAG system really is. (https://algorithmxlr8.io/ai/engineering/home/rag-vector-search/rag-fundamentals/what-is-rag)

And once you understand it this way, building one feels much less intimidating.

Want to Learn AI Engineering Step by Step?

Programming, Math & Data Foundations
│
├── Statistics & Probability
├── NumPy & Pandas
└── Data Cleaning & Visualization
│
Machine Learning & Deep Learning
│
├── Machine Learning Fundamentals
├── Deep Learning Fundamentals
└── PyTorch
│
LLM Engineering & Applications
│
├── Prompt Engineering
├── RAG & Vector Search
├── Fine-Tuning
├── MCP
└── Hugging Face
│
AI Agents & Orchestration
│
├── AI Agents & Tool Calling
└── LangChain & LangGraph
│
Deployment, Safety & Production
│
├── LLMOps
├── AI Safety & Evaluation
└── Security

Explore the complete at: https://algorithmxlr8.io/

Everyone talks about resigning. Very few people talk about the preparation that happens months before that decision.

The Thought Stayed in My Mind for a Long Time

I still remember the first time I seriously thought about leaving my 9–5 job.

It wasn’t because I had a bad manager.

It wasn’t because I hated my work.

It wasn’t even because I wanted to become an entrepreneur overnight.

The thought came from something much simpler.

Every evening after work, I would open my laptop again. Not because someone asked me to, but because I genuinely wanted to build something of my own.

Some days it was a small backend project.

Some days it was a DSA platform.

Sometimes I would just spend two hours reading about distributed systems or improving my engineering skills.

I started noticing something.

No matter how tired I was after office hours, I still found myself opening my laptop again.

That’s when I realized this wasn’t just a hobby anymore.

It was becoming something much bigger.

But even then, I never thought, “I’ll resign next month.”

Instead, I asked myself a different question.

“What needs to happen before I can even think about leaving?”

Looking back, that single question probably saved me from making a very emotional decision.

Watching People Quit Made Me Realize Something

Around the same time, I started seeing a lot of posts on LinkedIn and Twitter.

Someone would announce,

“Today is my last working day.”

Another person would post a picture of their resignation letter.

A few weeks later they would announce their startup.

It looked exciting.

For a few days, I also felt like maybe I should just do the same.

But social media only shows the final chapter.

Nobody posts about the months of uncertainty before that.

Nobody talks about wondering whether they’ll have enough money six months later.

Nobody talks about the nights when they question whether they’re making the biggest mistake of their career.

The more I thought about it, the more I realized something.

I wasn’t interested in quitting my job.

I was interested in building a life where leaving my job became a natural next step.

Those are two completely different things.

I Made One Rule for Myself

That weekend, I opened my notebook and wrote down one rule.

“Don’t resign because you’re frustrated. Resign only when you’re prepared.”

It sounds simple.

But it changed the way I looked at everything.

Whenever I felt like quitting after a stressful day at work, I would read that line again.

Frustration lasts for a few hours.

A resignation lasts much longer.

I promised myself that I would never make a career decision based on one bad week.

If I ever left, it would be because I had spent months preparing for it.

Not because I had a bad meeting on Monday morning.

My Evenings Started Looking Different

Earlier, once work was over, I would relax, watch random videos, scroll through social media, and call it a day.

Slowly, that changed.

I gave myself a small target.

Just two hours.

No unrealistic schedules.

No “I’ll study eight hours after office.”

That never worked.

Some evenings I worked on a side project.

Some evenings I learned AWS.

Some days I practiced DSA.

Other days I read about System Design or tried to understand a technology I had been avoiding.

There were many nights when I didn’t feel like doing anything.

Office had already drained my energy.

But I made a promise to myself.

Even if I wasn’t productive for two hours, I would at least sit in front of my laptop.

Some days those two hours became twenty minutes.

Some days they became four hours.

The important part wasn’t the duration.

It was showing up consistently.

I Stopped Waiting for the Perfect Idea

One mistake I made in the beginning was thinking I needed an amazing startup idea before I started building.

So I kept waiting.

Every week I thought,

“Maybe I’ll get a better idea next month.”

That month never came.

Eventually, I stopped chasing ideas.

Instead, I started solving problems I personally cared about.

If a tool didn’t exist, I built it.

If I found learning difficult, I created notes.

If I wanted a better way to study DSA and System Design, I started working on it little by little.

None of those projects were perfect.

Some never even got completed.

But every project taught me something.

How to deploy applications.

How to work with cloud services.

How to design better APIs.

How to fix production bugs.

How to deal with users.

More importantly, they taught me that building consistently matters far more than waiting for the perfect opportunity.

Money Was the Conversation Nobody Wanted to Have

For a long time, I ignored the financial side of leaving a job.

I only thought about passion.

Reality doesn’t work like that.

Rent doesn’t stop because you’re following your dream.

Bills don’t disappear.

Responsibilities don’t disappear.

I realized that if I wanted the freedom to make bold career decisions, I first needed financial stability.

So instead of rushing, I started saving.

Not because I expected to fail.

Because I wanted peace of mind.

Knowing that I could survive for several months without depending on my next salary completely changed the way I looked at risk.

Confidence doesn’t only come from skills.

Sometimes it comes from knowing you have enough time to recover if things don’t go according to plan.

The Biggest Lesson I Learned

Today, whenever someone asks me,

“How do I prepare before leaving my 9–5 job?”

I don’t tell them to resign.

I tell them to prepare until resigning feels obvious.

Build something while you’re still employed.

Learn skills that increase your value.

Save money.

Create habits that prove you can stay disciplined without someone managing your time.

If you can’t stay consistent for two hours after office, quitting your job won’t magically make you disciplined.

Freedom is amazing.

But freedom without discipline quickly turns into procrastination.

Looking Back

I’m glad I didn’t leave my job the first time I felt frustrated.

If I had, it would’ve been an emotional decision.

Instead, I spent months preparing quietly.

Nobody noticed those evenings.

Nobody saw the weekends I spent learning instead of going out.

Nobody celebrated the small milestones.

But those small efforts slowly built something much bigger than I had imagined.

Looking back now, I don’t think resigning was the hardest part.

Preparing for it was.

And that’s exactly why the transition felt less like jumping into the unknown and more like taking the next logical step.

If you’re thinking about switching from your 9–5, don’t ask yourself, “When should I resign?”

Ask yourself, “What can I start building today so that when the day comes, leaving feels like a decision not a gamble?”

That one question changed everything for me.

Maybe it’ll do the same for you.

If You’re Serious About Making the Switch

If you’ve read this far, chances are you’re thinking about something bigger than just changing jobs.

Maybe you want to become a better Software Engineer.

Maybe you’re preparing for AI Engineering roles.

Maybe you’re aiming for DevOps, SRE, or Backend Engineering interviews.

Whatever your goal is, one thing is true having a structured learning path makes a huge difference.

One of the biggest mistakes I made early in my career was learning random topics without any direction. I’d jump from one YouTube video to another, bookmark dozens of articles, and buy courses I never finished.

Progress became much faster once I started following a clear roadmap instead of learning whatever looked interesting that day.

That’s exactly why I built AlgorithmXlr8.

It isn’t just another collection of videos or interview questions. It’s a structured platform designed to help engineers master Software Engineering, AI Engineering, DevOps, DSA, System Design, and other interview-focused skills through a step-by-step learning path.

Everything is organized in the order you should actually learn it, so you always know what to study next instead of wondering where to begin.

If you’re serious about switching from your 9–5 job, growing as an engineer, or preparing for your next interview, I think you’ll find it genuinely useful.

You can explore it here: https://algorithmxlr8.io

A few months ago, I met a software engineer at a café. He wasn’t a beginner. He had a decent job, a good salary, and years of experience. But he looked stressed. After a few minutes, he finally said,

“I think I chose the wrong career.”

I asked him why.

He opened Instagram.

Then LinkedIn. Then YouTube.

Everywhere, the headlines looked almost identical.

“AI replaced another developer.”

“Software Engineering is dead.”

“Stop learning to code.”

“Developers won’t exist in five years.”

He looked at me and asked one question.

“Should I quit Software Engineering?”₹

For a moment, I didn’t answer. Instead, I asked him a different question.

If Software Engineering Is Dead… Then why are companies still hiring?

If software engineering is truly over…

Why are companies still interviewing engineers every single day?

Why are startups still building engineering teams?

Why are experienced engineers still getting multiple offers?

Why are companies spending lakhs and sometimes crores on skilled developers?

Because companies don’t hire people to write code. They hire people to solve problems. And those are two very different things.

Writing Code Was Never the Job

Most people think software engineering means writing code.

That’s only about 20% of the job. The real job is answering questions like:

• Why is the system slow?

• Why did production crash at 2 AM?

• Why are users losing payments?

• How do we scale from 10,000 users to 10 million?

• How do we protect customer data?

• How do we build something that still works five years from now?

AI can generate code. But it doesn’t own your business. It doesn’t understand your customers.

It doesn’t sit in meetings deciding technical trade-offs. And it doesn’t take responsibility when production goes down. That’s what software engineers do.

Let Me Tell You a Real Story

A friend of mine built an application using AI. In just three days.

He was excited. Everything worked perfectly.

Until… Real users started using it. The application became slow. The database started failing. APIs timed out. Memory usage kept increasing. Users couldn’t log in. Payments failed. Suddenly, AI wasn’t enough.

Now he needed someone who understood caching. Database indexing. Load balancing. Distributed systems. System design. Monitoring. Scaling.

Exactly the skills that experienced software engineers bring to the table. That’s when he realized something important.

Building an application is easy. Building one that survives millions of users is software engineering.

AI Isn’t Your Competition

It’s Your New Teammate. Think about calculators. Did calculators kill mathematics?

No.

They removed repetitive work. Think about cameras on smartphones. Did they eliminate photographers?

No.

Great photographers became even better.

Think about GPS. Did it replace drivers?

No.

It made navigation easier. AI is exactly the same. It removes repetitive coding. Now engineers have more time to think. More time to design. More time to innovate. More time to solve real problems.

The Engineers Who Will Struggle

Let’s be honest.

Some engineers will struggle. Not because of AI. Because they stopped learning. Technology has always changed.

From Java to Kotlin. From Monoliths to Microservices. From On-Premise to Cloud. From Manual Deployments to DevOps.

Every generation had to adapt. This generation is no different. The engineers who refuse to learn AI will struggle. The engineers who learn AI will move faster than ever before.

What Companies Actually Pay For

Imagine two developers.

The first one writes code very quickly. The second one prevents production outages.

Improves performance by 70%. Reduces cloud costs. Designs scalable systems. Mentors the team. Builds reliable architecture.

Which one would you hire?

Exactly.

Companies don’t pay for typing speed. They pay for impact.

The Biggest Opportunity Most People Are Missing

While thousands of people are posting,

“AI is replacing developers.”

A smaller group is quietly learning:

• AI tools

• System Design

• Cloud

• Databases

• Distributed Systems

• Architecture

• Security

• Performance Optimization

Five years from now…

Guess which group will be leading engineering teams.

The Future Doesn’t Need More Coders

It Needs Better Engineers.

The best software engineers of the future won’t be the ones who memorize every programming language.

They’ll be the ones who know:

How to think.

How to design.

How to communicate.

How to understand business.

How to use AI effectively.

How to solve problems nobody else can solve. Because code is becoming cheaper. Good decisions are becoming more valuable.

If You’re Feeling Scared…

That’s okay. Almost every engineer has felt that way at some point. But don’t let fear make your decisions.

Use it as fuel. Learn one new skill. Build one new project. Understand one new concept. Keep improving.

Small progress every day always beats endless worrying.

Here’s What I Believe

Software Engineering isn’t dying.

Average engineering is. Copy-paste engineering is. Learning just enough to pass interviews is. But engineers who understand fundamentals, system design, architecture, AI, and problem-solving?

They are becoming more valuable than ever.

So… Is Software Engineering Dead?

No.

It’s evolving. And every time technology evolves, the people who adapt become the biggest winners. Don’t spend your energy competing with AI.

Learn to work with it. Don’t fear the future. Build it.

Because the world will always need people who can solve difficult problems. And that’s what great software engineers do.

The keyboard may change. The tools may change. Even programming languages will change. But people who can think, build, and solve real problems will never go out of demand. So the next time someone tells you,

“Software Engineering is dead.”

One Last Thing…

If you’re serious about building a long-term career in software engineering, don’t waste your time worrying about whether the industry is dying.

Spend that time becoming so good that companies want you.

Master Data Structures & Algorithms. Master Low-Level Design. Master High-Level System Design. Understand real-world architecture, scalability, trade-offs, and problem-solving.

Visit:

Master Software Engineering Interviews

These are the skills that will stay valuable no matter how technology evolves. So if you truly want to crack software engineering interviews and become a better engineer, stop overthinking.

Start learning. Start building. Stay consistent. Your future self will thank you. The best time to start was yesterday. The second-best time is today.

It was around 1:30 AM.

The kind of night where everything feels slow. Your brain is tired, but your code is still not working.

I was stuck on a simple problem. Not even something big. Just a piece of logic that was running slower than expected.

I had already spent an hour trying to fix it myself.

Finally, like always, I opened Claude 4.6.

I pasted my code and wrote the most lazy prompt possible:

“Code is slow. Fix it.”

It replied in a few seconds.

The answer wasn’t bad. It pointed out a loop, suggested a small optimization, and gave an updated version.

I tried it.

Still not right.

So I asked again. This time more specific.

Then again.

After a few back-and-forth messages, I finally got what I needed.

At that moment, I didn’t think much about it.

Because honestly, that had become normal.

AI helps… but you still have to guide it a lot.

The Next Day, Same Problem, Different Experience

The next evening, I ran into a similar issue.

Same type of logic. Same frustration.

But this time, I decided to try Claude 4.7.

No special reason. Just curiosity.

I pasted my code again. Same lazy prompt:

“Code is slow. Fix it.”

And this time, something felt different.

It Didn’t Just Answer It Understood

Claude 4.7 didn’t jump straight into fixing the code.

It first explained what the code was doing.

Then it pointed out exactly where the inefficiency was.

Then it said something that caught my attention:

“This might work for small inputs, but it will fail when the data grows.”

That line felt real.

Because that’s exactly how bugs behave in real life.

Then it gave:

• an optimized approach

• a cleaner structure

• and a short explanation of why this approach is better

I copied the code.

Ran it.

It worked.

No second prompt.

No back-and-forth.

Just done.

That’s When It Clicked

The difference between Claude 4.6 and 4.7 is not something you see in a demo.

It’s something you feel while working.

Claude 4.6 felt like someone who knows the answer.

Claude 4.7 feels like someone who understands the problem.

And that’s a big difference.

Real Difference, Explained Without Tech Words

Let’s keep it simple.

Imagine two people helping you.

Person 1 (Claude 4.6)

• Smart

• Knows theory

• Gives correct answers

• But needs clear instructions

Person 2 (Claude 4.7)

• Also smart

• But listens better

• Understands incomplete questions

• Thinks a step ahead

In real life, you would always prefer Person 2.

Not because they are smarter.

But because they make your life easier.

Where You Actually Notice the Change

This is important.

If you just ask:

• “What is a stack?”

• “Explain arrays”

• “Write hello world”

You won’t feel much difference.

Both versions will work fine.

But things change when problems become real.

1. When Your Question Is Not Perfect

In real life, we don’t ask clean questions.

We say things like:

• “Something is wrong here”

• “This is not working”

• “Why is this slow?”

Claude 4.6 needs more clarity.

Claude 4.7 handles this mess better.

It fills the gaps.

2. When You’re Coding for Real Projects

This is where the gap becomes obvious.

Claude 4.6:

• Solves the problem

• But sometimes gives “textbook” answers

Claude 4.7:

• Thinks about real usage

• Handles edge cases

• Suggests better structure

It feels closer to how a real developer thinks.

3. When the Task Is Long

Try asking both to:

• design a system

• explain a full concept step by step

• debug a large piece of code

Claude 4.6 sometimes loses direction midway.

Claude 4.7 stays more consistent.

It doesn’t forget what it started.

4. When You Care About Accuracy

Earlier, one common issue was confidence.

Claude 4.6 could sound very sure… even when slightly wrong.

Claude 4.7 feels more careful.

It avoids overconfidence.

And that builds trust.

But Here’s the Honest Truth

Claude 4.7 is not perfect.

Sometimes:

• it over-explains

• it takes longer

• it feels like it’s thinking too much

Some people even prefer faster, shorter answers.

And that’s fair.

Because “better” depends on what you need.

The Real Upgrade No One Talks About

It’s not speed.

It’s not accuracy.

It’s not even coding.

It’s this:

Less friction.

With Claude 4.6:
You → explain → correct → refine → repeat

With Claude 4.7:
You → ask → get closer answer immediately

That small reduction in effort changes the whole experience.

Why This Matters More Than It Looks

If you use AI once a day, it doesn’t matter much.

But if you:

• code daily

• prepare for interviews

• build projects

• learn new concepts

Then this difference saves time.

And more importantly, it saves mental energy.

Final Thought

There was no big announcement in my life.

No dramatic moment.

Just one night where I realized:

“I didn’t have to ask twice.”

And that’s when I knew…

Claude 4.7 is not a big leap.

It’s just quietly better.

And sometimes, that’s exactly what we need.

Now, what do u all think,

End of Jobs ?

“My opinion is, I don’t think so.”

Stop asking AI to fix your bugs. Ask it to reproduce them first.

This one change made my debugging sessions dramatically better.

Here’s what I used to do: paste the buggy code, say “fix this,” and hope for the best. Sometimes it worked. More often I got a patch that looked right but didn’t actually address the root cause or worse, introduced something new.

The shift that changed everything:

Write a failing test first. Fix second.

The workflow that actually works:

1. Give Claude the full picture logs, relevant code, what you expected vs. what happened

2. Ask it to write a test that reproduces the bug

3. Run the test. If it passes, Claude doesn’t understand the bug yet — go back to step 1

4. Once it fails, ask for 2–3 different fix approaches

5. Let the tests pick the winner

Real scenario: Payment double-charge bug

Users were occasionally getting charged twice for the same transaction. Support tickets were piling up. The instinct? “Claude, fix the double charge bug.”

Bad move.

Step 1: Give full context:

“Here’s our payment processor webhook handler and the Stripe logs. Some users are seeing duplicate charges. Expected: one charge per order. Actual: sometimes two charges appear 2–3 seconds apart with the same order ID.”

Step 2: Ask for a failing test, not a fix:

“Write a test that reproduces this. Simulate two webhook events arriving simultaneously for the same order. Don’t fix anything yet.”

Claude wrote this:

@Test
void testDuplicateWebhookDoesNotDoubleCharge() throws InterruptedException {
    String orderId = “order_123”;

    // Simulate two payment.success webhooks arriving at the same time
    ExecutorService executor = Executors.newFixedThreadPool(2);
    CountDownLatch latch = new CountDownLatch(1);

    Callable<Void> webhookTask = () -> {
        latch.await(); // both threads wait, then fire simultaneously
        paymentWebhookHandler.process(orderId);
        return null;
    };

    List<Future<Void>> futures = executor.invokeAll(
        List.of(webhookTask, webhookTask)
    );
    latch.countDown(); // release both at the same time

    for (Future<Void> f : futures) f.get(); // wait for completion

    List<Charge> charges = chargeRepository.findByOrderId(orderId);
    assertEquals(1, charges.size()); // ← This failed. Returned 2.
}

Test failed. Bug confirmed. Claude actually understood the problem.

Step 3: Now ask for fixes:

“Good. Now give me 2–3 approaches to make this test pass.”

Claude returned three options:

Option A: Database-level unique constraint + idempotency key (most robust):

// In PaymentWebhookHandler.java
@Transactional
public void process(String orderId) {
    if (chargeRepository.existsByOrderId(orderId)) {
        log.info(”Duplicate webhook ignored for order: {}”, orderId);
        return;
    }
    // Schema has UNIQUE constraint on order_id column
    Charge charge = new Charge(orderId, amount);
    chargeRepository.save(charge); // throws on duplicate, transaction rolls back
}java

Option B: Redis distributed lock with TTL (fast, low DB pressure):

@Autowired
private RedissonClient redisson;

public void process(String orderId) {
    RLock lock = redisson.getLock(”payment-lock:” + orderId);
    try {
        if (lock.tryLock(0, 10, TimeUnit.SECONDS)) {
            if (!chargeRepository.existsByOrderId(orderId)) {
                chargeRepository.save(new Charge(orderId, amount));
            }
        }
    } finally {
        if (lock.isHeldByCurrentThread()) lock.unlock();
    }
}

Option C: Webhook deduplication table (simplest, no extra infra):

@Transactional
public void process(String orderId) {
    boolean inserted = webhookLogRepository.insertIfAbsent(orderId);
    // INSERT INTO webhook_log (order_id) VALUES (?) ON CONFLICT DO NOTHING
    if (!inserted) {
        log.warn(”Webhook already processed for order: {}”, orderId);
        return;
    }
    chargeRepository.save(new Charge(orderId, amount));
}

Tests ran against all three. Option A passed cleanly and handled edge cases the others missed the database constraint acted as the final safety net even if application logic had a gap.

What “fix the bug” would have given: probably a simple if (charged) return; check which breaks the moment two threads hit it at the same millisecond.

What the test-first approach gave: a race condition fix with proof it works, plus a regression test that lives in the codebase forever.

When you frame it this way, Claude stops being an autocomplete for fixes and starts acting like a proper debugging partner one that has to prove it understands the problem before touching anything.

Less guesswork. Better fixes. And you walk away with regression coverage that catches it if it ever comes back.

AI shouldn’t guess fixes. It should prove them.

Nobody designs a system thinking this will happen.

Cache is supposed to be the fast path.
Database is supposed to be the necessary evil.

And yet, one day, you realize something terrifying:

The cache is slowing you down.

Not breaking things.
Not crashing.
Just… quietly adding latency.

This is the story of how that happens and the system design lessons hiding inside it.

It Started With a Feeling, Not a Metric

Everything looked fine on dashboards.

Redis was up.
Database was healthy.
CPU was relaxed.

But users kept saying:

“The app feels heavy.”

That’s the kind of feedback engineers hate, because there’s nothing concrete to grab onto. No error code. No stack trace. Just vibes.

In system design terms, this is your first red flag:

Average metrics lie. Tail latency tells the truth.

The Architecture Was “Textbook Correct”

On paper, the flow was clean:

Client → API → Cache → Database

Read-heavy traffic?
Perfect use case for caching.

But real systems evolve.

Over time:

• More fields were added to responses

• More features depended on the same cached object

• Cache entries grew larger and larger

System design lesson #1, learned the hard way:

Caching coarse-grained objects feels convenient… until it isn’t.

The Forbidden Experiment

Late one night, someone asked a dangerous question:

“What if we just skip the cache for this endpoint?”

Not forever.
Just to compare.

Same request.
Same user.
Same data.

The response without cache came back faster.

That moment hurt.

Because it breaks a core assumption:

Cache ≠ faster by default.

The Database Had Quietly Improved

Here’s what we had ignored.

The database wasn’t “slow” anymore.

• Hot data lived in memory

• Indexes were tuned over months

• Queries were narrow and predictable

In system design terms:

Modern databases already behave like partial caches.

Meanwhile, Redis was:

• Remote

• Shared across services

• Handling bigger payloads

• Doing more work per request

Every cache read paid a network cost plus processing cost.

Cache Misses Were Costlier Than We Thought

The cache hit rate looked decent around 80–85%.

That number gave false confidence.

But the remaining misses were brutal.

A cache miss meant:

1. Network call to cache

2. Miss response

3. Query to DB

4. Write back to cache

So for some users, especially new or inactive ones, we had added an extra hop for no benefit.

System design lesson #2:

A cache with poor hit rate can be worse than no cache at all.

Redis Had Become a Bottleneck, Not a Booster

Another uncomfortable realization:

Redis was now on the critical path for everything.

Even simple reads.
Even low-traffic endpoints.

This violated a core design principle:

Shared infrastructure becomes a bottleneck before you expect it.

The DB had connection pooling, parallelism, and isolation.
Redis had queues.

The Fix Wasn’t “Optimize Redis”

We didn’t tune Redis.
We didn’t increase memory.
We didn’t add more nodes.

We stepped back and asked:

“Should this even be cached?”

Some data:

• Changed often

• Was cheap to compute

• Was already optimized in DB

So we let the database handle it.

Other data:

• Truly hot

• Truly reused

• Truly expensive

That stayed in cache.

System design lesson #3:

Cache only what is expensive, stable, and frequently reused.

The System Felt Lighter Overnight

Nothing dramatic happened.

No heroic deploy.
No massive refactor.

But the app felt smooth again.

Support tickets dropped.
Latency flattened.
The system became boring.

And boring is the highest compliment in system design.

The Real “What If” Question

The real question isn’t:

“What if cache becomes slower than DB?”

The real question is:

“What assumptions am I no longer questioning?”

Because systems age.
Traffic changes.
Access patterns shift.

And cache, if left unquestioned, slowly turns from an optimization into a liability.

It usually starts with a feeling, not a number.

Someone opens the app and says,
“Why does this feel slow?”

Not broken. Not failing.
Just… slow.

A few milliseconds of hesitation. A tiny pause. Enough to break the illusion.

So the requirement gets written:

Every user, everywhere, should get a response in under 50 milliseconds.

It sounds ambitious. It sounds premium.
It sounds like something great products should have.

What it really is… is a declaration of war against reality.

The World Is Bigger Than Your Backend

Most software begins life in one place.

One cloud region.
One database.
One comforting sense of control.

From that setup, everything looks fast—until the first real users arrive.

A tap in New York feels instant.
A tap in London feels okay.
A tap in India feels… delayed.
A tap in Australia feels broken.

Nothing changed in the code.

Distance did.

Data doesn’t move instantly. It travels through cables, routers, oceans, and continents. Every kilometer adds a little delay. Stack enough kilometers together, and suddenly your “fast” system feels old.

That’s the moment you understand something no performance chart can teach:

Latency is geography.

You Can’t Pull the World In, So You Push the System Out

At first, teams try the usual tricks.

Optimize queries.
Tune caches.
Upgrade servers.

It helps. A little.

But no amount of optimization can beat physics.

So the architecture shifts.

Instead of one powerful center, the system becomes many small outposts scattered across the globe. Parts of the application move closer to users—not just images and files, but logic, decisions, and sometimes even data.

Now when someone opens the app, the response doesn’t come from “the backend.”

It comes from nearby.

The app stops feeling like it’s on the internet and starts feeling like it’s on the device.

That’s when people say, “Wow, this feels fast.”

The Subtle Trade Nobody Notices

But something quietly changes when data lives everywhere.

Perfect agreement disappears.

Two users in different countries might see slightly different versions of the same thingfor a moment. A count updates later. A feed rearranges itself. A status changes after a second.

And here’s the surprising truth:

This is acceptable.

Most humans don’t experience software as databases.
They experience it as moments.

If the moment feels smooth, they trust the system.
If the moment feels delayed, they don’t.

So systems stop chasing perfection and start chasing perception.

Correct eventually.
Fast immediately.

Time Becomes… Fuzzy

When everything is global, time itself stops being clean.

Which update happened first?
Which server is ahead?
Which region is “right”?

Clocks drift. Networks lag. Messages arrive out of order.

So systems stop relying on exact time and start relying on ordering, versioning, and conflict resolution.

Instead of asking, “What time did this happen?”
They ask, “Which change should win?”

The system becomes less about storing facts
and more about negotiating reality.

When Speed Still Isn’t Enough

Even after spreading across the world, 50ms is still hard.

So modern systems do something bold.

They assume.

They preload data before you ask.
They render screens before confirmation.
They show optimistic results and fix mistakes later.

You click “like,” and the heart fills instantly
even if the server hasn’t replied yet.

If everything works, you feel joy.
If something fails, the system quietly adjusts.

You never see the machinery.
You only feel the smoothness.

At this point, the system isn’t responding to users anymore.

It’s running ahead of them.

The Invisible Complexity

From the outside, the app feels simple. Effortless. Clean.

Inside, it’s anything but.

Logs are scattered across continents.
Errors happen only in certain regions.
Bugs can’t be reproduced easily.
Fixes must be rolled out carefully, slowly, globally.

Every improvement risks breaking something far away.

But that’s the price of instant experiences.

What This Requirement Really Means

“Latency under 50ms worldwide” isn’t a performance goal.

It’s a philosophy.

It means:

• You prioritize feeling over precision

• You design for humans, not machines

• You accept uncertainty to gain speed

Most importantly, it means you understand this truth:

Users don’t remember correctness.
They remember how it felt.

Fast feels smart.
Fast feels reliable.
Fast feels modern.

And that’s why teams chase 50ms not because it’s easy, but because it changes how people experience the product.

The Final Shift

In the end, systems like this stop asking:

“Is the data perfect right now?”

They ask:

“Does this feel instant, alive, and trustworthy?”

When the answer is yes, users stay.

And that’s the real reason latency matters.

When the system was built, it was quiet.

Most requests just read data.
Writes happened, but only when something actually changed.

Because of that, nobody worried too much about writes. Every request was allowed to touch the database. Counters were updated directly. Logs were stored immediately. It all felt safe.

And for a long time, it was.

Slowly, without anyone noticing, the nature of traffic changed.

A page view was no longer just a read.
It updated a counter.
It stored “last seen”.
It recorded activity.

Nothing dramatic. Just a few extra lines of code.

At first, nothing broke.

Latency increased a little, but it was within limits.
CPU stayed high, but not alarming.
Replication lag appeared, then disappeared.

The system didn’t fail.
It just felt heavier.

That’s usually how write-heavy problems start.

The uncomfortable part was that traffic hadn’t grown much.

Same users.
Same requests.

But now every request did more work.
Work that had to be written, locked, replicated, and persisted.

The database was no longer storing state.
It was reacting to behavior.

Scaling helped, but only briefly.

A bigger machine bought time.
More IOPS reduced pressure.

But the feeling came back.

Because the system wasn’t slow due to lack of resources.
It was slow because it was doing too many writes that didn’t really matter.

The turning point wasn’t a new tool or a new database.

It was a simple question asked during a quiet debugging session:

“Why do we need to write this right now?”

Not later.
Not eventually.
Right now.

That question exposed a lot.

Many writes existed only because they were easy to add.
Some data was useful, but not urgent.
Some data was nice to have, but not essential.

Yet all of it was treated as critical.

Once that became clear, the system slowly changed.

Some data stayed in memory longer.
Some updates became async.
Some counters stopped being exact.
Some things stopped being written at all.

Nothing fancy.
Just honest decisions.

And gradually, the system became calm again.

The biggest lesson wasn’t about databases or performance.

It was about mindset.

Read-heavy systems allow you to be careless.
Write-heavy systems don’t.

They force you to respect every write, because every write has a cost.

The truth is, systems rarely stay read-heavy forever.

Products evolve.
Features grow.
Observability increases.

And one day, reads quietly turn into writes.

If your system isn’t ready for that, it won’t crash immediately.

It will just become fragile.

Final thought

A system that works only when writes are rare is not strong it’s lucky.

Design assuming one day, reads will want to write.

Because production always finds a way to make that happen.

Every experienced backend engineer eventually learns this the hard way.

Not from a blog.
Not from a system design interview.
But from a real incident, usually late at night, when everything looks healthy and customers are still getting hurt.

Duplicate events don’t come from bad code.
They come from reality.

Networks retry.
Brokers redeliver.
Services crash between two lines of code.
Timeouts fire even when work succeeded.

None of this is exceptional.
This is the normal operating mode of distributed systems.

When We Were Wrong

The system was simple on paper.

An event comes in.
We process it.
We update the database.
We move on.

There was even a comment in the code:

“This event will only be received once.”

It wasn’t arrogance.
It was just an assumption no one had questioned yet.

Then one deployment later, a consumer restarted mid-flow.
The message broker retried.
The same event came back.

Nothing crashed.
Nothing alerted.

But a refund was issued twice.

Why Preventing Duplicates Doesn’t Work

The first instinct is always the same:

• “Let’s dedupe at the producer”

• “Let’s add retries with care”

• “Let’s ensure exactly-once delivery”

And all of that helps.

But none of it guarantees uniqueness.

Because once your system crosses machines, processes, or networks, you no longer control delivery.

You only control how you react.

Better Systems Ask Better Questions

Junior systems ask:

“How do we make sure this event happens once?”

Senior systems ask:

“What happens if this runs twice?”

That single shift changes everything.

Instead of trusting the world, the system becomes defensive.
Instead of hoping, it becomes resilient.

Idempotency Is Just Common Sense

The fix is rarely complex.

Before doing any irreversible work:

• charging money

• sending emails

• updating state

The system checks:

“Have I already handled this?”

If yes, it exits quietly.
If no, it proceeds and records that fact.

No locks.
No distributed transactions.
No magic.

Just discipline.

This is why payment systems, notification systems, and order pipelines all talk about idempotency keys.

Not because duplicates are rare.
Because they are inevitable.

Why This Feels Boring (and That’s a Good Thing)

There’s no dopamine hit in idempotency.

It doesn’t show up in dashboards.
It doesn’t improve latency.
It doesn’t impress in demos.

But it saves you at 2 AM.
It saves customer trust.
It saves your on call sanity.

Most great system design decisions feel boring in the moment.
They only feel brilliant in hindsight.

The Real Lesson

Duplicate events are not a failure.
They are a signal.

A signal that your system is operating in the real world.

If processing the same event twice breaks your system,
then the system isn’t robust yet
it’s just lucky.

And luck doesn’t scale.

System design oriented article

When a database is small, it quietly supports you.

Everything feels smooth. Pages load fast, APIs respond instantly, and nobody really thinks about what is happening underneath.

You write queries quickly, ship features, and move on. Even if the design is not great, nothing complains. Small data is kind. It hides mistakes and makes you feel confident.

As time goes on, the product grows.
More users arrive, more actions are recorded, more history is stored. Orders, messages, logs, and events keep piling up.

The database grows slowly in the background, without making noise. Since nothing is broken, nobody pays attention to it.

Then one day, the database is 100× bigger, and the mood changes.

Things don’t crash, but they don’t feel right either.

Pages hesitate before loading. APIs feel unpredictable. Sometimes everything is fast, sometimes it is slow, and no one can clearly explain why.

Users start saying the app feels “heavy.” Engineers start checking dashboards more often. The system still works, but it no longer feels comfortable.

The team slowly understands what is happening.
The database is not slow by accident.

It is slow because it is doing too much unnecessary work.

Queries that once touched a few rows now touch millions.

Decisions that felt harmless earlier are now expensive.

The database has not changed the truth has.

Fear enters the workflow.

Migrations feel risky. Large updates are delayed.

Even simple changes are discussed twice. The database becomes critical and fragile at the same time. Everyone depends on it, but no one wants to touch it casually anymore.

Costs rise quietly in the background.

Storage grows. Replicas increase. Memory usage climbs. Bills go up every month. What hurts most is realizing that this money is being spent to support inefficiency, not innovation.

At this point, the team has a choice.
Either keep reacting to problems, or stop and fix the foundation.

How Teams Actually Handle a Database That Grows 100×

The first thing that changes is thinking.
Instead of asking why something is slow today, the team starts asking how it will behave next year.

Data growth becomes part of design, not an afterthought. This mindset shift alone prevents many future problems.

Then the team looks closely at how data is accessed.
Not all data is equal. Most features touch only a small part of the database.

Once indexes are built around real usage instead of guesses, queries stop scanning everything and start behaving efficiently. Performance improves without adding more machines.

Next comes separation of data.
Old records, logs, and history are moved away from active data. Frequently used tables become smaller and faster.

The database feels lighter, not because data was deleted, but because important data is no longer buried under noise.

Caching becomes an obvious step.
The team notices users repeat the same actions again and again. Instead of asking the database every time, results are reused for short periods.

Load drops. Stability improves. The database finally gets room to breathe.

As growth continues, structure increases.
Reads and writes are separated. Replicas handle traffic.

Data is split by users, regions, or time. The system stops being one big room and becomes an organized building. Complexity increases, but chaos decreases.

Most importantly, habits change permanently.
Queries are reviewed. Slow operations are tracked.

New features are built with future data size in mind. “It works now” is no longer acceptable. The question becomes, “Will this still work when data grows again?”

In the end, growth becomes less scary.
Not because data stopped growing, but because the system and the team finally grew up with it.

Follow for a daily system design series.

It’s evening.

You’re standing at a small grocery store.

You scan the QR code, enter the amount, and tap Pay.

For that moment, nothing else matters. You’re staring at the screen, waiting.

If the app responds instantly, you relax.
If it pauses for even a second, your heart skips.
“Did the money go? Should I pay again?”

This is why payment apps must feel instant, no matter where the user is in the world.

Behind the scenes, the biggest enemy isn’t hacking or bugs.
It’s distance.

If your payment request travels from India to a server in another country, it physically takes time. The data crosses oceans.

Even with perfect software, that delay can’t be avoided. And for payments, delay feels like danger.

So real payment systems don’t use one far-away brain.

Instead, they quietly work close to you.

When you tap Pay, your request goes to a server in your own region.

If you’re in India, it stays in India. The system checks your balance, blocks the amount, and gives you a result almost immediately.

That’s why you see “Payment Successful” so fast usually within a blink.

At that moment, your money is safe. The system has already made a local decision.

What happens next is calmer and slower.

After you’ve walked out of the shop, the system starts sharing that payment record with other parts of the world.

This syncing is important for reports, backups, and fraud checks but it doesn’t rush. The user doesn’t wait for it.

Life moves on.

This separation fast local action, slow global syncing is what keeps payments feeling smooth.

There’s also a quiet understanding in the system.

Some things must be perfect instantly, like deducting money.

Other things, like updating transaction history everywhere, can take a little time. Real payment systems know the difference and don’t treat everything with the same urgency.

That’s how they stay fast without being careless.

And when something goes wrong maybe a regional server is slow the system doesn’t panic. Traffic gently moves to the next nearest place.

The user may notice a tiny delay, but the payment still completes.

Trust is protected.

So keeping payment latency under 100ms globally isn’t about magic speed.

It’s about staying close to the user,

giving answers quickly, and doing heavy work quietly in the background.

When the user feels calm, the system has done its job.

I remember the exact time because I’d just convinced myself it was safe to grab dinner. My instant ramen was exactly three minutes from being ready when my phone lit up like a Christmas tree.

Slack notifications.

PagerDuty alerts.

A text from Sarah, our ops lead, that just said: “We have a situation.”

The ramen stayed in the pot, forgotten, while I grabbed my laptop.

When Your Safety Net Has Holes In It

Five minutes later, I was on a call with half the team, still in their pajamas, staring at dashboards that told a story none of us wanted to read.

Payments were completing successfully. The payment gateway was happy. Money was moving exactly as it should. But somehow, our order service had decided to take an unscheduled nap. Orders were stuck in eternal “processing” limbo.

Marcus from DevOps said what we were all thinking: “Let’s just roll it back.”

It sounded so simple. So clean. Roll back the deployment. Revert the database. Pretend the last twenty minutes never happened. We’d done it before. Standard playbook, right?

Then Priya from payments spoke up, her voice quieter than usual.

“Guys... we can’t. There’s already $47,000 in processed transactions.”

The call went silent.

That number wasn’t theoretical. It wasn’t sitting in some pending state. It was real money that had left real bank accounts.

Confirmation emails had already hit inboxes. SMS notifications had already pinged phones. Some people were probably already asleep, trusting that their order was on the way.

You can’t rollback reality.

You can’t unsend a text message.

You can’t make money jump back across the internet into the right accounts.

We were past the point of no return, and we hadn’t even realized we’d crossed it.

The Problem With Perfect Solutions

Here’s what they don’t tell you in architecture reviews and system design interviews: rollback is a lie we tell ourselves to sleep better at night.

It works great on slides. Draw a nice arrow pointing backward. Label it “rollback.” Everyone nods. Problem solved.

But real systems don’t have clean arrows. They have tentacles reaching into payment processors, email services, SMS gateways, inventory systems, third party APIs, and God knows what else. The moment your code touches any of those things, you’ve started a chain reaction you can’t just... undo.

We could roll back our database. Technically, we had the commands ready. But here’s what that would actually do:

• Our system would suddenly claim these orders never existed

• The payment provider would still show successful charges

• Users would see money missing from their accounts with nothing to show for it

• Our customer support team would wake up to hundreds of angry tickets

• We’d have no paper trail of what actually happened

That’s not a rollback. That’s setting your own house on fire and calling it “cleaning.”

The Uncomfortable Middle Ground

For about ninety minutes, our system was in what I can only describe as an “honestly broken” state.

There was no elegant way to describe what was happening. Users had paid money. Those orders existed somewhere in our system, but not everywhere they needed to be. The state machine had gotten wedged halfway through its process, like a drawer that’s neither open nor closed.

I watched Jen from customer support start her shift early, already drafting responses to confused users.

“We’re aware of the issue and working on it”

the most honest thing you can say when you’re currently debugging why thousands of dollars are in limbo.

Mike, our senior engineer who’d been with us for four years, looked physically uncomfortable.

“This feels so messy,”

he said. “The data’s just... wrong right now.”

He was right. It was messy. The system looked like someone had dropped their spaghetti code all over production. But here’s what I realized in that moment: messy doesn’t mean dishonest.

Our system was telling the truth. It was saying:

“Yes, payments happened. No, orders aren’t complete. This is the actual state of reality right now.”

A rollback would’ve been clean, sure. But it would’ve been a lie.

Fixing Forward (The Scary Way)

Around 1:30 AM, we made a decision that felt completely wrong at the time but was the only thing that made sense.

We weren’t going to undo anything. We were going to fix forward.

Sarah pulled up a spreadsheet (yes, a literal spreadsheet – sometimes the simplest tools are the best ones) and started categorizing every affected transaction:

Category A: Payment succeeded, order can still be fulfilled safely → Let it complete
Category B: Payment succeeded, but something’s wrong with the inventory/shipping → Issue immediate refund
Category C: Payment succeeded, user already contacted support and canceled → Refund with apology credit

Every single transaction got logged. Every action we took got recorded with a timestamp and a reason.

We didn’t delete a single row of “bad” data. Instead, we added more data corrections, refunds, manual overrides, support notes.

The database got bigger and messier, not cleaner.

Priya wrote a script that sent personalized emails to every affected user. Not a template.

Real explanations:

“Your payment went through, but our system had a hiccup. Here’s what we’re doing about YOUR specific order.”

By 4 AM, we had processed every single transaction. By 6 AM, most users had either received their order confirmation or their refund.

By 9 AM, when the East Coast woke up, the only evidence of the disaster was a stack of detailed logs and a few support tickets from people checking on refund timing.

What We Learned From Looking Stupid

Here’s the thing nobody tells you about production incidents: your system is going to look really stupid sometimes.

For a solid two hours, if you’d looked at our order dashboard, you would’ve seen chaos. Inconsistent states. Weird statuses.

Orders that were both “completed” and “refunded” depending on which table you looked at.

It looked like amateur hour. Like we didn’t know what we were doing.

But we did know what we were doing. We were choosing truth over cleanliness.

We were choosing to preserve the actual history of what happened rather than trying to create a fiction where nothing went wrong.

Three weeks later, a user emailed asking about a charge from “that night things broke.”

We could tell them exactly what happened to their specific transaction, down to the second. We could show them the refund, the reason code, the support note, everything.

If we’d rolled back? We’d have had to guess. Or worse, we’d have to tell them their payment never existed even though their bank statement clearly showed otherwise.

The Part Where I Sound Like I’m In Therapy

That night fundamentally changed how I, Himanshu, think about building systems.

I used to design for the happy path and treat failures as exceptional cases to be eliminated.

Now I design for failure as the default state. Success is the exception we’re grateful for.

I used to think good systems were systems that never broke.

Now I think good systems are systems that break gracefully, leave clear trails, and can explain themselves when someone asks “what the hell happened?”

I used to obsess over consistency making sure every part of the system agreed on the current state.

Now I obsess over convergence making sure that even if parts of the system disagree temporarily, they eventually reach a truthful state.

The messy middle isn’t a bug. It’s a feature. It’s reality teaching your system to be honest.

What Changed After That Night

We didn’t just patch the bug and move on. We rebuilt our entire approach to failures:

• We added “compensation transactions” everywhere. For every action that touches the outside world, we now have a documented way to undo its effects, not the action itself. Can’t unsend an email? Fine send a correction. Can’t uncharge a card? Fine issue a refund.

• We stopped deleting data. Ever. Now we only mark things as “superseded” or “corrected.” Every mistake stays in the database with a paper trail of how it was fixed. Our database is uglier, sure, but it’s honest.

• We made “reconciliation” a first class feature. Every night, a job runs to compare what our system thinks happened with what actually happened in payment gateways, email services, and shipping systems. Discrepancies get flagged beforeusers notice.

• We wrote runbooks for “unfixable” scenarios. What do you do when you literally cannot rollback? Not theoretical ones actual step-by-step guides for the panicking engineer at 2 AM.

• We got comfortable with temporary inconsistency. Not all inconsistency we still care about correctness. But we stopped treating brief moments of weirdness as catastrophic failures. Sometimes the system needs a minute to figure itself out.

The Truth About Production Systems

You know what the real difference between a junior engineer and a senior engineer is?

Junior engineers believe rollback will save them.
Senior engineers know that rollback is just another name for “I hope we didn’t affect anything yet.”

Real production systems are messy, sprawling things with tentacles in places you didn’t even know existed.

They send emails.

They charge credit cards.

They update inventory in warehouses.

They trigger workflows in other people’s systems.

They make promises to users in push notifications.

Once any of that happens, you’re not in control anymore. You can’t rewind. You can only move forward and try to do the right thing with the mess you’ve created.

That’s not a failure of engineering. That’s just reality.

The best systems aren’t the ones that never break.

They’re the ones that break in ways you can recover from. That leave trails you can follow. That admit when they’re wrong and correct themselves transparently.

What I’d Tell My Past Self

If I could go back to that Thursday evening, to the moment right before I started cooking my ramen, I wouldn’t tell myself to prevent the incident. I couldn’t. Bugs happen. Systems fail. That’s life.

But I would tell myself this:

Don’t panic when rollback isn’t an option. It’s not the only option. It was never the only option.

Trust your logs. Trust your data. Trust that you can fix things forward even when you can’t undo them backward.

And most importantly: a system that honestly reflects its broken state is infinitely better than a system that lies about being fine.

That night, our system looked stupid. But it survived. And it earned the trust of every user we managed to make whole again.

That’s what matters.

by the way, was completely inedible by the time I remembered it. Overcooked noodles, cold broth. I ate it anyway, at 5 AM, tasting like cardboard and small victories. Best meal I’d had all week.

I don’t wake up early. I don’t have a study timetable. I don’t drink green tea and meditate before coding.I wake up around 12PM. Sometimes 1.
I scroll through Instagram, eat leftover food, and chill.

But you know what?
Every Saturday, I still revise DSA.

No LeetCode. No Codeforces.
Just me, my laptop, and all the messy code I’ve written over the past year stored in random folders with names like final_dp, graph_done, dp_final_latest_final.

That’s my system. And weirdly… it works.

The Start: Lazy, Slow, But Intentional

It’s never a “productive morning.” There is no such thing.
I usually roll out of bed, brush, scroll memes, maybe watch some cricket or random YouTube junk.

At some point usually after 2PM I tell myself:

Bas 1–2 ghante. Open the laptop. Just check old code. That’s it.

I don’t think about solving new questions or doing anything big. I just want to revisit what I’ve already done.

That small mental trick changes everything. I’m not grinding. I’m just… catching up with myself.

The Setup: My Laptop Is My Only Source

No platforms. No bookmarks. No paid courses.

Everything I need is already sitting inside my laptop.

Folders like:

• BinarySearch

• Tree Problems

• Recursion_Important

• Heap

• Sliding Window
  (also let’s not forget: Misc, Unsorted, and Temp, which have the best gems 😅)

Some code files are clean. Some are garbage. Some have TODOs I never finished.
But I love this mess it’s mine. It’s my journey. My struggle. My growth.

The Routine: Re-Solve Without Looking

I randomly pick 3–4 files. Let’s say:

• isValidBST.java

• minWindowSubstring.java

• countIslands.java

I open a new file beside it, and try to solve from scratch. No looking at the old one.

I struggle sometimes. Blank out. Forget what I did last time.

But slowly… it comes back.

And when it doesn’t I open my old code, line by line, and laugh:

“Aree yaar, I used a visited matrix here!”
“WTH is this variable name I used? temp_ans_final2??”
“Ohhh I used stack here. Smart past-me.”

It feels like I’m having a chat with my old self.
That’s my revision. That’s how I learn.

I Talk to Myself While Coding

Sounds mad, I know. But I do this all the time.

“Okay, so I’ll start with a base case…”
“Yeh toh recursion lag raha hai…”
“Wait… should I sort first or use HashMap?”

I just say whatever comes to mind like I’m explaining it to someone sitting next to me.

And somehow, that habit makes things stick more.

You don’t need notes if you talk to yourself.

Sometimes I Clean Up Old Code

This is optional, but fun.

Sometimes I open an old brute-force file and rewrite it better:

• Rename the ugly variables

• Add comments

• Remove unnecessary loops

• Add a few test cases

It feels good. It’s like refactoring your past.

You read your old self’s code, and silently whisper:

“Nice try, bro. But let me fix this now.”

I Stop After 2 Hours. Always.

By 4 or 5PM, I shut the laptop.

Not because I’m tired. But because I don’t want to hate DSA.
I want to look forward to it every Saturday. So I stop before it drains me.

Sometimes I revise 5 problems. Sometimes just 1.
Sometimes I get nothing right and that’s fine too.

The win is: I showed up. I reconnected with my old self.

That’s more valuable than solving 20 new questions blindly.

What I Don’t Do (And Never Will)

• I don’t wake up early

• I don’t maintain notes or sheets

• I don’t use LeetCode for revision

• I don’t do topic-wise grinding

• I don’t have a study plan

I just open my laptop every Saturday…
…go through my own messy code…
…and try to understand myself better.

That’s my DSA revision.

Final Thoughts (From One Lazy Coder to Another)

If you’re someone who wakes up late, feels guilty, avoids DSA during the week but still wants to improve

I see you.
You’re not alone.
You’re not behind.

Just start with your own code.
Forget fancy platforms.
Forget online streaks.
Forget perfect productivity.

Your past code is the best teacher you’ll ever have.

And if you revise it even once a week you’re doing way more than you think.

Saturday doesn’t have to be perfect.
It just needs to be yours.

Master Software Engineering Interiews:

Master Software Engineer Interview

I had 1.5 years of experience and still felt lost during interviews, confused in meetings, and unsure about my skills.
I didn’t need a course. I didn’t need luck.
I just needed consistency.
So I gave myself 2 hours a day. For 31 days.
And that small decision changed everything.

Why I Started This Challenge

Let’s be honest most of us know what to do.

We know DSA is important.We know System Design matters.We know interviews test these.But we don’t do it.

Why?Because we’re “too busy”, or “we’ll start next week”, or “I’m not ready yet.”

In January 2024, I had 1.5+ years of experience as a backend developer.
And I realized something painful:

I can build APIs, but I can’t explain them.
I can write code, but I don’t understand the deeper design.
And every time someone said “Let’s make this scalable” I’d nod… and secretly Google later.

I was tired of faking confidence.
So I said: Enough.
Let’s get real.

The Simple Rule:

For the next 31 days2 hours every day. No excuses.

• 1 hour of DSA

• 1 hour of System Design

No targets. No pressure. Just show up. Daily.

Week 1: The Start Is Always the Hardest

I still remember Day 1.

I opened LeetCode. Picked an Easy problem.It took me 40 minutes.

Then I moved to System Design and tried to learn Rate Limiting.
Heard terms like Token Bucket, LRU, TTL…
I was lost within 15 minutes.

But you know what? I didn’t stop.
I told myself:

You’re not here to prove anything. You’re here to grow.

So I wrote down what I learned.Even if I understood only 20%, I noted it in my words.

The goal wasn’t perfection it was to build the habit.

Week 2: Small Wins That Felt Big

By Day 10, I started seeing patterns.

I solved a prefix sum problem in under 20 minutes.
I looked at another and said “Ah, this is sliding window!”

Not because I became smart overnight,
but because I was showing up daily.

In System Design, I understood how message queues help us go async.
I drew diagrams of how a payment system works using Kafka and Retry Queues.

I stopped saying “I’ve seen this”
and started saying “I understand this”.

And trust me that shift feels amazing.

Week 3: The Challenge Became a Ritual

By Week 3, I didn’t need motivation.

This wasn’t a “challenge” anymore.It became my favorite 2 hours of the day.

Just me, my laptop, a notebook and focus.

I didn’t scroll Instagram. I didn’t switch tabs.
I just solved problems, watched videos, drew systems.

Some days, I studied gRPC vs REST.
Other days, I designed small parts of WhatsApp or Swiggy.

I wasn’t doing this for an interview.I was doing it because I was finally enjoying learning again.

Week 4: I Felt Like a Real Engineer

Something shifted in my thinking.

Earlier, I’d panic in meetings.

Now? I spoke up.
I gave suggestions. I explained retries, DLQs, idempotency.

I wasn’t bluffing anymore.
I actually understood what I was saying.

At work, my teammates noticed.
In my code reviews, I was writing cleaner logic.
Even in casual chats, I could explain why we need a load balancer or cache.

It wasn’t magic.
It was 31 days of focused effort.

What Actually Changed After 31 Days

My improved:

DSA Thinking:
I now recognize patterns binary search, prefix sum, sliding window faster and cleaner.

System Design Clarity:
I can now explain things like Kafka, caching, queues, idempotency, retries without googling or fumbling.

Communication:
I stopped being afraid of technical discussions. I started contributing with confidence.

Confidence:
Not just in interviews but in everyday work. Because now I know I’ve put in the work.

Consistency > Motivation:
The biggest learning?Motivation fades. Discipline builds careers.

My Simple Daily Routine

Want to try it yourself? Here’s what I did:

Hour 1 DSA

• Picked 1–2 LeetCode problems (mostly Medium)

• Solved at my own pace

• Wrote clean code, tested edge cases

• Took notes: what worked, what didn’t

Hour 2 System Design

• Picked 1 concept each day (Cache, gRPC, DB sharding, Kafka, etc.)

• Watched 1–2 short videos

• Drew diagrams, wrote 2–3 line summaries

• Spoke the explanation out loud (yes, I literally talked to myself )

That’s it. Nothing fancy.
Just deep, slow, honest learning.

Final Words (From One Developer to Another)

If you’re reading this and you:

• Feel stuck at 1–2 years of experience

• Keep failing interviews despite “knowing stuff”

• Feel lost in real system design conversations

• Want to grow, but don’t know where to start…

Let me tell you this:

You don’t need 10 hours a day.
You don’t need another fancy course.
You don’t need to wait for the “perfect time”.

You need 2 hours.
Every day.
For 31 days.

Do this for yourself.Quietly.Consistently.And see the difference.

It worked for me. In 2024.
With just 1.5 years of experience.
And no magic at all.

It can work for you too.

NOTE: A tenant is one organization (or business) using the same application, but with its own data, configuration, and settings, isolated from others.

When we started, our payment system had one gateway.

That wasn’t a shortcut.
It was a reasonable decision for the scale we were at.

Payments worked. Users paid. Finance reconciled. Everyone moved on.

Then we onboarded Citi Bank.

Around 8 lakh users, real banking traffic, real pressure.

And suddenly, that one gateway stopped feeling “simple” and started feeling dangerous.

The System Was Holding But Only Just

During the first few weeks of Citi testing, nothing outright failed.

That worried me more than an outage would have.

Payment success rates dipped during peak hours.
Latency crept up slowly.
Retries increased, but quietly.

The gateway wasn’t down.

It was tired.

And tired systems fail when you least expect them to.

The Meeting Where Everything Changed

We scheduled a meeting.

It was supposed to be short.

Just me, my team, my manager, the architect, and the tech lead.
The agenda said: “Citi payment scaling discussion.”

No one walked in with a solution.

We talked about:

• Adding more servers

• Increasing gateway limits

• Optimizing retries

• Better monitoring

All good ideas. None of them felt enough.

At some point, the room went quiet.

And I said something without fully knowing how it would land.

Why Can’t One Tenant Have Multiple Active Gateways?”

I remember hesitating before saying it.

Then I said:

“What if one tenant doesn’t depend on one gateway at all?
What if a tenant can have multiple gateways, and the system decides which one to use?”

The room paused.

The architect leaned back.
The tech lead stopped typing.
My manager asked, “Go on.”

That was the moment I knew this wasn’t just a random thought.

Why That Idea Made Sense to Me

I wasn’t thinking about architecture patterns.

I was thinking about reality.

Gateways don’t fail cleanly.
They slow down.
They partially fail.
They struggle under load.

So I explained my thinking:

• Citi’s traffic is too large for one gateway

• Even two gateways aren’t enough if routing is static

• Load matters as much as availability

• Failover is reactive; load-based routing is proactive

What we needed wasn’t a backup.

We needed choice at runtime.

How the Architecture Team Reacted And What Each Question Really Meant

The architect didn’t push back on the idea itself. Instead, they did what good architects always do they started pulling on the weak threads.

Each question they asked wasn’t meant to reject the idea. It was meant to test whether it could survive reality.

How would routing work?

This was the first and most important question, because routing is the heart of the idea. If routing failed, everything else would collapse with it.

Himanshu: I explained that the biggest mistake we were making was routing based on configuration instead of conditions. Until now, we treated routing as something static: a primary gateway, then a fallback. That works only when traffic is low and gateways behave perfectly.

At Citi’s scale, that assumption was no longer true. Gateways didn’t fail cleanly. They slowed down, partially failed, or throttled requests. So instead of routing once at deploy time, we needed routing to happen at request time.

I told them that every payment request should be evaluated against the current state of all allowed gateways for that tenant. The system should look at which gateway was under the least load, which one was responding consistently, and which one wasn’t showing signs of stress. The goal wasn’t to be smart it was to avoid obviously bad choices.

That’s when routing stopped being a configuration problem and became a runtime decision problem.

What metrics would we trust?

This question went deeper than it sounded. Metrics can lie if you don’t understand their context, and the architect was right to be cautious.

Himanshu: I explained that trusting a single metric like success rate or latency was dangerous. A gateway could show a high success rate but still be struggling under increasing load. Another might have slightly higher latency but be far more stable.

So instead of trusting one number, we needed to trust patterns.

I suggested combining short-term success trends with latency movement and retry behavior. Not over hours or days, but over recent windows just enough to reflect current stress. If retries started increasing or timeouts clustered together, that was a stronger signal than any dashboard metric.

I made it clear that we weren’t trying to predict the future. We were just reacting faster than users could notice problems.

That’s when metrics stopped being reporting tools and started becoming decision inputs.

How do we avoid double charges?

This question changed the tone of the room, because this is where payment systems usually fail.

I didn’t jump to an answer right away, because double charges aren’t caused by one mistake they’re caused by ambiguity.

I explained that the core problem wasn’t retries themselves. It was unclear ownership of transaction state. When multiple gateways are involved, the system must be absolutely certain about what has already happened and what hasn’t.

The solution was strict idempotency. Every payment attempt needed a unique, globally consistent transaction ID that followed the request across retries and gateways. A retry wasn’t a new payment it was the same payment continuing its journey.

I also explained that the orchestration layer had to own the state machine. Gateways should be treated as execution engines, not sources of truth. The system decides when to retry, when to switch gateways, and when to stop.

Once that was clear, the fear around double charges didn’t disappear but it became manageable.

How do we keep this tenant-specific?

This question grounded the discussion back in reality. Not all tenants behave like Citi, and we couldn’t design a system that forced everyone into the same rules.

I explained that tenants shouldn’t just configure gateways they should configure intent.

Each tenant should be able to define:

• Which gateways are allowed

• How aggressive retries can be

• Whether reliability or cost matters more

• Any regional or compliance constraints

The routing logic would operate only within those boundaries. The system could make intelligent decisions, but it could never violate tenant rules.

That separation was critical. It meant we weren’t building a one-size-fits-all payment engine. We were building a flexible system that respected each tenant’s reality.

One Real Problem We Had to Solve

Once we agreed on supporting multiple gateways for a single tenant, a practical problem surfaced very quickly.

Not all payment gateways behave the same way.

Some gateways, like Razorpay or Stripe, expect you to create an order firstand then initiate the payment. Others, like PayU, work differently. They don’t have a clear “order creation” step at all they directly initiate the payment.

This inconsistency became a real issue.

If the system itself was choosing gateways dynamically, we couldn’t depend on gateway-specific order concepts. We needed a single, consistent source of truth, regardless of which gateway handled the payment.

Why We Created Our Own Order Table

So we made a simple but important decision.

Instead of relying on gateway orders, we created our own internal order table.

Every payment request first created an internal order in our system. That order:

• Had a unique ID

• Tracked payment state end to end

• Stayed consistent across retries and gateway switches

Gateways became execution layers, not owners of state.

Whether the payment went through Razorpay, Stripe, or PayU, the system always knew exactly which order it belonged to and what stage it was in.

That one decision removed a lot of ambiguity and made multi-gateway routing actually safe to run in production.

How We Identified the Gateway for Each Payment

Along with creating our own internal order, we also introduced a payment gateway ID into the flow.

Every time the orchestration layer selected a gateway, that decision was stored against the order using a gateway ID. This sounds small, but it mattered a lot. It meant we always knew which gateway was chosen, why it was chosen, and where the payment was actually processed.

When callbacks came back from gateways, we didn’t have to guess or infer anything. We simply used the gateway ID to route the response to the correct handler and update the same internal order. This made retries safer, debugging easier, and reconciliation much cleaner.

It also helped us later when we needed reports, audits, or tenant-level analysis because the system never lost track of which gateway handled which payment.

Sometimes, clarity comes from something as simple as writing the decision down and never letting it get lost.

This is a simplified payment data model created only for easy understanding. In real production systems, there are many more tables and attributes such as fees, dues, refunds, settlements, reconciliation, and ledgers, which are intentionally omitted here.

An Order represents the business request to collect money. One order can have multiple payment attempts. Each Payment is an attempt to pay an order using a selected Gateway. A payment can create multiple Transactions, which store the actual gateway responses like success, failure, or retries.

The Gateway table defines available payment providers per tenant. The Mandate table supports recurring or auto-debit payments by linking a user to a gateway for a valid time period. Overall flow is: Order → Payment → Transaction, with Gateway and Mandate supporting how payments are processed.

When the Questions Changed the Direction of the Conversation

At some point, the architect said something that stuck with me:

“So this isn’t really about adding more gateways.
It’s about moving decision-making into the system.”

That’s when the discussion shifted completely.

We stopped debating whether the idea was risky.
We started discussing how to make it reliable.

That’s the moment I knew the idea had landed.

What That Moment Taught Me

That meeting wasn’t about convincing anyone.

It was about explaining reality clearly enough that the architecture spoke for itself.

Each question wasn’t a blocker it was a filter. And once the idea passed those filters, it became stronger.

That’s how real systems are built.

From Idea to Design: One Tenant, Multiple Gateways

We agreed on a direction:

• A tenant could configure up to six payment gateways (Razorpay, Stripe, PayU, Paytm, Cashfree, and Adyen)

• All gateways could be active

• Selection would be dynamic, not static

• Decisions would be made by the system, not hardcoded rules

This wasn’t over-engineering.

It was designing for Citi’s reality.

Why We Built an Orchestration Layer

Instead of spreading gateway logic everywhere, we introduced a single decision point.

An orchestration layer that:

• Knew nothing about UI

• Knew nothing about loans

• Knew nothing about business flows

It only answered one question:

“Which gateway should handle this payment right now?”

To answer that, it looked at:

• Current load

• Recent success rates

• Timeout patterns

• Tenant-level preferences

Simple signals. Real signals.

Microservices Happened Naturally After That

Once payment decisions were centralized, it became clear that payments didn’t really belong inside the rest of the system anymore.

They had their own traffic spikes, their own failure patterns, and their own operational risks. When payments slowed down or retried, it shouldn’t affect checkout, loans, or anything else.

Separating payments into their own service just felt like the right move. It allowed us to scale payments without scaling everything else, monitor them without noise, and contain failures instead of letting them spread.

It wasn’t about following trends or “doing microservices right.”
It was simply the safest way to handle something as sensitive as payments.

What Changed After We Implemented This

When Citi traffic ramped up:

• Load spread naturally across gateways

• No single gateway became a bottleneck

• Failures were contained and expected

Most importantly, the system behaved calmly under stress.

That’s when I knew the idea was right.

What That Meeting Meant to Me Personally

That meeting taught me something important.

Architecture isn’t about big diagrams.

It’s about:

• Listening to the system

• Understanding real-world behavior

• Speaking up when something doesn’t feel right

That one sentence “Why can’t one tenant have multiple gateways?” changed how our payments worked.

And it changed how I saw ownership.

Final Thought

We didn’t start with six gateways.

We earned them.

Citi forced us to think bigger.
That meeting gave the idea a voice.
The architecture made it real.

And once the system could handle Citi’s scale, everything else became easier.

If you understand this, you will be able to build any payment system whether for an interview or for your product easily. Soon, I will share the complete flow of this system.

Just amazing, right? Think about it …’

‘Kya system hoga Yrr!’

Handling such massive scale, real-time processing, and near-zero downtime is only possible with world class architecture, deep engineering, and relentless optimization.

To truly understand how this works, you need to read this entire article first.”

A real-life story, with the full high-level design hidden inside it

My name is Himanshu. I write code for a living, but like most people in India, I don’t think about code when I pay for chai. I think about warmth, taste, and whether I’m late.

That night, the chai was ₹15. I scanned the QR, entered my PIN, heard the familiar sound, and stepped aside. The chaiwala didn’t look at the phone. He didn’t ask me to wait. He trusted the system more than he trusted cash.

Beep: “Paytm par 15 rupaye prapt hue, Davy aur Sajno”

That silence after the beep is not accidental.

It is the sound of a system that has already answered a thousand hard questions before I ever touched my phone.

What Really Started When I Tapped “Pay”

When I pressed “Pay”, my phone did not try to move money. Phones are unreliable. Networks disappear. Batteries die. Humans make mistakes.

Instead, my phone created a request. A carefully structured message that says who I am, who I’m paying, how much, and under what rules.

That request went to my bank through the app I was using. The app is not the system. It is just a door.

My bank checked that this request was real, that it was signed correctly, that it came from me, and that it deserved to enter the national payment highway.

From there, the request reached the system operated by National Payments Corporation of India, a system designed not like a startup product, but like electricity or roads something that must always work and must never surprise people.

The High Level Design ( NPCI x UPI )

UPI’s design starts with one quiet decision: no single party should own the whole flow.

That is why the system is broken into clear components, each with a limited responsibility, each distrustful of the others, each designed to fail safely.

At the top are the client apps. These are replaceable by design. They are allowed to innovate, experiment, and even fail. The core system does not depend on them emotionally or technically.

Behind the apps are PSP banks. These banks act as the first gatekeepers. They validate requests, enforce basic limits, and ensure that only clean, authenticated traffic reaches the core.

This protects the rest of the system from chaos at the edges.

At the center is the UPI Switch operated by NPCI. This is the heart, but not the brain. It does not store money.

It does not remember users. It only routes, validates, times, and enforces rules.

On the other side are the receiver banks, where money actually moves. These banks own accounts, balances, and final truth.

Around all of this exist invisible but critical systems: fraud detection, monitoring, logging, reconciliation, and settlement.

Each part knows its job. Nothing tries to do everything.

Why the Payment System(NPCI) Doesn’t Store User Data

One of the most important architectural choices in UPI is that the core switch is stateless.

This means when my ₹15 request arrived, the system did not care who I was yesterday. It did not check my previous transaction. It did not store context.

Everything it needed was already inside the request.

This single idea allows the system to scale horizontally. When traffic increases, more identical servers are added. Any server can process any request. No server is special. No server is irreplaceable.

That is how the system survives 100,000 or more requests every second without panic.

State is the enemy of scale. UPI keeps state at the edges, not at the core.

How a UPI Payment Works Step by Step

My request enters the UPI switch. The switch immediately validates its structure and identity. It reads the UPI ID and understands which bank should receive it. It starts a timer, because time matters more than completeness.

The request is forwarded to the receiver’s bank. That bank checks the account, applies fraud rules, ensures balance is sufficient, and performs an atomic operation where money is debited and credited as one indivisible step.

If anything goes wrong or takes too long, the bank responds clearly. No guessing. No waiting.

That response flows back through NPCI to my bank and finally to my phone.

The entire system agrees on one thing: there must be exactly one outcome.

Why You Rarely See “Processing”

UPI does not like uncertainty. It is designed to avoid “maybe”.

Every component operates under strict timeouts. If a response does not come back fast enough, the transaction is treated as failed. This is not cruelty. This is safety.

If my app retries because my network drops, the system recognizes the transaction by its unique ID. It knows this is not a new request. It does not move money again. It simply repeats the answer.

This property, called idempotency, is not a feature. It is a moral requirement in payment systems.

What Continues Working After You Pay

Once my phone shows success and I walk away with chai in hand, the system keeps working.

Audit logs are written so every step can be replayed. Metrics are updated so engineers can see health in real time. Fraud models observe behavior patterns quietly. Reconciliation records are prepared for the night.

All of this happens asynchronously, separated from the user path. My experience is protected from the weight of the system.

This separation is why UPI feels light even though it carries the financial weight of a nation.

Why Night Time Checks Matter

At night, when the city sleeps, systems compare notes.

Banks and NPCI reconcile millions of transactions. Every rupee must be accounted for. Every mismatch must be investigated. There is no hiding, no rounding, no excuses.

This is slow, careful, human work supported by machines. This is where trust is renewed daily.

What I Learned from This System

UPI changed how I understand good engineering.

Good systems are not loud. They are calm.
They do not show off. They remove fear.
They do not demand understanding. They earn trust.

The chaiwala doesn’t know about stateless services, distributed switches, or reconciliation engines.

He only knows the sound.

And that sound is enough.

Why NPCI Scales to 600 Million Daily Transactions

So how does NPCI handle 600 million users daily?
How do 100K real requests come every second without a break?

NPCI is able to handle around 600 million users daily because it does not remember users at all. It does not store who paid earlier, who will pay later, or what someone paid a second ago.

For NPCI, there is no “Himanshu” or “chai payment history”. There is only a payment request that exists right now.

When a UPI payment comes, NPCI only checks a few things: is the request valid, which bank should get it, and did the bank respond in time.

Once the payment succeeds or fails, NPCI forgets the user immediately. Nothing personal is stored at the core system.

Because of this, every payment request is independent. One request does not depend on another.

That means any server can handle any request. If traffic increases, NPCI simply adds more servers.

This is why the system scales horizontally without slowing down.

That is how 100,000 real requests can come every second without a break.

NPCI is not loading past data, not opening user sessions, and not holding memory. Requests just come in, get processed, and go out like a flowing river.

NPCI only remembers one thing carefully: the transaction ID. This is to avoid double payment. It forgets the user, but it never confuses the transaction.

In one line:
NPCI scales because it processes the present and forgets the rest.

Final Thought

That ₹15 chai was not small.

It passed through apps, banks, switches, networks, servers, logs, and humans and returned certainty in less than a second.

UPI did not just solve payments.

It solved confidence, at population scale.

And when engineering reaches that level, it disappears into life.

Just like the steam from that cup of chai.

11:57 PM in Bengaluru.
The Flipkart war room feels like a launchpad. Dashboards glow in the dim light traffic per second, inventory count

ers, Kafka lags, CPU & memory graphs. Some lines are green, a few turn amber, one or two flirt with red.

Coffee cups clutter the table. Engineers sit quietly, not coding anymore. Code freeze happened days ago. Tonight is not about writing code, it’s about watching systems perform.

Someone whispers: “T-minus 3 minutes.”

At exactly 12:00:00, the Flipkart banner flips to Big Billion Sale. Push notifications blast across India. Crores of taps per second slam into the system.

And somewhere in Pune, Himanshu taps “Buy Now.”
This is the story of what happens next.

The Mall in the Cloud

“Imagine Flipkart as a giant mall, but it exists in the cloud.”

• CDN (Content Delivery Network) → Like billboards all over India. Instead of fetching banners and scripts from Bengaluru, you read them from a server near your house. That’s why the homepage loads instantly.

• WAF (Web Application Firewall) + API Gateway → Like the mall’s main gate and security guards. They filter bots, block abusive traffic, and decide if Himanshu’s request is legitimate.

• Load Balancers → The traffic cops. A global LB sends Himanshu to the nearest regional data center (Mumbai, Chennai, Delhi). Then a regional LB routes him to a healthy server pod.

• Microservices → Inside the mall, each shop does one job: catalog, cart, payments, inventory, orders, notifications. They run inside Kubernetes, wrapped in a service mesh so they can talk securely.

• Kafka → The conveyor belt between shops. Instead of yelling, services drop messages on Kafka and others pick them up at their own pace.

• Databases → Stockrooms. Strict vaults (MySQL/Postgres) for money; wide bins (Cassandra/ElasticSearch) for catalog and search; fast drawers (Redis) for ultra-hot data like carts and stock.

• Observability → CCTV everywhere: Prometheus for metrics, Grafana dashboards, ELK logs, tracing. If one corridor jams, alerts ring instantly.

Himanshu’s Midnight Order, The Flow

Let’s trace Himanshu’s iPhone order in real-time.

1. Homepage

When Himanshu opens the app:

• 80% of the load is served by CDNs: banners, images, JS.

• Dynamic API calls go through the API Gateway, where a rate limiterensures one user/device/IP doesn’t flood the system. Suspicious ones get challenged.

• Feature flags (simple config switches) decide if Himanshu sees an experimental banner or a bank-specific promo.

The secret: by handling most stuff at the edge, core servers are shielded.

2. Search

Himanshu types “iPhone 15 Pro.”

• The query hits the Search microservice, backed by ElasticSearch. Optimized indexes return results in ~50 ms.

• Each result needs prices/promos. Instead of hitting MySQL every time, the Pricing service pulls from Redis(periodically synced with MySQL).

Why not just MySQL? Because at this scale, even reading 10 million rows/sec would melt it. Redis handles micro-updates and insane read volume.

3. Product Page

Himanshu taps the iPhone listing.

• Catalog service fetches structured product details from Cassandra + ElasticSearch.

• Inventory service checks Redis for stock count.

• That little line “Only 3 left in stock” is not fake. It’s updated in near real-time by Kafka events flowing from warehouse DBs.

This separation (catalog vs. inventory) avoids bottlenecks. Catalog = descriptive data (read-heavy). Inventory = small, hot, transactional counters.

4. Add to Cart

Himanshu adds the iPhone to his cart.

• Redis is updated immediately (primary source for speed).

• An async background write persists the cart in Cassandra for durability.

Why this hybrid? Redis alone is too risky (data lost if memory crashes). Cassandra ensures persistence without slowing the hot path.

5. Checkout

The Order Orchestrator takes charge. Think of it as a conductor ensuring the orchestra plays in sync:

1. Calls Inventory to reserve the item.

2. Confirms address & delivery slot.

3. Passes to Payment Orchestrator.

Every microservice call is logged with a correlation ID. If something goes wrong, engineers can trace Himanshu’s journey across dozens of services.

6. Reservation

This is the heart of the system. Here’s how Flipkart avoids overselling:

• Acquire a distributed lock (via Redis/ZooKeeper) on the iPhone SKU.

• Check the stock counter in Redis.

• If enough units exist → decrement atomically.

• Create a reservation TTL (5 minutes) → Himanshu has this time to finish payment.

• Kafka event informs other systems: “Item reserved.”

• Release the lock.

If Himanshu fails to pay, stock auto-releases when TTL expires. That’s why 1000 iPhones don’t become 2000 oversold orders.

7. Payments

Payments are unpredictable. Banks can be slow. Gateways can timeout.

The Payment Orchestrator is built for this:

• Generates an idempotency key so retries don’t double-charge.

• Chooses the best route (UPI/Card/Wallet) based on live success scores.

• Sets hard timeouts (e.g., UPI = 15 sec). If slow, retries on another gateway.

• Every transaction is logged in an append-only ledger DB → audit-proof, tamper-proof.

To Himanshu, this feels like: “Processing your payment…” even though in the backend, multiple retries and failovers may be happening.

8. Order Commit

If payment succeeds:

• Order service writes the final record into MySQL/Postgres (ACID-safe).

• Kafka events trigger:

• Notifications (SMS, email, push).

• Ekart logistics (nearest warehouse starts packing).

If payment fails? Himanshu sees a clear message not silence:

• “Processing…” if bank callback is late.

• “Payment failed, stock restored.” if reservation expires.

Predictable, honest UX > Silent failure.

Flash Sale Mechanics

n a flash sale, lakhs of people hit “Buy Now” at the same second. Without guardrails, servers crash, stock oversells, and nobody is happy.

To stop this chaos, Flipkart uses five key tricks that work together.

First comes the token bucket at the gateway. Think of it like giving every user a handful of tokens maybe 10 per second.

Every time you send a request (like add to cart or check stock), you spend a token. If you run out, you wait.

This stops a single user or bot from hammering the system with thousands of requests while still letting normal customers through smoothly.

Next is the waiting room (pre-queue). Imagine a stadium with only a few gates. If everyone pushes in at once, it’s chaos. Instead, the crowd waits outside, and security lets people in small waves.

That’s what happens here. You may see a “You’re in line” message for a few seconds, while the backend processes earlier requests. This way the system doesn’t collapse under sudden spikes, and users feel things are fair.

Now, the hot key problem. In Redis (or any cache), each product has a counter. During a flash sale, one product say the iPhone is requested millions of times.

That counter becomes a “hotspot” and one Redis server can melt under the load. The fix is sharding: split that iPhone counter into multiple smaller counters (like sku:iphone:a, sku:iphone:b…).

Updates spread across servers, and the system aggregates them when showing stock. This spreads the heat.

Another big issue is the cache stampede. Suppose a cache key (like the stock count) expires. Suddenly, thousands of requests rush to refresh it from the database at the same time, and the DB collapses.

The trick is single-flight refresh: the first request refreshes the data, and others just wait for that result instead of all hitting the DB.

Add in randomized expiries (jitter) so not everything expires together, and the system stays safe.

Finally, reservation TTLs keep things fair.

When Himanshu clicks Buy Now, the system reserves the item for him but only for a limited time (say 5 minutes). If payment succeeds, the item is his.

If he quits or fails to pay, the timer expires and the stock goes back into the pool. This prevents people from hoarding stock forever without completing the purchase.

Observability: Seeing Problems First

During a flash sale, millions of people are buying at once. If something breaks, you can’t wait for users to complain you need to spot problems instantly.

That’s why engineers watch a few golden signals on dashboards all night.

Golden Signals (What They Watch)

1. Checkout speed → Every checkout should finish fast. Target: under 300 ms for almost all users.

2. Payment success → At least 95% of payments should go through. If it drops, something’s wrong with a bank or gateway.

3. Inventory locks → Reserving stock should take less than 50 ms. If it’s slow, Redis or ZooKeeper is overloaded.

4. Kafka lag → Events (orders, stock updates, payments) should be processed within 2 seconds. If delayed, stock counts and notifications may be wrong.

What They Do When Things Break

• If Bank A fails → router quickly sends less traffic to it and more to other banks.

• If Redis shard is overloaded → hot product counters (like iPhone) are split across multiple Redis keys to spread the load.

• If Kafka is lagging → new consumers are started, or non-critical traffic is paused, so important events catch up.

• Failures don’t shock the team. They expect them.

Midnight Failures

• 12:12 AM → UPI-X latency spikes. Router cuts share 30% → 2%. Wallets take over. Pending orders settle later.

• Hot iPhone SKU overloads Redis → counter is split across 8 shards. TTL jitter avoids synchronized expiries.

• Kafka consumer lag → engineers scale up consumers, backlog clears in minutes.

• Cache stampede → single-flight prevents collapse.

The system bends but never breaks.

In the weeks before Big Billion Day, engineers practice at 3 AM when traffic is low. These are called chaos drills. They break things on purpose:

• Kill a database server.

• Cut off a data center link.

• Pretend a bank goes down.

Every time, they practice how to recover. They write down what worked, update playbooks, and fine-tune alarms. So by the time the real sale night comes, failures don’t surprise anyone they’ve already seen them in practice.

That’s why during BBD night, there’s no panic and no “hero” trying to code last-minute fixes. Instead, it’s calm and methodical. Engineers sit with dashboards open, radios on, and checklists ready.

You’ll hear quiet voices say things like:

• “Cut Bank-X traffic.”

• “Add more Kafka consumers.”

• “Turn off recommendations, keep checkout fast.”

And it works. By sunrise, crores of orders are confirmed, warehouses are buzzing, and the war room finally relaxes. Another Big Billion Sale survived.