Once you reach tens of millions of users and hundreds or thousands of transactions per second, issues start to surface—such as lock contention, replication lag, slow queries, and data no longer fitting into RAM, which leads to frequent disk reads. All of this significantly degrades service quality. While you can always scale vertically by adding more CPU or RAM, that approach has a hard ceiling: vertical scaling isn’t infinite.

Back in 2011, Instagram had just three engineers and over 25 million users. They were running on a single primary PostgreSQL instance, and their breaking point wasn’t storage space—it was RAM. When the dataset stopped fitting into memory, every write started hitting the disk, causing p95/p99 latency to skyrocket. Even adding replicas couldn’t solve the issue.

Pinterest hit the exact same wall with MySQL, but with a twist: they initially tried to scale by throwing different technologies at the problem (Cassandra, MongoDB, Membase). Their takeaway? Every new system introduced a completely new set of failure modes and trade-offs.

Ultimately, they returned to MySQL, but this time partitioned—applying the Database Sharding pattern.

Understanding sharding is fundamental if you work with large-scale systems.

What problem does Sharding solve?

Before resorting to sharding in a relational database, there is a sequence of optimization steps typically taken first. Sharding is the last resort.

• Vertical Scaling: Upgrading CPU, adding RAM, using faster storage hardware (SSD, NVMe), tuning file systems, etc.

• Replicas: Offloading read operations to dedicated instances that consume data from the primary writer via replication. Note that not all reads can go to a replica; replica data is only eventually consistent, meaning there will always be some degree of replication lag or delay.

• Partitioning: Data remains on a single instance but is internally partitioned (often horizontally/vertically depending on scheme, e.g., range partitioning by date). For instance, all July data lands in its own partition, resulting in smaller, local indices and allowing the query engine to apply optimizations like partition pruning and partition-wise joins.

• Sharding: Splitting rows of the same table across independent nodes, also known as horizontal sharding.

Implementing sharding solves the single-writer bottleneck by distributing both write and read workloads across independent database instances, spreading the overall operational volume across multiple distinct nodes.

If you want to dive deeper into system design and patterns, check out SACAViX System Design. It features real system design exercises, technical interview prep, key terminology, and a complete guide explaining 95 must-know patterns

The shard key: The most critical decision

Choosing the partition key (shard key) is the single most important decision when implementing sharding. This choice is fundamental because it directly dictates whether future queries will remain simple or become extraordinarily complex.

A well-chosen shard key should allow for “point queries” (routing directly to a single node) while avoiding what is technically known as Scatter-Gather.

• Single-Shard Queries (Point Queries): The system routes the query directly to a specific node without needing cross-shard joins, making execution simple and efficient.

• Scatter-Gather: There is no way to know which node holds the requested data. The system must query multiple nodes and aggregate the results, drastically increasing complexity and degrading performance.

Example:

If the shard key is user_id and you need a user’s profile information, you only query the specific shard hosting that user—and that’s it:

SELECT * FROM orders WHERE user_id = 456

En cambio, si la clave es user_id, pero por cuestiones del negocio tienes que hacer una consulta como la siguiente, entonces la shard key correcta no sería user_id.

SELECT * FROM orders WHERE status = ‘PENDING’

In this latter case, to find pending orders across all users, you would have to execute the query across every shard and then aggregate the results.

Properties of a good shard key

The core properties to look for when choosing a shard key are:

• High cardinality: Having many distinct values allows for fine-grained distribution across shards. Poor choices include fields like country (only ~200 values) or status (active, inactive). Excellent choices include user_id, customer_id, or account_number.

• Uniform distribution: Both data volume and incoming traffic must be spread as evenly as possible. Otherwise, the system creates hotspots—where a handful of shards absorb almost all the load.

• Alignment with dominant Access Patterns: Your most frequent and critical queries should naturally include the shard key in their WHERE clause.

• Immutability: Once written, a partition key should never be updated or modified.

The cardinal sin of sharding is executing cross-shard queries. If rendering a user’s feed requires querying 4,000 nodes and merging the responses in the application layer, you haven’t solved a scaling problem—you’ve simply traded it for latency and availability issues. Latency becomes tied to the slowest responding shard (tail latency), and if even a single node out of 4,000 goes down, the entire query fails. Always avoid cross-shard operations and cross-shard joins.

How do you determine which shard a record belongs to?

The most common strategies are:

• Range-Based sharding: Users 1 to 1M live on Shard A, 1M to 2M on Shard B, and so on. While simple to reason about and highly efficient for range scans, it creates major hotspots: newly registered users (who are typically the most active) all land on the latest shard.

• Hash-Based sharding: Assigning shards using a function like hash(key)%N. This distributes data uniformly and prevents hotspots. However, using modulo N comes with a severe flaw: if N represents the total number of servers and you add a new one, the modulo result changes for almost every row, forcing a near-total database migration. This is a massive operational headache.

To solve the challenge of scaling from N to N+1 nodes without massive data movement, two popular rebalancing algorithms are often used: Consistent Hashing and Jump Consistent Hash—though these fall outside the scope of this article.

Virtual Shards

Both Instagram and Pinterest solved the challenge of adding new nodes by introducing the concept of virtual shards. Virtual shards break the direct mapping between keys and physical database servers.

Without virtual shards, routing is typically straightforward. For example, with 8 physical servers, you might route like this:

server = user_id % 8

User 31,341 % 8 = 5 then this user resides on server 5.

As the business grows, you add a 9th node and change the modulo to %9. Now, 31,341 % 9 = 6. The user has shifted to a new server. And it’s not just one record: switching from modulo 8 to modulo 9 changes the destination for almost every row in your database.

Adding a server forces a massive database re-shuffling—something practically impossible to pull off in a live production environment. Every time you scale, you alter the mapping function, and changing the mapping function relocates virtually everything.

The solution: Adding an immutable intermediate layer

Instead of mapping the key directly to a physical server, we map it to a virtual shard. This number—for instance, 8,192 virtual shards—is fixed forever and never touched again:

shard_virtual = user_id % 8192      // Fixed. Never change.

The divisor is now a fixed system constant, not the number of physical machines. User 31,341 evaluates to 31,341% 8192 = 6981—and it will evaluate to 6981 today with 8 servers, and tomorrow with 800. The key-to-shard mapping function is immutable.

Now, where does virtual shard 6981 physically live? That is defined by a second lookup table—one that is small, lightweight, and mutable:

shard   0    – 1023  →  mysql-01
shard 1024   – 2047  →  mysql-02
...
shard 6144   – 7167  →  mysql-07 ## <<<<<
shard 7168   – 8191  →  mysql-08

How is this implemented in practice?

Writing the actual code implementation is something any AI assistant can easily handle, so I will focus on the high-level design principles to give you a clear architectural picture.

Once you’ve selected your shard key (or ensured it can be derived from an ID), the first step is defining the virtual sharded mechanism. To do this, you implement a database structure like the following:

CREATE DATABASE IF NOT EXISTS shard_topology;
USE shard_topology;

-- Inventory of physical MySQL servers hosting the shards.
CREATE TABLE physical_server (
  server_id   VARCHAR(32)  NOT NULL,            -- ‘mysql-01’
  host        VARCHAR(255) NOT NULL,            -- ‘mysql-01.internal’
  port        INT          NOT NULL DEFAULT 3306,
  role        ENUM(’PRIMARY’,’REPLICA’) NOT NULL DEFAULT ‘PRIMARY’,
  status      ENUM(’ACTIVE’,’DRAINING’,’OFFLINE’) NOT NULL DEFAULT ‘ACTIVE’,
  updated_at  TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP
                                  ON UPDATE CURRENT_TIMESTAMP,
  PRIMARY KEY (server_id)
) ENGINE=InnoDB;

-- The map where all the magic happens: each of the 8,192 virtual shards
-- points to ONE physical server and ONE database on that server.
CREATE TABLE shard_map (
  virtual_shard SMALLINT UNSIGNED NOT NULL,     -- 0 .. 8191
  server_id     VARCHAR(32) NOT NULL,
  db_name       VARCHAR(64) NOT NULL,           -- ‘shard6981’
  updated_at    TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP
                                  ON UPDATE CURRENT_TIMESTAMP,
  PRIMARY KEY (virtual_shard),
  KEY idx_server (server_id),
  CONSTRAINT fk_shard_server
    FOREIGN KEY (server_id) REFERENCES physical_server(server_id)
) ENGINE=InnoDB;

-- Global version counter: incremented on EVERY reshard operation.
-- The application polls it every few seconds; if it changes, the map is reloaded.
CREATE TABLE topology_version (
  id      TINYINT NOT NULL DEFAULT 1,
  version BIGINT UNSIGNED NOT NULL DEFAULT 1,
  PRIMARY KEY (id),
  CHECK (id = 1)
) ENGINE=InnoDB;
INSERT INTO topology_version (id, version) VALUES (1, 1);

topology_version serves as the cache invalidation mechanism. The application caches the mapping table in memory; to know when to reload it, it simply executes the lightweight query below every few seconds instead of continuously fetching all 8,192 rows.

At the application level, the flow looks like this:

user_id  --(% 8192)-->  virtual_shard  --(shard_map)-->  (server_id, db_name)
                                              |
                              server_id --> HikariDataSource (pool cacheado)
                                              |
                              getConnection() + setCatalog(db_name)

@Component
public class ShardRouter {

    public static final int NUM_VIRTUAL_SHARDS = 8192;

    private final TopologyRepository topology;
    private final ShardProperties props;

    // UN pool por servidor físico. serverId -> DataSource.
    private final Map<String, HikariDataSource> pools = new ConcurrentHashMap<>();
    // El mapa cacheado: virtual_shard -> mapping.  Se reemplaza atómico al recargar.
    private volatile Map<Integer, ShardMapping> shardMap = Map.of();
    private volatile long loadedVersion = -1;

    public ShardRouter(TopologyRepository topology, ShardProperties props) {
        this.topology = topology;
        this.props = props;
    }

    @PostConstruct
    public void init() { reload(); }

    /** Paso 1: user_id -> shard virtual. Determinista, nunca cambia. */
    public int virtualShard(long userId) {
        return (int) (Long.remainderUnsigned(userId, NUM_VIRTUAL_SHARDS));
    }

    /** Ruteo por objeto: el shard viaja embebido en los bits altos del id. */
    public int virtualShardOfObject(long objectId) {
        return (int) ((objectId >>> 46) & 0xFFFF);
    }

    /**
     * Paso 2 y 3: shard virtual -> (servidor, db) -> conexión ya posicionada.
     * El caller usa la Connection y la cierra (vuelve al pool).
     */
    public Connection connectionForUser(long userId) throws SQLException {
        return connectionForShard(virtualShard(userId));
    }

    public Connection connectionForObject(long objectId) throws SQLException {
        return connectionForShard(virtualShardOfObject(objectId));
    }

    private Connection connectionForShard(int vshard) throws SQLException {
        ShardMapping m = shardMap.get(vshard);
        if (m == null) throw new IllegalStateException(”Shard sin mapear: “ + vshard);

        HikariDataSource pool = pools.get(m.serverId());
        if (pool == null) throw new IllegalStateException(”Sin pool: “ + m.serverId());

        Connection c = pool.getConnection();
        c.setCatalog(m.dbName());   // <-- el “USE shardNNNN”: salta a la base del shard
        return c;
    }

    /** Recarga mapa y pools si la versión de topología cambió (reshard). */
    @Scheduled(fixedDelayString = “${sharding.refresh-interval-ms}”)
    public void refreshIfChanged() {
        long v = topology.currentVersion();
        if (v != loadedVersion) reload();
    }

    private synchronized void reload() {
        // 1. Asegura un pool por cada servidor activo.
        for (PhysicalServer s : topology.loadServers()) {
            pools.computeIfAbsent(s.serverId(), id -> buildPool(s));
        }
        // 2. Reemplaza el mapa de shards de forma atómica.
        Map<Integer, ShardMapping> fresh = topology.loadShardMap().stream()
            .collect(Collectors.toMap(ShardMapping::virtualShard, m -> m));
        this.shardMap = fresh;
        this.loadedVersion = topology.currentVersion();
        // (opcional) cerrar y quitar pools de servidores ya sin shards asignados.
    }

    private HikariDataSource buildPool(PhysicalServer s) {
        HikariConfig cfg = new HikariConfig();
        // Sin base por defecto en la URL: elegimos la BD por conexión con setCatalog.
        cfg.setJdbcUrl(String.format(
            “jdbc:mysql://%s:%d/?rewriteBatchedStatements=true”, s.host(), s.port()));
        cfg.setUsername(props.getShardUser());
        cfg.setPassword(props.getShardPass());
        cfg.setMaximumPoolSize(props.getPoolSizePerServer());
        cfg.setPoolName(”shardpool-” + s.serverId());
        return new HikariDataSource(cfg);
    }
}

Example of using the Router in a Spring Boot application.

@Repository
public class PinRepository {
    private final ShardRouter router;
    private final IdGenerators ids;   // un IdGenerator por shard virtual

    public PinRepository(ShardRouter router, IdGenerators ids) {
        this.router = router; this.ids = ids;
    }

    public long create(long userId, long boardId, String title) {
        int vshard = router.virtualShard(userId);
        long pinId = ids.forShard(vshard).nextId(); // shard embebido en el id
        try (Connection c = router.connectionForUser(userId);
             PreparedStatement ps = c.prepareStatement(
                 “INSERT INTO pins(local_id, user_id, board_id, title, created_at) “ +
                 “VALUES (?,?,?,?, NOW())”)) {
            ps.setLong(1, pinId); ps.setLong(2, userId);
            ps.setLong(3, boardId); ps.setString(4, title);
            ps.executeUpdate();
            return pinId;
        } catch (SQLException e) {
            throw new DataAccessResourceFailureException(”insert pin”, e);
        }
    }

    //.....
}

I hope you found this breakdown useful. While virtual sharding is a powerful and elegant pattern, implementing it successfully involves significant real-world complexity.

Handling high payment volumes drives processing companies toward distributed architectures. This often entails multiple independently deployed and scaled applications, database systems where strict ACID properties are no longer feasible, asynchronous processing, and more.

Distributed architectures bring an inherent challenge: transaction management. We shift toward eventual consistency issues, which are potentially much more difficult to handle.

In practical terms, a monolith or a single SQL database lacks the capacity to scale vertically to handle high transactional throughput; that is precisely why a distributed environment becomes necessary.

Idempotency: The core of preventing duplication

To solve the issue of duplicate payments, the fundamental element for any payment processing entity is guaranteeing the application of the concept of idempotency the moment a payment is received.

An operation is considered idempotent if executing it N times always produces the exact same outcome. In mathematics: f(f(x)) = f(x). In payment systems: processing the same request 10 times must charge the customer exactly once.

For this to work, payment processing systems must receive a unique ID alongside the client’s request. While a request with that ID is currently in-flight, no other request with the same ID can be processed. Once the operation is completed, the payment processing outcome is stored along with that ID. If another payment attempt is made with that same ID, it is not processed again; instead, the system simply returns the previously saved result.

Stating it this way is simple, but implementing it within a distributed system requires deep technical expertise from the solution’s development team.

“Standard” Architecture of a payment processing system

The following diagram outlines, in general terms, the architecture of a payment processing system:

Clients or merchants within the payment processing ecosystem submit requests to create card payments. The payment processor receives the request and, after security checks, access control, tokenization, and other policies are enforced at the API Gateway, routes it to the card processing system. This system interacts with external downstream systems to execute the actual payment and persists the processing outcome into its database.

If the client sends an ID alongside their request, the Card Processor must store the transaction outcome in the StorageDB together with that ID. If another payment attempt is made with the exact same ID, the Card Processor will bypass the external system and simply return the previously saved result. While this seems straightforward, this specific design has inherent flaws that prevent it from being truly idempotent, making it highly unreliable.

Scenarios that expose the flaws in this design:

• Race conditions via simultaneous requests: A client accidentally submits duplicate requests for the same order (using the identical ID) only a few milliseconds apart. The timing is so close that Request 1 is still in-flight and hasn’t finished processing. Because these systems typically rely on horizontal scaling and load balancers (using Round Robin), the Card Processor service runs on multiple instances. Request 1 lands on Card Processor Instance 1, while Request 2 is routed to Card Processor Instance 3. Since Instance 1 has not yet persisted the completed transaction, Instance 3 remains completely unaware of it, starts its own execution loop, and ultimately triggers a duplicate payment.

• Transient network failures and retry Storms: Another scenario demonstrating this fragility involves a transient network failure between the API Gateway and the Card Processor during a request transmission. Because the API Gateway fails to receive a timely response, it executes a retry following its built-in resiliency strategy. However, Request 1 actually did reach the processor successfully, but the gateway never found out. This drops the system into a split-state scenario nearly identical to the race condition described above.

• Additional edge cases: Other common failure modes include the StorageDB failing to provide immediate consistency (I highly recommend reading up on the CAP and PACELC theorems), or the system dispatching the payment to the external provider before safely persisting the state as “in-progress” within the StorageDB, among others.

Distributed mutual exclusion

If the previous architecture were a monolith, we would rely on a local mutual exclusion mechanism (mutex). In Java, it would look something like this:

import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.locks.ReentrantLock;
import java.util.function.Supplier;

public class IdempotencyLocker {

    // Concurrent map to store a dynamic lock per each idempotency key
    private final ConcurrentHashMap<String, ReentrantLock> locks = new ConcurrentHashMap<>();

    /**
     * Executes a critical section by ensuring mutual exclusion based on an idempotency key.
     *
     * @param key    The unique transaction key (e.g., request UUID from HTTP headers)
     * @param action The business logic or code block to be executed exclusively
     * @param <T>    The return type of the action
     * @return The result produced by the action
     */
    public <T> T executeWithLock(String key, Supplier<T> action) {
        if (key == null) {
            throw new IllegalArgumentException("Idempotency key cannot be null");
        }

        // Atomically retrieves the existing lock or maps a new one if it's absent
        ReentrantLock lock = locks.computeIfAbsent(key, k -> new ReentrantLock());

        // Attempts to acquire the mutual exclusion lock
        lock.lock();
        try {
            // Execute the critical section (e.g., database verification and API third-party calls)
            return action.get();
        } finally {
            // Crucial: always release the lock inside the finally block to avoid deadlocks
            lock.unlock();
            
            // Memory leak mitigation: remove the key from the map 
            // only if no other threads are currently waiting in the queue for this lock
            if (!lock.hasQueuedThreads()) {
                locks.remove(key, lock);
            }
        }
    }
}

When a transaction arrives, the very first thing we do is generate a “lock”—essentially putting a padlock on that specific idempotency key before starting to process the transaction (creating the payment entry in the StorageDB, calling the external processor, and updating the response outcome). Once processing is complete, we “release” the lock, opening the padlock. If a second attempt arrives while an active transaction already holds the lock, it will be queued and placed in a waiting state until the in-flight operation finishes. As soon as the current transaction completes, the waiting request wakes up and begins execution; however, since the record was already persisted in the database by the previous execution, the Card Processor will bypass the external downstream provider and simply return the saved result associated with that idempotency key.

The explanation and code sample above apply specifically to local mutual exclusion. In a highly distributed architecture like the one illustrated in our diagram, a local mutex is simply not enough. We must shift toward a distributed mutual exclusion strategy. The most popular solution for this is leveraging Redis by implementing the microservices pattern known as Distributed Lock. The implementation would look like this:

import org.redisson.api.RLock;
import org.redisson.api.RedissonClient;
import java.util.concurrent.TimeUnit;
import java.util.function.Supplier;

public class DistributedIdempotencyLocker {

    private final RedissonClient redissonClient;
    private static final String LOCK_KEY_PREFIX = "lock:idempotency:";

    public DistributedIdempotencyLocker(RedissonClient redissonClient) {
        this.redissonClient = redissonClient;
    }

    /**
     * Executes a critical section by queuing concurrent requests with the same key.
     *
     * @param key    The unique transaction key (e.g., UUID from HTTP headers)
     * @param action The business logic or code block to be executed exclusively
     * @param <T>    The return type of the action
     * @return The result produced by the action
     */
    public <T> T executeWithLock(String key, Supplier<T> action) {
        if (key == null) {
            throw new IllegalArgumentException("Idempotency key cannot be null");
        }

        String redisLockKey = LOCK_KEY_PREFIX + key;
        RLock lock = redissonClient.getLock(redisLockKey);

        boolean isAcquired = false;
        try {
            // PARAMETERS:
            // waitTime = 15s -> Concurrent requests will now QUEUE and WAIT up to 15 seconds.
            // leaseTime = 30s -> Safety lease time in Redis if the instance completely crashes.
            isAcquired = lock.tryLock(15, 30, TimeUnit.SECONDS);

            if (!isAcquired) {
                // This only triggers if the queue wait time exceeds 15 seconds (e.g., a massive timeout downstream)
                throw new IllegalStateException("Lock timeout exceeded for key: " + key + ". Queue wait time expired.");
            }

            // Execute the critical section (e.g., check DB -> call API -> save DB)
            return action.get();

        } catch (InterruptedException e) {
            Thread.currentThread().interrupt();
            throw new RuntimeException("Thread interrupted while waiting in Redis queue", e);
        } finally {
            // Only release if this specific thread holds the lock active
            if (isAcquired && lock.isHeldByCurrentThread()) {
                lock.unlock();
            }
        }
    }
}

The explanation of how it works is exactly the same as for local mutual exclusion. The fundamental difference now is that the idempotency key map resides in Redis, meaning all Card Processor nodes share and view the exact same state. This way, it makes no difference whether the incoming request lands on Card Processor Instance 1 or Instance 3.

Database consistency

This is a critical factor that directly impacts idempotent transactions. It is vital that the data store caching the idempotency key and its final outcome provides strong consistency. If the database relies on eventual consistency, you run a severe risk: the moment Redis releases the lock, a previously queued request with the same idempotency key will wake up and proceed. If this second request attempts to read the outcome from a read-replica of your StorageDB before the data has fully synchronized, it will find nothing.

The ultimate success of idempotent architectures relies on a robust mutual exclusion system combined with a strongly consistent persistence layer.

The general end-to-end workflow operates as follows:

The Steps:

1. Request Initiation: The user/client submits a request to create a payment, including the Idempotency-Key in the HTTP headers.

2. Lock Acquisition: A Card Processor instance receives the request and immediately requests an exclusive lock from Redis.

3. Exclusivity Granted: Redis grants exclusive processing rights for the request associated with that specific idempotency key.

4. State Initialization: The Card Processor persists the idempotency key in the StorageDB along with the payment details and an initial status, such as "Running".

5. External Execution: The Card Processor invokes the necessary external downstream systems to process the actual payment.

6. State Finalization: The Card Processor updates and stores the final transaction outcome and status in the database.

7. Lock Release: The Card Processor requests to release the key in Redis, relinquishing its exclusive processing condition.

Edge cases and failure modes

Although this is a highly robust architecture for managing idempotency in distributed systems, there is at least one vector where—under very rare conditions—certain factors can align to cause an exclusive lock acquisition failure, opening the door to a potential duplicate payment: Redis itself.

While Redis is an extremely battle-tested system, it is not immune to infrastructure failures. The most widely deployed architecture is a Master/Replica topology, where data is written to the master node and can be read from either the master or its replica(s). In this setup, two main issues can occur:

1. Asynchronous replication lag: Because Redis replication is asynchronous by default, a subsequent request might attempt to read a key from a replica before the state has been fully synchronized.

2. Failover and data loss: If the master node suffers a sudden crash, a failover mechanism will promote a replica to master. Any data or lock that had not yet been replicated to that specific replica will be permanently lost.

More powerful consensus algorithms with a different level of complexity, such as Raft, can completely eliminate these specific vulnerabilities. We will deep dive into that topic in a future article.

Thanks for reading this far!

In many scenarios, these operational problems don’t appear on day one, but rather over time, manifesting as operational failures, increased latency, coordination difficulties between teams, component failures that ultimately trigger cascading problems, and so forth. In most cases, this stems from the implementation of poor practices across the platform, not from a single component.

What problem do microservices solve?

To better understand this, it’s important to first examine what problem microservices solve.

In my opinion, the best definition of this concept is offered by Martin Fowler. I’ve simplified it in the following paragraph:

It is an architectural style that structures an application as a set of small, autonomous, and flexibly coupled services, organized around business capabilities and communicating with each other often using lightweight communication mechanisms.

This style addresses the following problems by offering the advantages illustrated below.

• Independent scalability: Allows scaling only the services that need it, without affecting the entire application.

• Faster and more secure deployments: Each service can be updated without redeploying the entire system, reducing risks. Significant improvement in time-to-market.

• Better code maintainability: Divides large applications into smaller parts, easier to understand and modify.

• Autonomous teams: Different teams can work on different services without blocking each other.

• Fault isolation: An error in one service does not necessarily impact the entire system.

• Technological flexibility: Each service can use the language or technology best suited to its needs.

• Agile business evolution: Enables rapid implementation of changes without affecting other areas of the system.

• Improved domain modeling: Facilitates the application of concepts such as Domain-Driven Design, aligning services with business capabilities.

• Complexity management: Divides large systems into more manageable components.

• Resource optimization: Allows for more efficient use of infrastructure according to the needs of each service. (Although it is much more expensive overall).

All these points sound great, but to achieve them you have to avoid certain practices that are already known in the market as the anti-patterns of microservices architecture.

Microservices killers

The following are the 7 fundamental killers of microservices architecture.

Problem 1: Shared databases

This is perhaps the most common killer. It occurs when multiple services read from and write to the same database schema.

• It eliminates autonomy: If Service A changes the name of a column, it breaks Service B.

• It eliminates the freedom to choose the appropriate database engine for each domain.

• It eliminates data ownership; with multiple systems and likely several teams, it becomes unclear who owns each piece of data.

• It limits scalability, or when scaling is implemented, there is a risk that one system could end up affecting the database and, consequently, impacting the stability of the platform.

Coupling ends up being shifted from the code to the data layer.

Problem 2: Distributed Monolith

This is a classic problem that occurs in some implementations. It happens when coupling is so high that it’s impossible to leverage the clear advantages of microservices related to independence.

• Waterfall deployments: The ability to truly deliver continuous service is lost. If Service A requires the exact version of Service B to function, the risk of errors in production increases exponentially.

• Complexity without benefit: You pay the “network tax” (latency, serialization) without gaining the agility of microservices.

• Runtime fragility: If one service goes down, the entire system collapses because dependencies are rigid and synchronous instead of being decoupled by events.

• Cognitive overload: Developers must know the state of multiple repositories and services just to make a small change to business logic.

Ultimately, the monolith problem ends up being distributed across multiple teams, without any benefit.

Problem 3: Chatty services API

This problem is more common in teams with limited experience adopting microservices. When an inexperienced team begins migrating to microservices, it’s very common to turn “anything” into a small service. This is extremely dangerous and leads to excessive communication between components to resolve a request.

• Accumulated latency: The total response time for the user is the sum of all internal network calls, resulting in a poor user experience and a degraded overall API level.

• Inefficient resource Consumption: Each call involves TLS negotiation (if present), HTTP headers, and JSON/gRPC serialization, which unnecessarily consumes CPU and network bandwidth. Furthermore, it leads to having more services and therefore a larger infrastructure to maintain.

• Larger failure surface: The more calls, the greater the likelihood that a timeout or a 500 error in an intermediate service will break the entire transaction. Although this can be mitigated by applying stability patterns such as timeout limiters, retries, or circuit breakers, it unnecessarily increases the possibility of failures, and the fallacies of distributed computing must always be considered.

• Interface coupling: The client (or the calling service) must know too many internal details of other services’ structures to construct a complete response.

Excessive communication occurs when there are too many services due to incorrect context delimitation during the architecture design.

Problem 4: The God Service

This problem usually arises when there is a central, complex, and extremely fundamental service in the execution of the business process. This is the service that some companies call “the service” or “the API.” It has complex logic, is large, and few people touch it, or no one wants to touch it. An error in this service has a high operational impact on the business.

Some problems with having a god service are:

• Organizational bottleneck: Waiting queues are created in teams because everyone needs to modify the same service for their respective functionalities.

• Inefficient scaling: You are forced to scale giant instances of a service (with a lot of RAM/CPU) just because a small functionality within it is under heavy load or so important that it’s better to have it scaled “just in case.”

• Long compilation and test times: The service becomes so heavy that unit and integration tests take too long, slowing down the CI/CD pipeline.

• Resistance to change: The code becomes “spaghetti” and difficult to refactor due to the enormous number of responsibilities mixed together. Sometimes it’s better to “leave it alone,” as doing so poses a very high risk.

Having a “god service” is sometimes a normal part of architecture during a transition from monolithic to microservices decomposition. What is NOT normal is failing to recognize the need to work on improving or decomposing it into simpler components adapted to the business’s capabilities.

Problem 5: Shared Infrastructure

This problem is more general than Problem 1 (shared database), but it’s common for not only the database to be shared. In some microservices implementations, other components are also shared, which negatively impacts the architecture. Examples include deployment nodes (VMs, VPSs, etc.), shared distributed caching systems like Redis, and so on.

The fundamental problems with sharing infrastructure are:

• A traffic spike in a low-priority microservice can exhaust resources (CPU, memory, I/O) and affect a critical service hosted on the same node.

• Single point of failure (SPOF): If the shared Redis or Kubernetes cluster isn’t properly partitioned, a misconfiguration in one area can affect the entire ecosystem.

• Difficulty in cost management: It’s nearly impossible to determine how much each service consumes individually if they all share the same resources without clear labels or boundaries.

• Configuration conflicts: Limitations in environment variables or library versions at the operating system level that affect all services equally.

Problem 6: Nanoservices

This is the opposite of a monolith. It occurs when we see extremely simple applications in the architecture, for example: a service that performs a simple calculation, such as taking an amount and calculating the VAT by simply adding 21% of its value, or whose logic is to return a person’s age based on their date of birth, etc.

If it is too simple or does not cover a bounded context (DDD), it is probably a bad idea to convert it into a service. I personally like to look at how many tables or entities in the database the service has to manage. If it is only one, for example, a User’s Address, it is a red flag to consider.

Nanoservices involve some significant problems:

• High operational complexity: Managing 50 services for simple logic implies configuring 50 CI/CD pipelines, 50 monitoring systems, and 50 security schemes.

• High latency: A simple business operation that should take 20ms in memory ends up taking 500ms due to jumps between services (resulting in chatty services).

• Data inconsistency: Maintaining data integrity across so many small services forces the use of distributed transactions (Sagas) for things that shouldn’t be so complex.

• Knowledge fragmentation: It’s difficult to understand the complete flow of an application when the logic is scattered across insignificant pieces of 10 lines of code each.

Problem 7: Everything is a Service

Closely related to the previous problem, this is more of the same. Every new feature to be developed is conceived as a microservice and not as a feature in itself, or ultimately, as a reusable library that will be used by several components.

Thinking of everything as a service ends up combining all the negative aspects of the previous anti-patterns:

• Nanoservices are created.

• Maintenance/infrastructure costs increase.

• Code duplication sometimes occurs.

• Additional points of failure are introduced.

• Excessive communication (chatty services) is generated.

Not everything is a service; sometimes it just needs a little organization and to be placed in the right bounded context.

I hope you enjoyed the article. In these times when everyone’s talking about AI (including this blog), it’s good to review some architecture.

Spanish version is available in SACAViX Tech Newsletter, read here

For an LLM, a traditional REST endpoint is a black box requiring external documentation (OpenAPI) and rigid manual integration. This is where the Model Context Protocol (MCP) comes in. MCP is the standard that allows language models to “see” and “touch” business logic natively, securely, and, above all, in a standardized way. It’s an open standard that facilitates the connection of AI assistants with systems where data resides, such as content repositories and enterprise tools. Its goal is to allow language models to access relevant and up-to-date information, improving the quality of their responses. (Thanks to Anthropic for this 🩵)

In this article, we’ll explore how to transcend the REST backend. Using the power of Spring AI, we’ll explain the process of exposing a REST API as an MCP service. We’ll see how to move from data exposure to providing dynamic tools that models can reason, invoke, and orchestrate in real time.

What is Spring AI?

Spring AI is a Spring ecosystem project designed to facilitate the integration of artificial intelligence (AI) capabilities into Java applications, especially those based on Spring/Boot. Its best feature is that it provides a single interface for interacting with different models; using Spring AI, you can connect to OpenAI, Amazon Bedrock, Vertex Gemini, or a local model in Ollama, among others, without changing your application’s code. And no, it’s not what you might think; it’s not just a client for connecting to LLMs—it’s much more than that.

From a REST API to an MCP Server

Any Spring application that exposes an HTTP REST service can expose its endpoints via MCP, but some concepts change.

In the REST world, applications expose endpoints; in modern MCP, these are called tools. End of story.

So, the way to do this is to create a “Controller” that exposes the “tools” instead of the endpoints.

Before showing you how to do this with a practical example, it’s important to understand some things related to the types of transport that MCP supports. Learn more about transports in MCP official doc.

Project dependencies

We’re going to implement our sample REST API and expose endpoints as tools. We’ll use the streamable HTTP transport mechanism.

Currently, MCP in Spring is at version 2.x, but it’s still under development (Milestone versions). We’ll use version 1.x. It’s important to note that whether our API uses servlets or the reactive stack, we can do this; it’s independent.

The main dependencies are the web starter (to create the standard REST API), and the mcp server webmvc; this is important, if we were using webflux we would have to use the corresponding dependency of the mcp server.

<dependency>
   <groupId>org.springframework.boot</groupId>
   <artifactId>spring-boot-starter-web</artifactId>
 </dependency>

 <dependency>
    <groupId>org.springframework.ai</groupId>
    <artifactId>spring-ai-starter-mcp-server-webmvc</artifactId>
 </dependency>

REST Controller + Streamable MCP

The REST API is the same as always; let’s see an example in the code.

@RestController
@RequestMapping(”/rates”)
public class ExchangeRateController {

    private final ExchangeRateService exchangeRateService;

    public ExchangeRateController(ExchangeRateService exchangeRateService) {
        this.exchangeRateService = exchangeRateService;
    }

    @GetMapping
    public ExchangeRate getExchangeRate(
            @RequestParam(”from”) String from,
            @RequestParam(”to”) String to) {
        return this.exchangeRateService.getExchangeRate(from, to);

    }
}

As the code itself explains, it’s a service that allows you to obtain the exchange rate from a source currency to a destination currency using `GET /rates`.

Now, how do we convert this into a tool exposed via MCP so that LLMs can run it?

In Spring AI, it’s quite simple; the dependency we included does the “magic” behind the scenes.

You just need to annotate the methods appropriately, and you’re done. To keep the project clean, I prefer not to “clutter” the Spring controllers with MCP annotations. Therefore, I organize the tools in a package called “tools,” which would be the equivalent of “controller” in an n-tier architecture.

In the tools folder, we add as many as we need; we can follow a structure based on resources or however we want.

import com.yoandypv.exchangerate.apirest2mcp.service.ExchangeRate;
import com.yoandypv.exchangerate.apirest2mcp.service.ExchangeRateService;
import org.springaicommunity.mcp.annotation.McpTool;
import org.springaicommunity.mcp.annotation.McpToolParam;
import org.springframework.stereotype.Component;

@Component
public class ExchangeRateTool {

    private final ExchangeRateService exchangeRateService;

    public ExchangeRateTool(ExchangeRateService exchangeRateService) {
        this.exchangeRateService = exchangeRateService;
    }

    @McpTool(
            name = “getExchangeRate”,
            description = “Get the exchange rate between two currencies using ISO codes (e.g., USD to EUR)”
    )
    public ExchangeRate getExchangeRate(
            @McpToolParam(description = “Source currency in ISO format (e.g., USD)”)
            String from,

            @McpToolParam(description = “Target currency in ISO format (e.g., USD)”)
            String to) {
        return exchangeRateService.getExchangeRate(from, to);
    }
}

As you can see, it’s a simple Spring component that injects the bean with the service logic (which is shared with the REST controller).

The @McpTool and @McpToolParam annotations.

• @McpTool: Allows you to define the tool (”the endpoint”). It’s important that it has a name so that LLMs know how to invoke it, and a clear description that expresses exactly what the tool does, so LLMs know what it’s used for.

• @McpToolParam: Used to annotate parameters. Similarly, the description should be clear so that LLMs know what value to use when invoking the tool. It allows other options, such as indicating whether the parameter is mandatory or not.

MCP Server configuration

It is important to define the appropriate configurations for the MCP service to function properly.

spring.ai.mcp.server.name=exchange-rate-mcp
spring.ai.mcp.server.version=0.0.1
spring.ai.mcp.server.type=SYNC
spring.ai.mcp.server.protocol=streamable
spring.ai.mcp.server.stdio=false

That's enough.

An additional topic is the issue of CORS. Since it’s HTTP, keep in mind that enabling CORS may be important depending on the client. The example code provides access to everything, but follow CORS best practices if you’re going to expose the API.

To connect an MCP client like Claude, Visual Studio Code, Cursor, or any other, the endpoint will be: http://localhost:8080/mcp.

• The introduction of lightweight and cheaps threads controlled by the JVM to optimize traditional platform threads; initially known as fibers, now called virtual threads.

• Providing a structured concurrency mechanism to the Java platform.

• Adapting ThreadLocal for virtual threads through Scoped Values.

As an interesting fact, virtual threads are daemon threads.

In Java 21, JEP 444 was officially released, providing the JDK with "stable" virtual threads. However, the pinning problem still remained to be solved. If you're not very familiar with it, I recommend reading this article (Spanish version) on the blog.

To understand virtual threads, you need to know four basic concepts:

• Carrier threads: These are the platform threads on which virtual threads run. By default, the number of platform threads created matches the value of Runtime.getRuntime().availableProcessors(), though this is configurable.

• Mount: The process by which the Virtual Thread Scheduler assigns the execution of a virtual thread to a platform thread (putting it on a carrier thread so it can run).

• Unmount: The opposite of mounting. If a virtual thread enters a waiting state, the scheduler unmounts it to free up the platform thread, preventing it from staying blocked and allowing it to be used for another task.

• Pinning: The issue that occurred before Java 24, where a virtual thread, under certain conditions, remained mounted on a carrier thread, preventing the resource from being freed.

With these concepts and this image, you’ll be able to see how virtual threads operate on top of platform (carrier) threads.

Java 21 vs 24: Pinning solved

To see if the pinning problem was resolved, we conducted an experiment with a block of code that produces pinning in Java 21 using the Java synchronized primitive.

The example:

private static final Object lock = new Object();

.....

try (ExecutorService executor = Executors.newVirtualThreadPerTaskExecutor()) {
    for (int i = 0; i < 100; i++) {
        executor.submit(() -> {
            synchronized (lock) {
                System.out.println(Thread.currentThread());
                try {
                    Thread.sleep(1000);
                } catch (InterruptedException e) {
                    Thread.currentThread().interrupt();
                }
            }
        });
    }
}

In short, we create an executor that will execute each task submitted in a virtual thread. We sent it a group of tasks with a block synchronized by a global variable.

To see if the execution of a code produces pinning, we have the option: -Djdk.tracePinnedThreads=full.

For the experiment, we executed the previous code with the flag enabled in JDK 21 and 24. We also visualized the execution with the Visual VM profiling tool.

Java 21: With pinning

Java version used was:

openjdk version "21.0.6" 2025-01-21 LTS
OpenJDK Runtime Environment Corretto-21.0.6.7.1 (build 21.0.6+7-LTS)
OpenJDK 64-Bit Server VM Corretto-21.0.6.7.1 (build 21.0.6+7-LTS, mixed mode, sharing)

When executing the code block on standard output we have:

VirtualThread[#27]/runnable@ForkJoinPool-1-worker-2
VirtualThread[#27]/runnable@ForkJoinPool-1-worker-2 reason:MONITOR
    java.base/java.lang.VirtualThread$VThreadContinuation.onPinned(VirtualThread.java:199)
    java.base/jdk.internal.vm.Continuation.onPinned0(Continuation.java:393)
    java.base/java.lang.VirtualThread.parkNanos(VirtualThread.java:635)
    java.base/java.lang.VirtualThread.sleepNanos(VirtualThread.java:807)
    java.base/java.lang.Thread.sleep(Thread.java:507)
    com.yoandypv.vt.PinnedThreadsExample.lambda$sample2$1(PinnedThreadsExample.java:50) <== monitors:1
    java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
    java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
    java.base/java.lang.VirtualThread.run(VirtualThread.java:329)
VirtualThread[#33]/runnable@ForkJoinPool-1-worker-8
VirtualThread[#35]/runnable@ForkJoinPool-1-worker-8
VirtualThread[#32]/runnable@ForkJoinPool-1-worker-7
VirtualThread[#37]/runnable@ForkJoinPool-1-worker-7
......

This section is relevant:

VirtualThread[#27]/runnable@ForkJoinPool-1-worker-2 reason:MONITOR
    java.base/java.lang.VirtualThread$VThreadContinuation.onPinned(VirtualThread.java:199)

It indicates that some threads remained pinned during execution, specifically that virtual thread #27 was pinned to carrier thread worker-2. This is a problem because the thread remains busy and is not freed to take on other work, which impacts the scalability promised by virtual threads.

In the code, you'll see Continuation.onPinned—this is crucial. Virtual threads run thanks to Continuation, which I believe is the real revolution behind Loom; virtual threads are just one application of continuations. We’ll discuss this in another article.

If we analyze this execution in VisualVM, it looks like this:

The most important thing is that the carrier threads (workers 1 to 8) remain in the WAIT state during execution, which effectively represents a blockage. The pinning problem is present.

Java 24: Bye bye pinning

With JDK 24 we execute the experiment with the same example, the specific data of the JDK are:

openjdk version "24" 2025-03-18
OpenJDK Runtime Environment Corretto-24.0.0.36.2 (build 24+36-FR)
OpenJDK 64-Bit Server VM Corretto-24.0.0.36.2 (build 24+36-FR, mixed mode, sharing)

After code execution the output was:

VirtualThread[#26]/runnable@ForkJoinPool-1-worker-1
VirtualThread[#103]/runnable@ForkJoinPool-1-worker-1
VirtualThread[#132]/runnable@ForkJoinPool-1-worker-2
VirtualThread[#131]/runnable@ForkJoinPool-1-worker-2
VirtualThread[#134]/runnable@ForkJoinPool-1-worker-2
VirtualThread[#133]/runnable@ForkJoinPool-1-worker-2
VirtualThread[#130]/runnable@ForkJoinPool-1-worker-1
VirtualThread[#129]/runnable@ForkJoinPool-1-worker-2
VirtualThread[#128]/runnable@ForkJoinPool-1-worker-2
VirtualThread[#127]/runnable@ForkJoinPool-1-worker-1
....

The first point to note is that the log indicating pinning didn't appear when using the flag. If we go to VisualVM, it looks like this:

Carrier threads are shown in the Parked state. The Parked state indicates in this context that the scheduler has unmounted the virtual thread, avoiding blocking the platform thread. The carrier thread has been "parked" and could receive more work.

As Neil Armstrong's famous phrase says:

One small step for man, one giant leap for mankind

Final thoughts

I haven't yet fully confirmed that the pinning issue has been resolved in other scenarios, such as those related to JNI. We'll have to wait for this new version to mature a bit and undergo further testing. But the fact that it's resolved for Synchronized is already a fundamental leap forward.

See you in the next one. Share if you think it might help others learn.

The article delves into the internal workings of MySQL, focusing on how data is stored and how indexes function to optimize query performance. Key topics include:

• Data Storage in MySQL/InnoDB: Data is stored in structures called "pages," each typically 16KB in size. These pages contain rows of data, and when one is filled, a new page is created. Pages are grouped into "extents," and extents into "segments."

• Page Structure: Each page comprises a header with metadata, the main body with data rows, and a footer acting as an internal directory or index for quick data location within the page.

• Indexes in MySQL: Indexes are structures that enhance search operations on data tables, improving retrieval performance. There are two main types:

  • Clustered Indexes: In InnoDB, there's a single clustered index per table, built using the primary key. Data is stored in a B-Tree structure, with leaf nodes containing the actual data rows.

  • Non-Clustered (Secondary) Indexes: These indexes don't store data themselves but hold pointers to data in the data pages.

• Analyzing and Optimizing Queries: The article emphasizes the importance of understanding how data is stored and indexed to effectively analyze and optimize query performance.

Let’s start.

How is information stored in MySQL?

In MySQL, when we create a table and add data, the information is stored in structures called pages. A page is the basic unit of information and has a default size of 16KB. Pages store the rows of data that we add. When a page is full, a new one is created, and so on.

Two other important concepts for understanding MySQL's internal structure are extends and segments. Extends are a group of contiguous pages, and segments are a collection of extends.

Regarding pages, there are two main types: data pages and index pages. In this post, we will discuss both, starting with data pages.

A data page can be graphically represented as follows:

Each page contains a header with general information and metadata about the page, such as the number of records it holds and the available space, among other details. The main body of the page contains the data we add to the table, and at the end, there is a kind of directory or internal index for the page, which allows direct access to the position of each record. Do not confuse these indexes with the ones we create; these are specific to the page itself.

In MySQL with InnoDB, if a table is not explicitly assigned a primary key, one will be implicitly assigned. This ensures that created tables already have an implicit clustered index (we will discuss this later). In other database engines, this does not happen, and in tables without an explicit index defined, the pages are grouped in a structure called a heap.

As data is added and pages are filled, new pages of information are created.

How to know how many pages your table has?

In reality, there is no easy way to know this, and I don’t think it’s very relevant. However, if you’re as curious as I am, with the command I show below, you can get the size of the table. By dividing it by 16K, you can estimate the approximate number of pages (16K is the default size, but this value can be configured).

SHOW TABLE STATUS LIKE 'tu-tabla';

Now that we know how information is stored, let’s talk about indexes.

What is an index in MySQL?

An index in MySQL is a data structure that improves the speed of data retrieval operations on a database table. It works similarly to an index in a book, allowing the database to quickly locate and access specific rows without having to scan the entire table. Indexes are particularly useful for speeding up queries that involve filtering, sorting, or joining data.

When you create an index on a column (or a set of columns), MySQL builds a separate structure that stores a sorted version of the data in that column, along with pointers to the corresponding rows in the table. This allows the database engine to find the required data much faster than if it had to scan the entire table row by row.

However, while indexes improve read performance, they can slightly slow down write operations (such as inserts, updates, and deletes) because the index must be updated whenever the data in the table changes. Therefore, it’s important to use indexes strategically, balancing the benefits of faster queries with the overhead of maintaining the index.

Indexes do not store data; they only store pointers to data that resides in the data pages.

The clustered index in MySQL with InnoDB is built using the primary key, where the data is stored in a BTree structure sorted (fundamentally) by the primary key. Each node of the tree, which is an index page, contains a reference to either a data page or another index page. In the data page, the data is ordered by the primary key, which is why there can only be one clustered index per table. If we do not define a primary key, MySQL will assign one by default. It is called "clustered" because it refers to a group of keys (see the graphic below for better understanding).

The secondary index is any additional index created on a table, used to improve searches on columns other than the PRIMARY KEY. It is also based on a BTree, but it only stores the indexed column and a pointer to the primary key. This allows for faster record retrieval without the need to perform a full scan of the table. Secondary indexes can be simple or composite. A simple index uses a single column, while a composite index uses more than one column.

Both types of indexes are represented in a BTree structure, as mentioned earlier. A binary tree structure looks like this:

In the case of the tree used to represent indexes, the leaves of the tree will be the data pages, while the intermediate nodes will be the index pages.

Let’s look at an example of a clustered index (based on the primary key).

If, using these tables, a query were executed to retrieve the data of players whose IDs are between 1 and 5, only the leftmost branch of the tree would be traversed, avoiding a full scan of the table. Let’s see this in practice below.

The representation of secondary indexes will be covered in another article, but we will see how to handle them next.

Let’s look at the importance of indexes.

Suppose we have a table like the following:

CREATE TABLE employees (
    id INT, 
    name VARCHAR(100),
    last_name VARCHAR(100),
    email VARCHAR(50) UNIQUE,
    department VARCHAR(100), 
    salary DECIMAL(10, 2) 
) ENGINE=InnoDB;

As you can see, it is a simple table that does not have a primary key defined. For our example, we have created a stored procedure that will help us populate this table. I’ll leave the code below in case you’re interested (I made this procedure with the assistance of DeepSeek AI).

DELIMITER $$

CREATE PROCEDURE InsertRandomEmployees()
BEGIN
    DECLARE i INT DEFAULT 0;
    DECLARE random_name VARCHAR(100);
    DECLARE random_last_name VARCHAR(100);
    DECLARE random_email VARCHAR(50);
    DECLARE random_department VARCHAR(100);
    DECLARE random_salary DECIMAL(10, 2);

    WHILE i < 100000 DO
        SET random_name = CONCAT('Name', FLOOR(RAND() * 100000));
        SET random_last_name = CONCAT('LastName', FLOOR(RAND() * 100000));
        SET random_email = CONCAT('email', i + 1, '@example.com');  -- 
        SET random_department = ELT(FLOOR(1 + RAND() * 5), 'Sales', 'Marketing', 'IT', 'HR', 'Finance');
        SET random_salary = RAND() * 10000;

        INSERT INTO employees (id, name, last_name, email, department, salary)
        VALUES (i + 1, random_name, random_last_name, random_email, random_department, random_salary);

        SET i = i + 1;
    END WHILE;
END$$

DELIMITER ;

As you can see, we inserted about 100K records. Once the table is populated, we execute a query like the following:

select count(*) from employees where department='Marketing';

At first glance, nothing seems wrong—it is syntactically correct. However, if we analyze the query, we will see:

 explain select * from employees where department='Marketing';
+----+-------------+-----------+------------+------+---------------+------+---------+------+-------+----------+-------------+
| id | select_type | table     | partitions | type | possible_keys | key  | key_len | ref  | rows  | filtered | Extra       |
+----+-------------+-----------+------------+------+---------------+------+---------+------+-------+----------+-------------+
|  1 | SIMPLE      | employees | NULL       | ALL  | NULL          | NULL | NULL    | NULL | 99661 |    10.00 | Using where |
+----+-------------+-----------+------------+------+---------------+------+---------+------+-------+----------+-------------+
1 row in set, 1 warning (0.00 sec)

No index could be used, and the query performed a full scan, going through all 100K records in our table. At this rate, as the table continues to grow, the query performance will deteriorate further. (The column "rows" shows 99661 because it is only an approximate value.)

Let’s take the EXPLAIN to the next level by using **ANALYZE** for more details:

 explain analyze select * from employees where department='Marketing';
+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| EXPLAIN                                                                                                                                                                                                                               |
+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| -> Filter: (employees.department = 'Marketing')  (cost=10102.35 rows=9966) (actual time=0.037..50.405 rows=20052 loops=1)
    -> Table scan on employees  (cost=10102.35 rows=99661) (actual time=0.017..42.679 rows=100000 loops=1)
 |
+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

Here the problem is clearer. Explanation of the result:

Operation: Filter:

• Description: A filter is applied to select only the rows where `department = 'Marketing'`.

• Estimated cost: 10102.35 cost units. This value is an estimate of how much effort MySQL requires to perform this operation.

• Estimated rows: 9966 rows. MySQL estimates that approximately 9966 rows will meet the condition `department = 'Marketing'`.

• Actual time:`0.037..50.405`: The time it took to perform this operation. The first value is the initial time to find the first row, and the second value is the total time to process all rows.

• Actual rows: 20052 rows. This is the actual number of rows that meet the condition.

• Loops: 1. Indicates that this operation was executed once.

Operation: Table scan on employees:

• Description: A full table scan is performed on the `employees` table to find the rows that meet the condition.

• Estimated cost: 10102.35 cost units.

• Estimated rows: 99661 rows. MySQL estimates that it will scan approximately 99661 rows.

• Actual time:`0.017..42.679`: The time it took to perform the full table scan.

• Actual rows: 100000 rows. This is the total number of rows in the `employees` table.

• Loops: 1. Indicates that this operation was executed once.

If we finally execute the query:

select count(*) from employees where department='Marketing';
+----------+
| count(*) |
+----------+
|    20052 |
+----------+
1 row in set (0.04 sec)

The estimates were not wrong—the actual filtered rows from above were 20052, and the estimated scan time was 0.04.

While this works, it is not acceptable and needs improvement. Let’s try making two changes: first, by creating a primary key on the `id` column (this will require rebuilding the table), and second, by creating a secondary index on the `department` column since it is part of the search condition (WHERE clause).

Adding PK and secondary index:

(table will be re-created and re-populated again)

CREATE TABLE employees2 (
    id INT PRIMARY KEY AUTO_INCREMENT, 
    name VARCHAR(100),
    last_name VARCHAR(100),
    email VARCHAR(50) UNIQUE,
    department VARCHAR(100), 
    salary DECIMAL(10, 2) 
) ENGINE=InnoDB;

New index:

CREATE INDEX idx_department ON employees2 (department);

Once created, let’s repeat the EXPLAIN.

 explain analyze select * from employees2 where department='Marketing';
+--------------------------------------------------------------------------------------------------------------------------------------------------------+
| EXPLAIN                                                                                                                                                |
+--------------------------------------------------------------------------------------------------------------------------------------------------------+
| -> Index lookup on employees2 using idx_department (department='Marketing')  (cost=4335.55 rows=39268) (actual time=0.200..32.259 rows=20062 loops=1)
 |
+--------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.04 sec)

Only the created index will be used for the execution of the query in a single iteration.

The results:

select count(*) from employees2 where department='Marketing';
+----------+
| count(*) |
+----------+
|    20062 |
+----------+
1 row in set (0.01 sec)

The query went from taking 0.04 to 0.01. Just 25% of execution time.

I had fun and learned a lot while writing this article. I hope you enjoyed it too.

See you in the next one!