When people hear the term soft skills, they often think about speaking fluent English, giving presentations, or answering interview questions.

In reality, soft skills are much broader than that.

Soft skills are your ability to understand people, adapt to situations, communicate effectively, and work professionally in different environments. They often determine how far you grow in your career, especially after the first few years.

I’ve worked with highly talented engineers who struggled because they lacked soft skills. I’ve also seen average engineers grow into leadership positions because they were excellent at working with people.

Here are the most important soft skills every IT professional should develop, ranked by their long-term impact on career growth.

1. Emotional Intelligence (EQ)

If there is one soft skill that influences almost every other soft skill, it is Emotional Intelligence.

EQ is the ability to understand and manage your emotions while also understanding the emotions, motivations, and perspectives of others.

People with strong EQ:

• Handle criticism professionally

• Stay calm during production incidents

• Manage disagreements without damaging relationships

• Understand what others are trying to communicate

• Build trust within teams

Many workplace problems are not technical problems. They are people problems.

A person with strong EQ can navigate those situations effectively.

2. Communication

Communication is not about speaking more.

It is about ensuring that your message is understood by the right person in the right way.

Good communication includes:

• Explaining technical concepts clearly

• Writing effective emails and documentation

• Providing project updates

• Conducting meetings efficiently

• Presenting ideas confidently

One of the biggest misconceptions about communication is that there are fixed rules.

For example, some people say you should never repeat information already present in your resume when answering “Tell me about yourself.”

In reality, effective communication depends on context. Different interviewers expect different things.

The real skill is understanding your audience and adapting accordingly.

3. Adaptability

The technology industry changes faster than most professions.

New tools emerge.
New frameworks appear.
Teams get reorganized.
Managers change.
Business priorities shift.

People who resist change often struggle.

People who adapt thrive.

Adaptability means:

• Learning new technologies

• Adjusting to new work environments

• Working with different personalities

• Handling changing priorities

The ability to adapt is often more valuable than any single technical skill.

4. Problem Solving

Organizations do not hire engineers simply to write code.

They hire engineers to solve problems.

Strong problem solvers:

• Investigate root causes

• Think critically

• Analyze options

• Make informed decisions

• Focus on solutions rather than complaints

When production systems fail or projects encounter obstacles, problem-solving skills become more valuable than technical knowledge alone.

5. Stakeholder Management

As your career grows, your success becomes less dependent on code and more dependent on collaboration.

You will work with:

• Product Managers

• Business Teams

• Architects

• Project Managers

• Clients

• Senior Leadership

Each group has different priorities.

A good engineer understands how to communicate effectively with each stakeholder and align everyone toward a common goal.

6. Ownership and Accountability

One of the biggest differences between junior and senior professionals is ownership.

Junior mindset:

That’s not my responsibility.

Ownership mindset:

Let me see how I can help solve this.

Ownership does not mean doing everyone else’s work.

It means taking responsibility for outcomes, communicating proactively, and ensuring issues are addressed rather than ignored.

Managers consistently trust people who demonstrate ownership.

7. Time Management

Technical expertise is valuable, but it loses effectiveness if work is consistently delayed.

Good time management involves:

• Prioritizing tasks

• Estimating work realistically

• Managing deadlines

• Avoiding unnecessary distractions

• Communicating delays early

Professionals who manage their time well become more reliable and less stressed.

8. Conflict Resolution

Disagreements are inevitable.

Teams disagree on:

• Architecture decisions

• Priorities

• Resource allocation

• Project timelines

The goal is not to avoid conflict.

The goal is to resolve conflict professionally.

Strong professionals can disagree without becoming personal, emotional, or defensive.

9. Decision Making

Many workplace situations do not have perfect answers.

Often, you must choose between:

• Speed vs quality

• Cost vs performance

• Simplicity vs flexibility

Strong decision-makers gather information, evaluate trade-offs, and move forward confidently rather than remaining stuck in analysis.

10. Learning Agility

Technology evolves continuously.

The skills that are valuable today may become less relevant tomorrow.

Learning agility is the ability to:

• Learn quickly

• Stay curious

• Explore new technologies

• Apply new knowledge effectively

The most successful professionals are usually lifelong learners.

Final Thoughts

Many people think soft skills are about using polished language or following communication formulas.

They are not.

Soft skills are about understanding people, situations, and context, then responding appropriately.

Technical skills help you build solutions.

Soft skills help you work effectively with people.

In the long run, career growth depends on both.

If you focus on developing Emotional Intelligence, Communication, Adaptability, Problem Solving, and Stakeholder Management, you’ll build a foundation that will serve you throughout your entire career, regardless of the technologies you work with.

Here’s the surprise: LinkedIn isn’t looking at your pixels. It never analyzes the image to guess whether it’s AI. Instead, it reads a hidden label the AI tool stamped into the file when it was created.

That label is called C2PA (Content Credentials). Think of it as a signed nutrition label baked into the file — what made it, when, and whether AI was involved. OpenAI adds it to every image. Many other tools don’t. So images without it arrive at LinkedIn wearing no name tag, and there’s nothing to flag.

Which leads to the one thing worth remembering:

No badge doesn’t mean the image is real.

It’s also fragile — a screenshot or a metadata strip erases it. So treat the badge as a helpful disclosure, not a lie detector.

The badge is small. The shift toward “every file carries its origin” isn’t.

#!/usr/bin/env python3
"""
c2pa_tool.py — Detect and remove C2PA Content Credentials from images.

Install deps:
    pip install c2pa-python Pillow

Usage:
    python c2pa_tool.py detect  image.jpg
    python c2pa_tool.py strip   image.jpg  out.jpg            # surgical (JPEG: keeps EXIF)
    python c2pa_tool.py strip   image.jpg  out.jpg --resave   # brute force (drops ALL metadata)

Note: C2PA is a voluntary transparency signal. Some jurisdictions (e.g. the EU AI
Act) place disclosure obligations on AI-generated media; removing the marker to
pass synthetic content off as human-made may run against those rules or a
platform's terms. Use on your own files and for legitimate reasons.
"""

import sys
import struct
import mimetypes

try:
    from c2pa import Reader
except ImportError:
    Reader = None


# ---------- DETECTION ----------

def detect(path):
    """Return a dict of C2PA info, or None if the file carries no manifest."""
    if Reader is None:
        raise RuntimeError("c2pa-python not installed: pip install c2pa-python")
    import json
    mime = mimetypes.guess_type(path)[0] or "image/jpeg"
    with open(path, "rb") as f:
        reader = Reader.try_create(mime, f)   # returns None when no manifest
        if reader is None:
            return None
        store = json.loads(reader.json())
    active = store.get("active_manifest")
    m = store.get("manifests", {}).get(active, {})
    return {
        "claim_generator": m.get("claim_generator"),
        "title": m.get("title"),
        "is_embedded": reader.is_embedded(),
        "num_manifests": len(store.get("manifests", {})),
    }


# ---------- REMOVAL: surgical (JPEG only, preserves EXIF etc.) ----------

def strip_c2pa_jpeg(in_path, out_path):
    """Drop only APP11 segments that carry C2PA/JUMBF data. Keeps other metadata.
    Returns the number of segments removed."""
    with open(in_path, "rb") as f:
        data = f.read()
    if data[:2] != b"\xff\xd8":
        raise ValueError("Not a JPEG — use --resave for other formats")

    out = bytearray(b"\xff\xd8")
    i, n, removed = 2, len(data), 0
    while i < n:
        if data[i] != 0xFF:
            out.extend(data[i:]); break
        marker = data[i + 1]
        if marker == 0xDA:                       # start of scan: copy rest verbatim
            out.extend(data[i:]); break
        if marker in (0xD8, 0xD9) or 0xD0 <= marker <= 0xD7:   # standalone markers
            out.extend(data[i:i + 2]); i += 2; continue
        seg_len = struct.unpack(">H", data[i + 2:i + 4])[0]
        seg = data[i:i + 2 + seg_len]
        payload = data[i + 4:i + 2 + seg_len]
        if marker == 0xEB and (b"jumb" in payload[:60] or b"c2pa" in payload[:120]):
            removed += 1                          # this is the C2PA segment: skip it
        else:
            out.extend(seg)
        i += 2 + seg_len

    with open(out_path, "wb") as f:
        f.write(out)
    return removed


# ---------- REMOVAL: brute force (any format, drops ALL metadata) ----------

def strip_by_resave(in_path, out_path, quality=95):
    """Re-encode pixels into a fresh file. Removes C2PA and every other metadata."""
    from PIL import Image
    img = Image.open(in_path)
    clean = Image.new(img.mode, img.size)
    clean.putdata(list(img.getdata()))
    save_kwargs = {"quality": quality} if out_path.lower().endswith((".jpg", ".jpeg")) else {}
    clean.save(out_path, **save_kwargs)


# ---------- CLI ----------

def main():
    args = sys.argv[1:]
    if not args or args[0] not in ("detect", "strip"):
        print(__doc__); sys.exit(1)

    if args[0] == "detect":
        info = detect(args[1])
        if info is None:
            print("No C2PA / Content Credentials found.")
        else:
            print("C2PA FOUND:")
            for k, v in info.items():
                print(f"  {k}: {v}")
        return

    # strip
    in_path, out_path = args[1], args[2]
    if "--resave" in args:
        strip_by_resave(in_path, out_path)
        print(f"Re-saved (all metadata dropped) -> {out_path}")
    else:
        removed = strip_c2pa_jpeg(in_path, out_path)
        print(f"Removed {removed} C2PA segment(s) -> {out_path}")

    # verify
    still = detect(out_path)
    print("Verification:", "C2PA still present!" if still else "clean, no C2PA.")


if __name__ == "__main__":
    main()

If you have ever written code that looks like this, you are the reason MERGE exists:

-- Does this student already exist?
SELECT COUNT(*) FROM students WHERE id = 1;

-- If yes...
UPDATE students SET marks = 95 WHERE id = 1;

-- If no...
INSERT INTO students VALUES (1, 'Anu', 95);

Three round trips to the database, an if statement in your application code, and a race condition waiting to happen if two people run it at once. MERGE collapses all of that into a single statement.

This post walks through it from scratch. No prior knowledge assumed beyond INSERT, UPDATE, and JOIN.

The problem in one sentence

You have fresh data arriving, and some of it is new while some of it is an update to rows you already have. You want both handled in one pass.

This pattern is so common it has a nickname: upsert (update + insert). MERGE is the ANSI SQL way to do it.

Setting up the example

We need two tables: the one we want to change, and the one holding the new data.

The target table — what we already have

CREATE TABLE students (
    id    INT PRIMARY KEY,
    name  VARCHAR(50),
    marks INT
);

INSERT INTO students VALUES (1, 'Anu',  70);
INSERT INTO students VALUES (2, 'Bala', 80);

id name marks 1 Anu 70 2 Bala 80

The source table — what just arrived

CREATE TABLE students_new (
    id    INT PRIMARY KEY,
    name  VARCHAR(50),
    marks INT
);

INSERT INTO students_new VALUES (1, 'Anu',    95);
INSERT INTO students_new VALUES (3, 'Chitra', 88);

id name marks 1 Anu 95 3 Chitra 88

Look carefully at what is going on here, because it is the whole point of the exercise:

• id 1 exists in both tables, but the marks are different → needs an UPDATE

• id 3 exists only in the new data → needs an INSERT

• id 2 exists only in the old data and is not mentioned in the new data → should be left alone

The MERGE statement

MERGE INTO students AS target
USING students_new AS source
ON target.id = source.id

WHEN MATCHED THEN
    UPDATE SET target.marks = source.marks

WHEN NOT MATCHED THEN
    INSERT (id, name, marks)
    VALUES (source.id, source.name, source.marks);

That is it. Run it, then check the table:

SELECT * FROM students ORDER BY id;

id name marks 1 Anu 95 2 Bala 80 3 Chitra 88

Anu’s marks were updated. Bala was untouched. Chitra was inserted. One statement, three different outcomes.

Reading the statement line by line

Do not try to memorise the syntax. Understand what each line is answering.

Clause The question it answers MERGE INTO students Which table am I changing? USING students_new Where is the new data coming from? ON target.id = source.id How do I know if a row already exists? WHEN MATCHED Found it — now what? WHEN NOT MATCHED Didn’t find it — now what?

The mental model that makes it click: MERGE is a JOIN that takes action. The ON clause is a join condition, exactly like in a SELECT. Rows that join successfully fall into WHEN MATCHED. Rows in the source that find no partner fall into WHEN NOT MATCHED.

The words target and source are just aliases. You could call them t and s, or old and new. They carry no special meaning to the database.

The source does not have to be a table

This trips people up because tutorials always use two tables. In reality the USING clause accepts anything that produces rows — a subquery, a join, or a single hardcoded row:

MERGE INTO students AS target
USING (SELECT 4 AS id, 'Dinesh' AS name, 76 AS marks) AS source
ON target.id = source.id
WHEN MATCHED THEN
    UPDATE SET target.marks = source.marks
WHEN NOT MATCHED THEN
    INSERT (id, name, marks)
    VALUES (source.id, source.name, source.marks);

This is how MERGE gets used for a single record coming from an application, rather than a bulk load.

Adding conditions to a branch

You can attach an extra condition to WHEN MATCHED so the update only fires when it is actually worth doing:

WHEN MATCHED AND target.marks <> source.marks THEN
    UPDATE SET target.marks = source.marks

Now a student whose marks have not changed is skipped entirely. On a large table this saves a lot of pointless writes, and it keeps your “rows affected” count honest.

The third branch: deleting

There is a clause for rows that exist in the target but are missing from the source:

WHEN NOT MATCHED BY SOURCE THEN
    DELETE

Add that to our example and Bala disappears, because he is not in students_new.

Be careful with this one. It is the fastest way to accidentally empty a table. If your source data is incomplete for any reason — a partial file, a failed export, a filtered query — this clause will happily delete every row that was missing. Many teams ban it outright and use a soft-delete flag instead:

WHEN NOT MATCHED BY SOURCE THEN
    UPDATE SET target.is_active = 0

Four mistakes beginners make

1. Forgetting the semicolon. In SQL Server, a MERGE statement must end with ;. Leave it off and you get an error that does not obviously point at the missing semicolon. This is the single most common first-attempt failure.

2. Putting filters in the ON clause. The ON clause is the matching rule, not a WHERE clause. Writing ON target.id = source.id AND source.marks > 50 does not mean “only process rows above 50” — it means rows below 50 fail to match and get sent to WHEN NOT MATCHED, where they are inserted as duplicates. If you want to filter the source, filter it inside the USING subquery instead.

3. Assuming MERGE exists everywhere. It does not. More on this below.

4. Running it against production without a transaction. MERGE touches many rows at once and there is no undo. Wrap it, check the result, then commit:

BEGIN TRANSACTION;

MERGE INTO students AS target
USING students_new AS source
ON target.id = source.id
WHEN MATCHED THEN UPDATE SET target.marks = source.marks
WHEN NOT MATCHED THEN INSERT (id, name, marks)
     VALUES (source.id, source.name, source.marks);

SELECT * FROM students ORDER BY id;   -- inspect before committing

-- COMMIT;      -- happy?
-- ROLLBACK;    -- not happy?

Which databases actually support MERGE?

Database Support SQL Server Yes, full support Oracle Yes, full support PostgreSQL Yes, from version 15 onwards MySQL / MariaDB No — use INSERT ... ON DUPLICATE KEY UPDATE SQLite No — use INSERT ... ON CONFLICT DO UPDATE

If you are on MySQL, the equivalent is shorter:

INSERT INTO students (id, name, marks)
VALUES (1, 'Anu', 95) AS new
ON DUPLICATE KEY UPDATE marks = new.marks;

And on PostgreSQL or SQLite, most people still reach for ON CONFLICT even where MERGE is available, because it is more concise for simple cases:

INSERT INTO students (id, name, marks)
VALUES (1, 'Anu', 95)
ON CONFLICT (id) DO UPDATE
SET marks = EXCLUDED.marks;

Here EXCLUDED is a virtual table holding the row you tried to insert — it plays the same role that source plays in MERGE.

Note that these shorter forms need a PRIMARY KEY or UNIQUE constraint on the column to work. Without one, the database has no definition of “duplicate” and the conflict never triggers.

One caveat worth knowing about

SQL Server’s MERGE has a well-documented history of bugs and concurrency problems under heavy concurrent load, and some experienced DBAs avoid it in high-traffic systems, preferring an explicit UPDATE followed by a conditional INSERT inside a transaction with appropriate locking hints.

For learning, for batch jobs, and for the overwhelming majority of ordinary applications, MERGE is completely fine. But if a senior engineer on your team pushes back on it during code review, this is why — and it is a reasonable position, not pedantry.

Summary

MERGE answers a single question: for each incoming row, does it already exist? Then it branches.

• WHEN MATCHED → update

• WHEN NOT MATCHED → insert

• WHEN NOT MATCHED BY SOURCE → delete, if you dare

Get comfortable with the two-table example above, run it, change the data, run it again, and watch what happens. Once the join-that-takes-action model clicks, every variation is just detail.

Every data engineer has been handed a bug that “cannot happen.” A primary key that duplicated. A COUNT(*) that changed with no writes. A join on two identical strings that matched nothing.

These aren’t sloppy mistakes. They’re what happens when a single-machine mental model meets a system that is distributed across time and space and only eventually consistent. The failures are as predictable as physics — once you know which force is acting on you.

Twenty of them, with the fix for each.

Part I — Time

1. The primary key that duplicated itself

Symptom: A “guaranteed unique” key appears twice, for two genuinely different events, roughly twice a year.

Cause: The key is derived from local wall-clock time — YYYYMMDDHHMMSS + sequence, or a natural key built from a local date-hour. During the DST fall-back, the 01:00–02:00 hour happens twice, so two real events stamp the same key. (Note: UTC-based schemes like Snowflake IDs are immune. This bites legacy ERP, telecom, and batch systems.)

Fix: Generate and store all keys and timestamps in UTC; render local time only at the presentation layer. If a local-time key is unavoidable, include the UTC offset in the key. Add a uniqueness assertion in the pipeline instead of trusting the generator.

2. The event that arrived before it was created

Symptom: Negative latency in your metrics. received_at < created_at.

Cause: Clock skew between services. Service A’s clock runs a few hundred milliseconds ahead of Service B’s. Worse than the ugly metric: “keep the latest record” dedupe logic silently retains the older version, because skew reordered them.

Fix: Never order events across machines by wall-clock time. Use a broker-assigned sequence (Kafka offsets), a monotonic version column, or a logical clock. Enforce NTP and alert on drift. Clamp negative latencies and count them — a rising count is your skew monitor.

3. The day that gained or lost revenue

Symptom: Daily totals disagree with the source system by a few hours’ worth of transactions, but the monthly total matches.

Cause: Timezone-naive timestamps. The warehouse buckets by UTC date, the business defines “day” in local time, and everything between midnight and the offset lands in the neighbouring day. Multi-region businesses have several answers to “which day is it.”

Fix: Store timestamp with time zone (or UTC plus an explicit source-timezone column). Define one business calendar and one canonical reporting timezone, in a documented, queryable dimension table. Bucket by that, not by the raw timestamp.

4. The row that arrived after the door closed

Symptom: Yesterday’s dashboard number changes weeks later — or never does, and quietly stays wrong.

Cause: Late-arriving data landing in a partition that was already processed. Mobile clients buffer offline events for days. Upstream systems replay. The batch job scanned that partition on schedule and moved on.

Fix: Partition by ingestion time and query by event time, or maintain a watermark plus a rolling reprocess window (e.g. re-run the last N days each night). Track and chart event-time-to-ingestion-time lag so you can size the window from evidence rather than hope.

Part II — Space

5. The COUNT(*) that changed with no writes

Symptom: Same query, same table, ten seconds apart, no ETL running — two different numbers.

Cause: Load-balanced read connections routed to different read replicas, each at a different point in the replication stream. From the analyst’s side it’s “the same table.” Physically it’s several copies converging asymmetrically.

Fix: Pin reconciliation and reporting queries to the primary or to a designated replica. Expose replication lag as a first-class metric next to the dashboard. For anything auditable, read from a versioned snapshot (a table format with snapshot isolation) rather than “the current table.”

6. The write that vanished right after you made it

Symptom: Insert succeeds, immediate read returns nothing, second read a moment later returns the row.

Cause: Read-after-write against an asynchronous replica. Common after a failover, when the new primary is momentarily behind what clients think they wrote.

Fix: Route read-your-own-write paths to the primary, or use session/monotonic consistency if your engine offers it. Never build orchestration logic that writes then immediately polls a replica for confirmation.

7. The file that isn’t in the listing

Symptom: A job writes 500 files; the downstream job processes 497. No error anywhere.

Cause: Eventually consistent object storage listing — a LIST returns stale or partial results depending on which node served it. Also produced by data lake table formats used without strict snapshot isolation.

Fix: Don’t discover work by listing a prefix. Have the writer emit an explicit manifest of what it produced, and have the reader consume the manifest. Use a table format with atomic commits (Iceberg, Delta, Hudi) so readers see a consistent snapshot or nothing.

8. The row that exists but no query finds it

Symptom: SELECT * on the whole table finds it. Every dated or filtered query misses it. Row counts and row sums permanently disagree.

Cause: A record whose partition key was updated and which physically moved, or which landed in a partition outside the queried range. Partition pruning then legitimately skips it.

Fix: Treat partition keys as immutable — delete and re-insert rather than update. Run a periodic reconciliation that compares unpartitioned totals against the sum of partition totals; a persistent gap is the alarm.

9. The update that lost to an older update

Symptom: A correction is applied, confirmed, and then silently reverts to the previous value.

Cause: Multi-region or multi-master writes resolved by last-write-wins using wall-clock timestamps. The “later” write had an earlier clock, so the correct value lost the tiebreak. Data loss with no error and no log line.

Fix: Resolve conflicts with version vectors, monotonic revision numbers, or a CRDT — not timestamps. Where the business can tolerate it, route all writes for a given entity to one region. Log every conflict resolution so losses are visible instead of silent.

10. The dimension that changed under a running join

Symptom: A report built from two joined tables shows a total that matches neither source, and can’t be reproduced.

Cause: The fact table and dimension table were read at different moments. The dimension was updated in between — a customer’s region changed mid-run — so part of the join used the old value and part the new one.

Fix: Read all inputs of a job from one consistent snapshot (a single transaction, or pinned table versions). For dimensions, model them as slowly-changing with validity ranges and join on effective date rather than on current state.

Part III — Consistency and Determinism

11. The SUM() that returns a different total each run

Symptom: Finance reports a reconciliation break of a few paise. Rerun the query and it’s fine. Rerun again and it’s back.

Cause: Floating-point addition isn’t associative. A parallel engine partitions the data differently on each run, adds the values in a different order, and the trailing decimal places drift.

Fix: Store money as DECIMAL/NUMERIC, never FLOAT or DOUBLE. Where floats are unavoidable, round at a defined precision before comparison and give reconciliation checks an explicit tolerance.

12. The pagination that skips and repeats rows

Symptom: An export of 100k rows in pages of 1,000 ends up with duplicates and gaps. Total row count is right, contents aren’t.

Cause: ORDER BY on a non-unique column plus LIMIT/OFFSET. Rows with equal sort values have no defined order, so each page request may order them differently.

Fix: Always add a unique tiebreaker to the sort key. Better, paginate by keyset (WHERE (sort_col, id) > (last_sort, last_id)) rather than by offset, which is also faster on large tables.

13. The GROUP BY that picks a value at random

Symptom: A column returns a plausible but different value on every run.

Cause: Selecting a non-aggregated column that isn’t in the GROUP BY. Some engines reject this; some pick an arbitrary row from the group and don’t tell you.

Fix: Enable strict mode (ONLY_FULL_GROUP_BY and equivalents) so this becomes an error. Make the intent explicit with MIN, MAX, or a windowed ROW_NUMBER() over a defined ordering.

14. The backfill that didn’t match the original run

Symptom: Rerunning a historical job produces different output than the first execution, for the same input window.

Cause: Non-deterministic functions baked into the logic — CURRENT_TIMESTAMP, NOW(), RAND(), CURRENT_USER — or a view that reads “today’s” state rather than the state as of the processing date.

Fix: Pass the logical execution date in as a parameter and use it everywhere instead of NOW(). Ban non-deterministic functions from transformation logic and enforce it in code review or a linter. A job should be a pure function of (input snapshot, execution date).

15. The retry that double-counted revenue

Symptom: Totals are exactly right most days and inflated on days when a task failed and was retried.

Cause: A non-idempotent pipeline step. The task appended rows, failed partway through a later step, and the retry appended them again. The scheduler considers this a success.

Fix: Make every write idempotent — MERGE/upsert on a business key, or delete-the-partition-then-insert. Use atomic commits so a partial write is never visible. Carry an idempotency key through any external API call the pipeline makes.

Part IV — Identity

16. The ID that was issued twice

Symptom: Two entirely different orders share an order ID, months apart.

Cause: Either an auto-increment counter reset after a restart (older MySQL kept it in memory and recomputed it as MAX(id)+1, reissuing IDs that downstream systems had already consumed), or a promoted replica that hadn’t replicated the latest sequence state and resumed handing out IDs the old primary already used. The database’s uniqueness guarantee is real — it’s just per node, per lifetime.

Fix: Use UUIDv4/v7 or a coordinated ID service for anything that leaves the database. Never treat a source auto-increment ID as globally unique in the warehouse: key on (source_system, source_id, load_date) and add a hard duplicate check at ingestion.

17. The unique constraint that allowed duplicates

Symptom: Twelve “unique” rows for the same customer, with the constraint still in place and valid.

Cause: UNIQUE(customer_id, region_code) doesn’t prevent duplicates when region_code is NULL, because NULL is not equal to NULL in most engines.

Fix: Make the columns of a unique key NOT NULL and use an explicit sentinel for “unknown.” Alternatively use a unique index over COALESCE(col, sentinel) if the engine supports expression indexes.

18. The hash key with real collisions

Symptom: Two unrelated records collapse into one, or a “collision-proof” surrogate key produces duplicates far too often for a cryptographic accident.

Cause: A surrogate key built as MD5(concat(col_a, col_b)) with no delimiter, so ('AB','C') and ('A','BC') hash identically. Not a hash collision — a concatenation bug wearing one’s costume.

Fix: Concatenate with a delimiter that cannot appear in the data, and represent NULLs explicitly rather than as empty strings. Normalise casing and trimming before hashing, not after, so the rule is applied consistently everywhere.

19. The upsert that merged two people

Symptom: No duplicate — the opposite. Two customers become one row with blended history, and nobody gets an error.

Cause: A merge key of LOWER(TRIM(email)). Two people at a company share info@acme.com. Or a case-folding rule maps distinct characters onto one (the Turkish dotless-i is the classic) and the collation treats them as equal.

Fix: Merge on a genuinely identifying key, not on a contact attribute. Where fuzzy matching is required, write matches to a review queue rather than applying them silently, and keep the pre-merge rows so a merge can be undone.

Part V — Encoding and Types

20. The join on identical strings that matched nothing — and the column that became null

Symptom: Two values look identical on screen and don’t join. Separately, Namibia disappears from the country list.

Cause: Two failures of the same kind — the data changed in transit.

• Unicode: “José” stored as a single precomposed é in one system (NFC) and as e plus a combining accent in another (NFD). Byte-different, visually identical. Same story with non-breaking spaces from a copy-paste, trailing whitespace, and zero-width characters from web forms.

• Type inference: the country code NA parsed as NaN by a default CSV reader; leading zeros stripped from postcodes and account numbers; IDs like 1.23E5 coerced to floats; gene names converted to dates by a spreadsheet.

Fix: Normalise all text to NFC at ingestion and strip invisible characters explicitly. Never rely on inferred schemas at a system boundary — declare every column’s type, read identifiers as strings, and set the null sentinel explicitly (keep_default_na=False and equivalents). Add a contract test that fails when an inferred type changes.

Closing

The pattern under all twenty: each one is impossible in a mental model with one clock, one machine, and one copy of the data. Production has none of those things. Once you stop being surprised by that, these stop being ghost stories and start being a checklist.