Reminder: I’m offering a limited-time 50% discount on the annual plan:

50% OFF FOREVER

Once you claim it, the discount will be applied forever.

Now, with only $5/month, you will have access to:

• 200+ deep-dive data engineering articles

• practice-spark: 65 LeetCode-style problems to practice Spark SQL/DataFrame

• learn-spark/dbt/airflow: CLI tools to master Spark/dbt/Airflow

If you’re a Vietnamese user, please DM me for an upgrade due to payment issues

Intro

I’ve written quite a few articles about how to learn data engineering skills: the roadmap, the mindset, OLAP, Python, stream processing, data modeling, open table formats…

However, I don’t feel they provide a comprehensive, straightforward guide for newcomers; some LinkedIn folks or my Substack readers often reach out and say they still don’t know where to begin learning data engineering.

So, I think it would be nice to have an article like this one: not a step-by-step guide to learning all the technical skills, but rather a guideline that gives you direction on what to do and what not to do, based on my observations and experience.

The ideal audience for this article is recent graduates, career switchers, and those who have just started their careers in data engineering. If you’ve already been working in data engineering for a while, you can think of this as a “review” guide, a chance to reflect on what you’ve learned along the way.

—

Here are all the things I would do if I had to start learning data engineering all over again in 2026.

Note: Read this article only if you’re sure you want to pursue a career in data engineering and already understand what a data engineer is responsible for. If not, please make sure you have that understanding before going any further.

tl:dr

• At first, you learn “enough” to do an end-to-end project that could actually provide “business value”. (a pet project that helps you answer some questions with data)

  • The goal is to be aware of how you can bring value.

• Invest in data modeling early.

• Invest your time in deep-dive processes later; learn how to scale your learning of fundamentals to new concepts/ideas/tools.

• Be proactive

• Leverage AI to learn, but don’t abuse it too soon at your early career stage. Making mistakes and actually doing something is a privilege.

• Keep in mind: AI is the new customer now.

Mistake

One of the mistakes I see among newcomers is that they try to master a single skill before moving on (e.g., mastering Python, SQL, or an OLAP service). The truth is, to provide value as a data engineer, you must do a lot of things.

A simple pipeline that reads from PostgreSQL, transforms it, and writes the result to Snowflake requires you to, at least, know how to do these things:

• Data modeling to design how tables in Snowflake are organized.

• Write Python code to read data from PostgreSQL.

• Write SQL to transform the data and sink it to Snowflake

• Create an Airflow DAG to glue everything together.

• Write tests to validate all the logic.

• Automate the development process with CI/CD.

• Optimize the SQL query if it is slow.

• Scale everything accordingly if the data volume increases.

• Talk to the stakeholders if there is an incorrect number or the requirement changes.

• …

To bring value fast and effectively, you don’t need to go down the rabbit hole with one skill. The key is to learn a bit here and there.

A bit here and there

Reminder: I’m offering a limited-time 50% discount on the annual plan:

50% OFF FOREVER

Once you claim it, the discount will be applied forever.

Now, with only $5/month, you will have access to:

• 200+ deep-dive data engineering articles

• practice-spark: 65 LeetCode-style problems to practice Spark SQL/DataFrame

• learn-spark/dbt/airflow: CLI tools to master Spark/dbt/Airflow

If you’re a Vietnamese user, please DM me for an upgrade due to payment issues

Read more

Reminder: I’m offering a limited-time 50% discount on the annual plan:

50% OFF FOREVER

Once you claim it, the discount will be applied forever.

Now, with only $5/month, you will have access to:

• 200+ deep-dive data engineering articles

• practice-spark: 65 LeetCode-style problems to practice Spark SQL/DataFrame

• learn-spark/dbt/airflow: CLI tools to master Spark/dbt/Airflow

If you’re a Vietnamese user, please DM me for an upgrade due to payment issues

Intro

Polars and DuckDB have been the obvious choices for many organizations for single-node processing.

They’re performant, easy to use, and easy to set up.

One day, I just thought it would be fun to run both of them on the same dataset to see how they perform.

In this article, I will try to run DuckDB and Polars on datasets of 10GB, 50GB, 100GB, 200GB, and 465 GB.

The data volume here is the amount of data in memory; when generating data, the on-disk size will be 3x-4x smaller, since the data will be written in compressed Parquet.

The motivation for the dataset size is that I want to see how DuckDB and Polars perform if the data fits entirely in my laptop's 24 GB RAM (with a 10 GB dataset), then stress test them with a dataset larger than the memory (from 50GB, 100GB, 200GB, to the extreme case of 465GB). Besides speed, I also try to observe reliability (e.g., query failures) and memory usage. All the code is available here.

If you only care about the results and observations, feel free to jump right away to the “Result” section :d

Note 1: This benchmark was run on my laptop, so the results could be very different if it were run on a more isolated machine. Also, there are likely performance tuning techniques for both tools that I'm not aware of, so feel free to share your suggestions in the comments.

Note 2: I asked Claude to help me set up the code for this benchmark. However, all the design decisions and intentions were mine.

Note 3: I run this in the way I like so I’m pretty sure it’s not the standard way to run a benchmark :d

Note 4: This article won’t dive much into DuckDB or Polars. You can check out my previous articles here:

Setup

I ran this on my laptop: an Apple M5 with a 10-core CPU, 24 GB of RAM, and a 1 TB SSD. I also tried to dedicate as many resources as possible to the benchmark; during the run, I closed all other applications.

Python dependencies

First, clone this repo.

Enter its folder and run

uv sync

…to install the required Python libraries.

We leverage ibis-bench, the ibis library with utilities for benchmarking across engines such as DuckDB, Polars, and DataFusion.

We use native functions in ibis-bench to generate data and run benchmarks. However, we need a thin wrapper (/src/bench/tuned_run.py) on top of it because there are two shortcomings:

• ibis-bench’s own CLI doesn’t have a way to pass DuckDB memory_limit/temp_directory/max_temp_director_size setting.

• ibis-bench’s own write_data.py always materializes Polars LazyFrame results using an in-memory engine rather than a streaming engine (given the incompleteness of the Polars streaming engine in the past). With the recent introduction of Polars' new streaming engine, this wrapper attempts to force the Polars data-writing process to use it.

With Polars streaming engine, instead of processing all at once, the data is split into small batches that flow through the query pipeline, rather than the entire table being materialized in memory.

This contrasts with the in-memory engine, which tries to fit the entire dataset into memory.

For DuckDB and Polars, we will use the latest stable versions (at the time of this writing)

dependencies = [
    "ibis-bench==0.2.0",
    "duckdb==1.5.5",
    "polars==1.43.2",
]

The queries

You can view all the queries here.

The ibis-bench library provides 22 tcph queries written in DuckDB SQL and the Polars DataFrame API. Let’s take a brief overview of these queries:

Q1

Pricing summary: revenue, quantity, discount grouped by return status/line status for items shipped before a given date. Full sequential scan + aggregation over all of lineitem (biggest table); simple GROUP BY, no joins

Q2

Cheapest suppliers for a part type/size, in a given region. Multi-way join (5 tables) + correlated subquery; Tests join planning and subquery execution

Q3

Highest-revenue unshipped orders for a market segment. 3-table join + GROUP BY + ORDER BY + LIMIT; tests join + sort/top-N

Q4

Count of orders per priority where at least one line item was received late. Join + correlated EXISTS subquery; tests semi-join / subquery-to-join rewriting

Q5

Revenue by nation for a region within a year. 6-table chain join (customer → orders → lineitem → supplier → nation → region) where supplier and customer must resolve to the same nation; tests the optimizer’s join-order selection across a long join chain, since joining in the wrong order blows up intermediate result size long before the final small GROUP BY

Q6

Total revenue lost to discounts in a date/discount/quantity range. Single-table filter + aggregation on lineitem; tests scan + predicate pushdown, no joins

Q7

Revenue between two nations via suppliers/customers, by year. Self-referential nation join (both supplier and customer nation) + multi-table join.

Q8

Market share of a nation’s suppliers for a part type over time. Large join (8 tables) + conditional aggregation (CASE-WHEN)

Q9

Profit by nation/year for parts matching a name pattern. Large join (6 tables) + string pattern match + GROUP BY on derived year

Q10

Customers with the highest returned-item revenue, by nation. Join + GROUP BY + ORDER BY + LIMIT; similar shape to Q3 but keyed on customer.

Q11

Parts whose total supply value is a significant fraction of a nation’s total. Aggregation + correlated subquery comparing to an aggregate threshold (HAVING against a scalar subquery)

Q12

Correlation between shipping mode and order priority (late shipments). Join + CASE-based conditional counts; tests conditional aggregation.

Q13

Distribution of orders per customer (including customers with zero orders). LEFT OUTER JOIN + GROUP BY + a second aggregation over the first (nested grouping); tests outer join handling

Q14

Percentage revenue from promotional parts in a time window. Join + conditional SUM (CASE) over lineitem/part; tests filtered aggregation.

Q15

Supplier(s) with the highest total revenue in a quarter. View/CTE + join + subquery to find the max; tests subquery materialization and top-N-by-group

Q16

Count of suppliers who can supply parts matching a spec, excluding certain suppliers/complaints. Join + NOT IN / anti-join subquery + COUNT DISTINCT; tests anti-join and distinct aggregation.

Q17

Avg yearly revenue loss from small-quantity orders of a specific part. Correlated subquery computing a per-part average as a filter threshold; tests scalar subquery pushdown.

Q18

Top orders by total quantity, for customers with very large orders. Join + GROUP BY + HAVING on an aggregate + correlated subquery; tests large-order/HAVING filtering.

Q19

Revenue from parts meeting one of several brand/quantity/shipping combinations. Single join with a complex OR-of-ANDs predicate; tests predicate simplification/optimization.

Q20

Suppliers in a nation who over-supply a part relative to half its total shipped quantity. Nested correlated subqueries (2 levels) + anti-join-like filter; one of the most subquery-heavy queries

Q21

Suppliers with late-shipped line items on multi-supplier orders, excluding certain cases. Self-join on lineitem + EXISTS/NOT EXISTS subqueries; tests complex existence checks.

Q22

Customers with no orders in the last 7 years but above-average account balance, by country code. Correlated subquery (avg balance) + NOT EXISTS + string prefix matching; tests subquery + anti-join combo

The experiment

Generate data and run benchmark

Reminder: I’m offering a limited-time 50% discount on the annual plan:

50% OFF FOREVER

Once you claim it, the discount will be applied forever.

Now, with only $5/month, you will have access to:

• 200+ deep-dive data engineering articles

• practice-spark: 65 LeetCode-style problems to practice Spark SQL/DataFrame

• learn-spark/dbt/airflow: CLI tools to master Spark/dbt/Airflow

If you’re a Vietnamese user, please DM me for an upgrade due to payment issues

Read more

Reminder: I’m offering a limited-time 50% discount on the annual plan:

50% OFF FOREVER

Once you claim it, the discount will be applied forever.

Now, with only $5/month, you will have access to:

• 200+ deep-dive data engineering articles

• practice-spark: 65 LeetCode-style problems to practice Spark SQL/DataFrame

• learn-spark/dbt/airflow: CLI tools to master Spark/dbt/Airflow

If you’re a Vietnamese user, please DM me for an upgrade due to payment issues

Parquet is the most important file format in data engineering.

Its columnar (hybrid) format and self-contained nature make it work so well for analytical workloads.

Learning Parquet matters not only because it’s one of the most widely adopted file formats in data engineering, but also because you can apply Parquet’s fundamentals to nearly all analytical storage systems: from how data is organized and encoded to how metadata plays an important role in query performance.

That said, I realized it can be a bit challenging for beginners to learn about Parquet, especially without a detailed visualization of its internals.

I created a tool (with Claude) to help you visualize any Parquet file.

It’s totally free.

Just:

pip install git+https://github.com/vutrinh274/petquar.git

And:

petquar <your-path-to-the-parquet-file>

All the source code available here: GitHub repo.

It will then open a new browser tab with that Parquet file’s internals. The UI includes:

• The beginning and end of the file, including the magic numbers

• Row group cards that visualize the file’s row groups

• Inside each row group are the column chunks

• A row group’s size (shown once expanded) and a column chunk’s size are proportional to their real, compressed size on disk

• Clicking a row group or column chunk opens an Inspector panel with a plain-language explanation and the raw facts, so you can see exactly what’s inside the file.

• Column chunk information covers encoding, whether a bloom filter is present, and min/max statistics -- and drills down to the page level too, showing the number of pages plus each page’s encoding and min/max statistics.

• The schema (every column’s name, type, and nullability) is available from the FileMetadata block.

• All of this information is parsed by Rust arrow-rs.

So, give it a try and see if it helps you.

Don’t hesitate to send feedback (the link next to the “petquar” logo, top left) so I can improve it further.

Reminder: I’m offering a limited-time 50% discount on the annual plan:

50% OFF FOREVER

Once you claim it, the discount will be applied forever.

Now, with only $5/month, you will have access to:

• 200+ deep-dive data engineering articles

• practice-spark: 65 LeetCode-style problems to practice Spark SQL/DataFrame

• learn-spark/dbt/airflow: CLI tools to master Spark/dbt/Airflow

If you’re a Vietnamese user, please DM me for an upgrade due to payment issues

Reminder: I’m offering a limited-time 50% discount on the annual plan:

50% OFF FOREVER

Once you claim it, the discount will be applied forever.

Now, with only $5/month, you will have access to:

• 200+ deep-dive data engineering articles

• practice-spark: 65 LeetCode-style problems to practice Spark SQL/DataFrame

• learn-spark/dbt/airflow: CLI tools to master Spark/dbt/Airflow

If you’re a Vietnamese user, please DM me for an upgrade due to payment issues

Intro

In the previous article, I shared my experience of self-learning stream processing from zero. While writing it, I realized that theory alone won’t make it click. That’s why I designed a project to better understand what a stream processing project looks like.

This article acts as a guideline to walk you through the project. There will be 4 scenarios, each with a different setup, output, and relevant concepts. My intention is to give you a playground where you can tune all kinds of configurations to learn about stream processing.

Throughout this article, I also share what I learned after running these scenarios.

We will go through them one by one.

Note 1: This project is developed with the help of Claude. The planning, design and intention still from myself.

Note 2: This project aims to dive deeper into the technical aspects of stream processing, so we will skip the "business" aspect here. If you're building a side project to learn the full data engineering lifecycle or strengthen your resume, I still recommend focusing on the business value and doing data modeling before moving on other things.

Prerequisite

This project requires you to have experience working with Python and Docker. Also, it would be great if you read or learned Kafka, Flink, and ClickHouse fundamentals, as well as stream processing concepts (event/process time, windowing, watermarking, …) beforehand.

If not, I suggest you read this article first.

Set up

You can follow along by cloning the GitHub repo here.

After cloning, make sure you’re in the folder and start the Docker containers:

docker compose up -d --build

The project has three main components:

• Kafka for message absorption: A single Kafka broker container that exposes port 9092. We will publish messages using this port.

• Flink for stream processing: two containers:

  • Flink JobManager container that exposes port 8081. We will submit a Flink job using this port. Also, access localhost:8081 for the Flink UI.

  • Flink TaskManager container is where the Flink job is executed. The TaskManager will have 6 slots. We will understand what these slots mean later.

• ClickHouse is the destination: a single container that exposes port 8123.

Prometheus and Grafana are also there for observability. All the Grafana charts’ configurations can be found in charts.json

You can access the Grafana UI for monitoring metrics via localhost:3000. The dashboard name is “Stream Processing Project”

—

Then install the Python libraries for some related Python scripts.

uv sync
source .venv/bin/activate

Those Python scripts are:

• event_gen.py: the synthetic clickstream event generator. Its job is to simulate user sessions moving through a funnel (page_view → product_view → add_to_cart → purchase). Each event will be ~1KB. We will use logic in this script for our source data in this project.

• producer.py: a configurable Kafka producer logic. Leverage event_gen.py for generating events and publishing them (to Kafka). Accept inputs from outside to control running duration, the Kafka batch size, or the linger ms.

• multi_producer.py: run several producer.py processes in parallel.

• event_injector.py: publishes a set of ad-hoc events (from a JSON spec file) with controllable event_time and send-order. It is used for the two scenarios that we observe on the correctness (basic window sum, late-event threshold).

• run_experiment.py: this is the scenario runner. It reads experiments/scenarios.yml to get each scenario’s configuration and automates each scenario: Flink TaskManager resource limiting, Kafka topic recreation, Flink job cancel/submit, launching producers, and printing a pipeline measurement report at the end.

• windowed_revenue.py: the PyFlink job that reads clickstream from Kafka, counts every event (no filtering) in a tumbling window grouped by (category, country), writes results to ClickHouse.

• raw_event_sink.py: simpler PyFlink job with no-windowing logic. Writes every event straight to ClickHouse as its own row (no aggregation), used to measure true single-event latency.

Understand TaskSlot

For Flink execution resources in this project, we set the number of TaskSlots to 6.

But what is a TaskSlot?

A TaskManager is a JVM process, whereas a TaskSlot is a piece of the resource within that process.

• The TaskManager’s memory is shared between the slots.

• There is no CPU isolation between the TaskSlots.

• By default, a slot can run subtasks of different tasks as long as they are from the same job.

  A Flink job is a dataflow graph with a series of transformation. You can think of task is a step in it where subtask is a parallel unit execution of a task.

  • This mechanism is called slot sharing. Tasks that share the same slot-sharing group can be executed in the same slot and, thus, share resources. By default, all tasks are assigned the same SlotSharingGroup, called default

  

• This means that a slot can hold one parallel job instance, which by default can contain one subtask from each task in the job. In our scenarios, the Flink job will have two tasks (e.g., A and B); each will have 6 subtasks that run in parallel (e.g., A1, A2…, B1, B2…), for a total of 12 subtasks. Because subtasks from different tasks can run on the same slot, these 12 subtasks can share 6 slots, for example:

  • Slot 1: A1 + B1

  • Slot 2: A2 + B2

  • Slot 3: A3 + B3

  • Slot 4: A4 + B4

  • Slot 5: A5 + B5

  • Slot 6: A6 + B6

Now let’s jump on the first scenario.

Reminder: I’m offering a limited-time 50% discount on the annual plan:

50% OFF FOREVER

Once you claim it, the discount will be applied forever.

Now, with only $5/month, you will have access to:

• 200+ deep-dive data engineering articles

• practice-spark: 65 LeetCode-style problems to practice Spark SQL/DataFrame

• learn-spark/dbt/airflow: CLI tools to master Spark/dbt/Airflow

If you’re a Vietnamese user, please DM me for an upgrade due to payment issues

Read more

Reminder: I’m offering a limited-time 50% discount on the annual plan:

50% OFF FOREVER

Once you claim it, the discount will be applied forever.

Now, with only $5/month, you will have access to:

• 200+ deep-dive data engineering articles

• practice-spark: 65 LeetCode-style problems to practice Spark SQL/DataFrame

• learn-spark/dbt/airflow: CLI tools to master Spark/dbt/Airflow

If you’re a Vietnamese user, please DM me for an upgrade due to payment issues

Intro

Over the last 3–4 years, several key terms have gained popularity.

Some I’ve had the chance to learn and apply; others still leave me skeptical.

One of them is the “data contract”.

In this article, I want to look into this concept to understand what it is, its motivation, and how it would be applied.

Note that I don’t have any prior experience with data contracts, so I’m approaching the topic with a bit of skepticism, based purely on my experience and observation.

So, enjoy :d

Context

I found the term “data contract” the same way you guys probably do, scrolling LinkedIn and seeing it stands next to words like “data mesh” or “data product”.

“Is it a thing? Is it legit? “

—

Software engineers have had API contracts for a long time.

It’s an agreement that hides complex, internal things and only exposes what the API consumer needs.

By doing it this way, the producer can change things behind the scenes without any worry about breaking the interface for anyone who depends on them.

Sound very like the object oriented huh?

—

Andrew Jones is widely recognized for introducing the “data contract” in 2021, while building the data platform at GoCardless, treating data as a product with explicit interfaces to manage data quality, ownership, and governance.

His point is that almost every data platform begins with pulling data out of an organization's OLTP databases, where the data was designed for the company applications. This means we (usually data engineers) have to put a lot of effort into adapting it into something useful for analytical reporting or modeling.

Disasters happen when changes are made to the source without the consuming team being fully aware. Breaking changes and data migrations are part of any backend application lifecycle.

Because the data is organized to serve the applications, the engineers responsible for making these changes may not understand how they affect the way the data is used in the OLAP realm.

He proposes to stop exposing the internal shape of the data inside the source system. Instead, design a separate, deliberate structure for what we hand downstream, one that stays stable even when the source changes underneath it.

So when something changes in the source, we don’t just expose the change as-is; we produce an interface designed for consumption, decoupled from how that data is represented internally.

Then treat that downstream-facing structure with the same discipline software engineers already apply to customer-facing APIs: document it, version it (and try not to break it)

What is it?

Set aside the context; what is a data contract, at the most basic level?

A data contract is an agreement between data producers and data consumers.

It has details that link the business side (the logical meaning of the data) with the technical side (how it's physically implemented), plus extra metadata like data quality rules, SLAs, and expected behavior.

—

Instead of a contract between two services, a data contract sits between whoever generates the data and whoever consumes it.

It’s a promise about well-modeled, high-quality, trusted data, and the producers try hard to provide these properties.

—

Effective data contracts typically include four things:

• Schema: field names, data types, and relationships between elements)

• Ownership: which team or individual is responsible for maintaining it. This information helps streamline communication when something breaks or needs to be discussed. (or someone to blame)

• Service level objectives: agreed-upon metrics like update frequency, acceptable error rates, and response times. This sets expectations for how data should be delivered.

• Version information: how the contract has changed over time

It’s essentially a digital record — it could be a JSON or YAML file.

—

It's just a digital record to … record the promise; so, how could it be enforced?

How we implement it

Reminder: I’m offering a limited-time 50% discount on the annual plan:

50% OFF FOREVER

Once you claim it, the discount will be applied forever.

Now, with only $5/month, you will have access to:

• 200+ deep-dive data engineering articles

• practice-spark: 65 LeetCode-style problems to practice Spark SQL/DataFrame

• learn-spark/dbt/airflow: CLI tools to master Spark/dbt/Airflow

If you’re a Vietnamese user, please DM me for an upgrade due to payment issues

The data contract itself doesn't enforce anything.

It's just the source of truth.

The enforcement happens in the actions around it.

It’s mostly found in:

• CI checks that validate the contract before deployment: contract violations block the deployment.

• Production validation that runs against the live dataset: confirming schema, quality, and SLA commitments are genuinely being met once data is actually flowing (what pre-deployment checks missed)

• Consumer-side checks: critical consumers run their own validation on top of everything, confirming what they’re actually receiving matches what the contract promised.

Essentially, it’s schema checking, plus other promises. Remember from the previous section: a contract isn’t only a schema; it also carries ownership and service level objectives, like update frequency and acceptable error rates.

So enforcement has to check more than “are the field names and types right.” It has to check whether the data is actually showing up on time.

The contract standard and validation engine

The contract

As discussed, a contract is just a record.

However, it must follow some structure to allow the validation engine to access it, and more importantly, I think it should have a standard to standardize the implementation.

When searching online, the dominant one I found right now is the Open Data Contract Standard (ODCS), governed by Bitol, a Linux Foundation project. ODCS leverages YAML as its format, which makes contracts easy to version and manage. So the contract itself, the actual document with the ID, status, description, schema, and everything else, is written as a .yaml file. It’s supported by many vendors.

That YAML file is then checked for validity using a separate JSON Schema that Bitol publishes alongside the standard: a JSON Schema for ODCS is provided specifically so it can be imported into an IDE or a validator to check whether a given YAML file is a valid contract (based on ODCS).

The YAML gets checked against the JSON Schema, which defines every required and optional field, its expected type, and the rules attached to it.

In the ODCS YAML file, there are:

You can view the ODCS YAML example here.

Identity fields:

• a unique ID (to avoid naming collisions)

• a name

• a version number

• a status: “proposed,” “draft,” “active,” “deprecated,” or “retired,” so a contract’s lifecycle stage is explicit.

Description fields

• purpose: the intended use for the data

• limitations: technical, compliance, or legal restrictions on how it can be used

• usage: the recommended way to actually consume it

Dataset-related fields:

• datasets and schema: the data’s shape

• data quality: rules like completeness or accuracy checks on individual columns

• stakeholders and roles: who’s accountable, and for what part

• service-level agreements: what response time or freshness is promised

The validation engine

As discussed, a piece of YAML file doesn’t enforce anything. There must be a validation engine.

An ideal engine must validate those things:

• The contract’s format: is in the correct format?

• Schema validation: does the data still match the structure the contract promised? Field names and types are correct; nothing is renamed or dropped underneath the consumer without warning. There must be a central place to store and version the tables’ schema here.

• Data quality: does the data “behave” the way it’s supposed to? This covers things like completeness, validity (does it fall within expected values), uniqueness (no unintended duplicates), volume (roughly the expected amount arrived), or freshness (it showed up when it was supposed to).

However, there are some gaps I believe the programmatic engine cannot validate. It is the dataset’s purpose, usage, and limitations. They are free-text description fields, written for a human to read. These are more like guidelines than measurable rules.

—

Based on my imagination, the validation engine must be observed as two parts: the rule engine and the compute engine.

Contract’s format and schema validation is lightweight, as it only touches a small amount of metadata.

However, data quality is a different story.

Although some rules can be checked using metadata (e.g., updated time to check freshness), most data rules require data scanning; compute engine resources must be a concern here. Ideally, this engine is also the one your company uses for data transformation to avoid syntax mismatch.

However, this does not rule out using different compute engines for different contract validation use cases. For example, near-real-time validation could use Spark Streaming or Flink.

—

Also, the validation engine should be flexible enough to allow the user to add or adjust rules. This is where the rule engine comes into play; it stores all the rules somewhere and exposes a method to let the user interact with the rule.

The crucial thing to note is that there is no single tool called the “validation engine“, it must be a solution where the technical component depends on the organization's requirements and context.

It could be Great Expectations, dbt test, or a self-development solution for the engine rule.

It could be Databricks, BigQuery, Snowflake, Trino, Spark, Flink, or DuckDB for the compute engine.

—

Although the idea of a data contract is attractive, implementing it is not easy. The contract recording is just the tip of the iceberg. You also need to build the validation engine for it, which requires as much effort as the main data pipelines.

However, the obstacle of data contracts does not stop at the technical side.

The cultural obstacle

Data contracts are not a familiar concept like “data warehouse“, “ETL“, “data lake“ or “data quality“.

Not to mention that applying it requires time and resource investment.

Also, it can easily become something people feel like they have to do rather than something they want to do for the benefits it brings.

The tools can’t fix this; it requires mindset shifting.

How do you convince the CEO to invest time and resources in a data contract?

How do you convince the data producer to follow through on the “promises”?

How do you convince consumers to validate the dataset based on the contract?

These actions give people more work to do and add more time to the overall raw-to-insight process. That takes a lot of education and alignment, which involves many people. And whenever something involves a lot of people, it’s always a hard mission.

—

I’m writing this section to identify the culture obstacle, not to solve it. It’s more about bringing awareness. That said, I’d guess one of the first steps is starting with the organization’s current, specific problem, and elaborating on how a data contract would actually help with that.

An MVP built around a data contract can help catch issues that would have already happened before they reach production. That kind of concrete.

Outro

In this article, I tried to learn and research data contracts, a fairly new concept in data engineering. We started with its context, what it is, and how to implement it with the contract encoding and the validation engine. Finally, I laid out some thoughts about the obstacles to data contract implementation; besides the technical, mindset shifting is also a big one.

Thank you for reading this far; see you in my next article.

Reference

[1] Andrew Jones, Data Contracts (2021)

[2] Andrew Jones, A Guide to Data Contracts with Andrew Jones (2025)

Reminder: I’m offering a limited-time 50% discount on the annual plan:

50% OFF FOREVER

Once you claim it, the discount will be applied forever.

Now, with only $5/month, you will have access to:

• 200+ deep-dive data engineering articles

• practice-spark: 65 LeetCode-style problems to practice Spark SQL/DataFrame

• learn-spark/dbt/airflow: CLI tools to master Spark/dbt/Airflow

If you’re a Vietnamese user, please DM me for an upgrade due to payment issues

You know what? After finishing this article, I realized it would be even better to have a companion article that walks through a hands-on project to reinforce the concepts covered here.

I'm designing a practical stream-processing project you can learn from. See you next week ;)

Intro

For a long time, batch processing has been a commodity; it's easy to get your head around, it serves many use cases, and it’s straightforward to implement.

In contrast, stream processing has been like a high-end product; only big players want and use it. It has many more concepts to learn, and it’s more complex to implement than batch processing.

—

AI has changed a lot of things.

One of those is to give organizations a more compelling reason to adopt real-time analytics.

Instant business decisions, model training with the latest data, or agent observability.

(At least that is the thing a lot of data vendors are saying now)

Low-latency insights are usually the result of stream processing, the computational paradigm where data is processed continuously.

—

In this article, I share my experience and research so you can start learning stream processing on your own.

Important note: This is a learning guideline; it does not guarantee you can successfully run a real-time analytics project in production. That requires a lot of factors and cover them not in the scope of this article.

The difference

The first thing you have to learn is the difference between batch and stream processing. Because it will define everything you learn later, from motivation and performance monitoring to fault tolerance.

—

Definition

In batch processing, you gather the data over a period and then process it in a single operation.

For example, you read and process a Parquet file (with Spark or whatever framework you prefer), and then write the results somewhere.

Or, you take a snapshot of an OLTP database, transform it with SQL, and load it into the OLAP database.

The two scenarios share a property: the data is captured in a defined boundary. It could be an hour, a day, 3 days, or a week of data.

When a system processes data in a batch, it knows the size of the data, which simplifies the processing a lot.

—

For stream processing, there isn’t a defined boundary.

It follows the nature of the data flow.

Most of the data we work with doesn’t have a natural boundary; it is unbounded: from the user interaction on the website to data from IoT sensors (unless the website is down or the sensor runs out of power).

We place artificial boundaries on the data flow so the processing can happen seamlessly (batch processing)

Data operations and fault tolerance

When a database does aggregation and joins, it comes down to looking for records with the same key. With batch processing, the searching scope is defined. This makes the system know when the searching is done, thus finishing the aggregation or join.

With stream processing, aggregation and joins have, in theory, an infinite search space, which requires us to apply techniques (windowing and watermarks; we'll be back to these later) to limit it temporarily.

Regarding fault tolerance, you reload the batch data from the source (if the batch is still available). Typically, you reprocess the entire batch, not part of it.

In stream processing, things are different: you need to know which stream position to restart processing from, and the source must be able to “replay” the data stream from that position.

Performance

In a batch, the performance of the processing is usually defined by:

• The data volume? 100GB, 1TB or 10TB

• How long does processing take? 1 hour, 2 hours or 12 hours.

Typically, these two define and identify a less straightforward factor: your required throughput.

Reminder: I’m offering a limited-time 50% discount on the annual plan:

50% OFF FOREVER

Once you claim it, the discount will be applied forever.

Now, with only $5/month, you will have access to:

• 200+ deep-dive data engineering articles

• practice-spark: 65 LeetCode-style problems to practice Spark SQL/DataFrame

• learn-spark/dbt/airflow: CLI tools to master Spark/dbt/Airflow

If you’re a Vietnamese user, please DM me for an upgrade due to payment issues

Read more

Reminder: I’m offering a limited-time 50% discount on the annual plan:

50% OFF FOREVER

Once you claim it, the discount will be applied forever.

Now, with only $5/month, you will have access to:

• 200+ deep-dive data engineering articles

• practice-spark: 65 LeetCode-style problems to practice Spark SQL/DataFrame

• learn-spark/dbt/airflow: CLI tools to master Spark/dbt/Airflow

If you’re a Vietnamese user, please DM me for an upgrade due to payment issues

Intro

I bet some of you might think like:

“AI can code better than us; why would I need to learn to code anymore?”

I thought that too. I hesitated a bit: “Is this article necessary?“

After reflecting on my journey with Python, from 6 years ago to when I delegated nearly all code writing to AI 6 months ago, I came to this conclusion: learning to code (Python) still matters, especially in this era when nearly everyone on the Internet tells you not to.

Note: All my sharing here, including my attitude toward AI coding tools, is purely from my observation and experience.

What you're reading is my approach to learning Python effectively as a data engineer. It starts with understanding Python's core concepts and syntax.

Then gradually moves on to the production-ready skills you'll actually need: debugging, testing, dependency management, code organization, object-oriented programming, working with APIs, and DataFrame libraries, using familiar data engineering tools such as Airflow and Polars.

The motivation behind knowing how to do things right

The reason I write about Python here might not be questioned much. It’s obviously the most dominant programming language in data engineering. We speak Python all the time (only less than SQL).

Python earns that position thanks to its user-friendliness and large ecosystem of libraries, which let you do anything in data engineering.

It’s easy for humans to read and understand.

The proliferation of AI coding tools seems to make writing Python code easier.

But more Python code does not mean more quality Python code.

I don’t feel comfortable handing everything to AI, although I know it could do very well if I give it enough context, requirements, and expectations. The feeling of not knowing how the code that runs a critical function works keeps me from sleeping well.

I believe we still need to engage with understanding what AI produces, not only to guarantee correctness but also to ensure scalability, readability, and maintainability.

To achieve this with Python code, we must know how to do all of that ourselves before validating the AI’s work. You still need to have an understanding of the programming language’s nature and your own experience of writing maintainable code for the AI tool.

You can say we can achieve all of that by installing some skills for the AI; one for planning, one for reviewing, one for linting,…

That’s true; that works, even well.

But again, that’s only effective if you have the mental model for how to do things right. When you rely on an AI coding tool to do things right without knowing how to do that by yourself, two things happen:

• You lose total control of what AI produces; when issues occur and features need to be added, you rely on AI. When issues occur and features need to be added again, you rely on AI another round. It just happens like that over and over again. There is no way to get back control.

  

• You will soon lose joy and motivation. For all day long, the things you need to do are enter some prompt, install some skills, and press enter 90% of the time- no thinking, no reasoning. Just tell me the difference between doing that and scrolling short videos; you’re on autopilot and rely on external factors to give you the joy.

  

What I’m trying to say is that, to achieve long-term feasibility of using AI for coding, you must know how to do the right things.

This is why I think using an AI coding tool too early in your career is not a good idea.

You need time to practice and observe on your own to gain enough experience doing the right things when drafting code. When you hand everything over to AI, you lose the opportunity. Gradually, you will rely more and more on AI; without it, you don’t know how to do things right, and as we discussed above, that’s not feasible in the long term.

—

If you pick up any truly successful case study of vibe coding, you will reach the same conclusion yourself: the people behind those cases all have experience coding themselves, and they know how to do the right things before letting AI do that.

Don’t let some stories about a CEO pitching a product that is vibe coded completely demotivate you; that CEO will certainly lose total control over the app and hire an engineer, the one who actually knows how to code, to help him.

Now, after being on the same page on whether we need to learn code or not, let’s dive into how to learn Python effectively.

Syntax and the way you write code

Reminder: I’m offering a limited-time 50% discount on the annual plan:

50% OFF FOREVER

Once you claim it, the discount will be applied forever.

Now, with only $5/month, you will have access to:

• 200+ deep-dive data engineering articles

• practice-spark: 65 LeetCode-style problems to practice Spark SQL/DataFrame

• learn-spark/dbt/airflow: CLI tools to master Spark/dbt/Airflow

If you’re a Vietnamese user, please DM me for an upgrade due to payment issues

Read more

Reminder: I’m offering a limited-time 50% discount on the annual plan:

50% OFF FOREVER

Once you claim it, the discount will be applied forever.

Now, with only $5/month, you will have access to:

• 200+ deep-dive data engineering articles

• practice-spark: 65 LeetCode-style problems to practice Spark SQL/DataFrame

• learn-spark/dbt/airflow: CLI tools to master Spark/dbt/Airflow

If you’re a Vietnamese user, please DM me for an upgrade due to payment issues

Intro

If you spend enough time in data engineering, you must already know the importance of data modeling; it might slow the team down a bit at first, but in the long term, it literally backs your entire data foundation.

The data model defines many things:

• Which entities are there? Customer? Device? Subscription Package?

• What is the relationship between them?

• Which data will be collected? Source A? Source B? Extract all the fields or just five fields?

• How to calculate a metric?

• Which constraints and business rules does this table have? Thus, it defines what quality data looks like.

• …

—

I used to link data modeling → dimensional modeling (Kimball), but in fact there are other data modeling approaches such as Inmon or Vault.

—

In this article, I want to deliver my understanding of the difference between the Kimball and Inmon approaches. The goal is to give you a mental model of what these approaches look like and their ultimate goals; from that, you can understand why a company chooses one or combines them.

Note: This article does not deep-dive into these two approaches; instead, I want to focus on the differences in how the two guide the data modeling process.

But first, we will learn about data normalization and denormalization. We will spend a long time on this; it’s a good opportunity to learn about them.

Normalization vs. denormalization

Let’s imagine we have a table with the schema:

• employee_id

• certifications, which is a list of nested fields:

  • { cert_name: "AWS Certified", issuer: "Amazon", expiry: "2027-03" }

  • { cert_name: "Scrum Master", issuer: "Scrum.org", expiry: "2026-11" }

• manager_id

• manager_name

• department.

Denormalization

This table can be considered a form of denormalization, in which it can contain nested and repeated data.

When you want to answer “which certifications expire in the next 90 days”, you can’t just filter a column for it. The dates you need are nested within the “certifications” section.

You need to unpack every row’s nested object first.

Nested data like this prevents the basic database operations (which are supposed to be easy with normalization): querying and updating a single sub-value without touching the surrounding structure.

Let’s normalize it.

Database normalization is the method used to control the relationships of tables and columns (in a table). Its purpose is to avoid data redundancy, thus guaranteeing data integrity.

The concept was first introduced by Edgar Codd in the early 1970s. He also introduced normalization forms, which are sequential, with the higher one including the characteristics of the previous one and being stricter (solving more data integrity problems).

—

In my experience as a data engineer, understanding the first three forms is enough. In the next section, we will go through each form and apply it to normalize our example data to see which problem each form solves.

“NF” stands for normal form

1NF

Let’s flatten and unnest the data:

At first, we can use the employee_id as the unique identifier for a row; after flattening, this is no longer feasible, as you can see; id 42 appears twice because this employee has two certifications: AWS and Scrum.

—

To uniquely identify a row, let’s combine the employee_id + cert_id and use them as the primary key. Each row can now be addressed.

By flattening the data and defining the primary key (in this case, the combination of employee_id and cert_id), we normalize it to 1NF.

In a 1NF table, each column holds a single, atomic value. The table has a unique primary key. It ensures that each row in the table has a unique identifier and that a column doesn’t contain mixed types or multiple values.

However, it still has data integrity problems. Back to our example, as you can see, the “department” and “manager_name” are now repeated across every cert record for the same employee.

2NF

Reminder: I’m offering a limited-time 50% discount on the annual plan:

50% OFF FOREVER

Once you claim it, the discount will be applied forever.

Now, with only $5/month, you will have access to:

• 200+ deep-dive data engineering articles

• practice-spark: 65 LeetCode-style problems to practice Spark SQL/DataFrame

• learn-spark/dbt/airflow: CLI tools to master Spark/dbt/Airflow

If you’re a Vietnamese user, please DM me for an upgrade due to payment issues

Read more

Reminder: I’m offering a limited-time 50% discount on the annual plan:

50% OFF FOREVER

Once you claim it, the discount will be applied forever.

Now, with only $5/month, you will have access to:

• 200+ deep-dive data engineering articles

• practice-spark: 65 LeetCode-style problems to practice Spark SQL/DataFrame

• learn-spark/dbt/airflow: CLI tools to master Spark/dbt/Airflow

If you’re a Vietnamese user, please DM me for an upgrade due to payment issues

Intro

In production environments, whether you process terabytes of data or serve 10,000 user requests per second, optimization ultimately comes down to utilizing your hardware resources as efficiently as possible to achieve the desired results.

To do that, we must understand how things work internally, especially how CPUs are utilized.

—

In this article, I outline my understanding of multithreading and multiprocessing and discuss them in the context of Python, the most widely used programming language for data engineers.

This article is based purely on my understanding. If you notice anything incorrect or outdated, feel free to leave a comment.

Process and thread

A process is an independent running program.

When you launch a process, the Operating System (OS) gives the process a set of separate resources that it doesn’t need to share with others, including its own piece of memory space.

A process never touches RAM directly.

It works with addresses the OS hands it, and the OS decides which piece of physical memory each one points to. So two processes can have data sitting next to each other in RAM and still not be able to access each other’s data. Neither one has an address that points into the other’s memory.

If they want to talk, they have to communicate via an intermediary, the OS, with mechanisms like pipes, sockets, shared memory segments, or message queues.

This is called inter-process communication (IPC): data sent from one process must be serialized, passed through an OS-managed channel, and deserialized on the receiving process.

The exception is when you use shared memory; the OS maps the same physical memory pages into both address spaces.

Process A writes to its pointer, process B reads from its pointer, and they're the same bytes in RAM. No de/serialization is needed. The challenge is to ensure that the shared memory is synchronized to prevent a race condition.

A race condition occurs when the result depends on the order in which two actors happen to touch the same variable; one can overwrite the other’s work, or read a value that’s only half updated.

A thread lives inside a process.

A single process can have multiple threads, and all of them share that process resource, including the memory space: the same heap or the same global variables.

These properties make sharing between threads cheap, as communication incurs less overhead than cross-process communication.

Also, creating a thread is lighter than creating a process.

—

In short, processes give you isolation and safety (two processes can’t see each other’s data → higher security). However, they are expensive to create and talk between.

Threads live inside a process. They share resources. They are cheap to create and to talk between. However, it’s unsafe by default (two threads can edit the same variable at the same time; require a mechanism to coordinate, such as locking)

CPU cores

A core is an individual processing unit inside a CPU. It fetches instructions and does the actual computing. A core runs one thread at a time.

Multiprocessing and Multithreading

Reminder: I’m offering a limited-time 50% discount on the annual plan:

50% OFF FOREVER

Once you claim it, the discount will be applied forever.

Now, with only $5/month, you will have access to:

• 200+ deep-dive data engineering articles

• practice-spark: 65 LeetCode-style problems to practice Spark SQL/DataFrame

• learn-spark/dbt/airflow: CLI tools to master Spark/dbt/Airflow

If you’re a Vietnamese user, please DM me for an upgrade due to payment issues

Multiprocessing means running multiple independent processes, each with its own memory space, that work on different parts of the same problem.

Because each process has its own memory, there’s no shared state to protect. Give them separate cores (assuming there are enough cores), and they run at the same instant, with nothing to coordinate on by default. (If you use shared memory, synchronization is still needed)

You might guess the challenge here. It’s communication.

If two processes need to work together, things are not as straightforward, as they have to go through the OS as the intermediary. Pipes, sockets, and message queues charge you per message: for serialization, copying, and deserialization.

Shared memory avoids this, but introduces a synchronization problem instead.

The sharing overhead scales with how much these processes need to communicate.

In multiprocessing, each process can run on a CPU core or even different CPUs

Some high-performance server could have multiple CPUs.

Multithreading means running multiple threads inside a single process, all sharing that process’s resources.

Because they share memory, one thread can access data from other threads; no serialization/deserialization is involved.

But that sharing is the danger.

Two threads writing to the same variable without coordination can corrupt it or read a half-updated value (a race condition). This is why every programming language with multithreading support ships some form of lock, mutex, or semaphore; mechanisms whose job is to say “only one thread touches this at a time” and to make sure the next thread actually sees what the previous one wrote.

Multithreading is cheap to set up and cheap to communicate through, but it requires effort to ensure safe data sharing.

If all are read-only, that’s fine; however, if it involves modification, race conditions need to be considered.

Concurrency and Parallelism

In short,

Concurrency is a way of organizing work so multiple things are in progress at once, even if only one of them is actually executing at any given instant. Two or more tasks start and complete in overlapping time slices, but not necessarily at the same time.

Parallelism allows multiple things to happen simultaneously. It has the same shape as concurrency: multiple things in progress at once, but adds a stricter requirement.

The executions must occur at the same time.

Multiprocessing + Multithreading and Concurrency and Parallelism

Multiprocessing gets you parallelism more easily. Each process is resource-isolated, so nothing in the code prevents the scheduler from placing them on separate cores and allowing them to run at the same time.

But “more easily” isn’t “by definition.” The number of cores is still what decides. Run 16 processes on a 4-core machine and 12 of them are waiting for a core at any given moment.

Multithreading is interesting.

Imagine we have 4 threads inside one process; they can run:

• Concurrent only: if those threads run on a single CPU core.

  

• Also in parallel: if those threads could be run on 4 separate CPU cores.

  

The OS scheduler examines every runnable thread in every process on the system and decides which thread runs on which core. A process can use as many cores as it has runnable threads at that instant, limited by the number of cores on the machine.

If you have more threads than available cores, some threads will need to run on the same core and will alternate: one thread runs for a short time, then the OS stops it to resume another, and so on.

This means that in this case, only a subset of threads achieve true parallelism at any given instant, while the rest remain only concurrent, taking turns on a shared core.

GIL

In the standard Python interpreter, CPython, there is a lock called the Global Interpreter Lock (GIL), which ensures that only one thread can execute Python bytecode at any given moment. Doesn’t matter how many cores are sitting free in your laptop. Only one thread is running Python code at a time.

Why CPython single-thread will need a dedicate article :d

A thread releases the GIL lock whenever it is waiting on something outside the interpreter: a network call, a disk read, anything that hands control to the OS and blocks.

One thing worth knowing, GIL is optional now. PEP 703 added a separate CPython build with no GIL, experimental in 3.13, officially supported in 3.14. It’s a different binary (python3.14t), you have to explicitly choose to use it. Everything in this article about GIL describes the standard build, which is still what you get by default.

Multithreading

But a released lock with nobody to catch it is wasted.

A single thread waiting for a file download releases the GIL, and since no other thread is waiting to use it, it just waits for the response as if the GIL were never a factor.

So, we bring in other threads with multithreading.

The threading module is Python’s interface for creating and managing these threads directly.

Thread(target=some_function) wraps a function as a unit of work. Then calling the start() method hands it to the OS to schedule, and you can have as many of these running “at once” as you like, each one an independent thread the OS is free to interleave with the others.

This is the mechanism that actually allows you to have multiple tasks in flight. Without it, you’re back to the single-thread loop from before, one download after another, with nothing to hand the GIL to when it’s released. Now with the threading library, the released lock can be caught by some OS thread.

The described behavior also explains why threading earns its reputation on I/O-bound work but falls apart on CPU-bound work.

I/O-bound code spends nearly all its time waiting for outside, which is exactly the state that releases the GIL. CPU-bound Python code doesn’t wait for anything, so it holds the lock the whole time.

With one exception: libraries like NumPy do their heavy work in C, which doesn’t need the interpreter, so they release the GIL while it runs.

Another observation is that multithreading is a form of concurrency; threads are run in turn.

Race conditions

But “run in turn” doesn’t mean “safe.” It’s tempting to assume that since threads never execute Python bytecode at the same instant, they can’t step on each other so race conditions never happen.

—

The GIL guarantees atomicity at the level of a single bytecode instruction, but a series of instructions is another story.

A simple counter += 1 looks atomic, but underneath, it reads the count, adds 1, and writes it back, three separate steps.

counter = 0

def increment():
    global counter
    # many instructions behind the scenes of this
    counter += 1

Besides giving up the GIL during I/O, CPython also switches threads that are busy computing.

A thread that wants the GIL waits, and after the switch interval (5 ms by default, sys.getswitchinterval()), it raises a flag asking for the lock. The running thread checks that flag between bytecode instructions and hands it over.

That check doesn’t know or care that “read, add, write” was supposed to happen in a single batch. It can land between any two of those steps.

Imagine thread A reads a counter value of 0. Before it adds or writes anything back, it checks the flag, sees thread B waiting, and hands over the lock. Thread B picks it up, runs the full counter += 1, increasing the counter to 1. Then A resumes, still holding a stale 0 from before it was paused, adds 1, and writes 1 back. Two increments happened.

The counter is now 1, not 2.

Thus, in Python multithreading, race conditions can still occur, even though threads can’t run at the same time.

—

I found that there is an optimization in Python 3.10 to the way the GIL is released.

Before 3.10, the release/acquire could occur at essentially any bytecode instruction.

After a specific 3.10 optimization, GIL release/acquire only happens at certain instructions; not every instruction is “releasable” anymore.

For example, given the previous example, the instructions to read the counter, increment it, and write it back don’t contain any eligible release instruction. Thus, the whole process of counter += 1 could happen til the end on a single thread, preventing the chance of a race condition in our case.

Caution: that only holds for this specific case. It doesn’t mean += is safe from race conditions in general.

In the scope of this article, I won’t dive into the level of bytecode.

However, saying this does not mean that the optimization in Python 3.10 completely prevents race conditions in a multithreading environment without proper synchronization mechanisms.

Async

Since we just discussed Python’s multithreading, I think we should move on to Python Async, as these two are often confused by Python users.

I will just give an overview of Python Async here, especially how it compares to multithreading.

In simple terms, async processing is a method in which a system handles other tasks without waiting for the current task to finish; when it finishes, the system can switch back (via a notification) and handle the result.

Python’s asyncio library allows users to apply the described method. Given two tasks: one downloads a file from the Internet and the other performs a simple calculation. Async allows jumping to perform a simple calculation while waiting for I/O to complete.

You might wonder: it is quite similar to multithreading, where a download thread releases the GIL, and the perform thread acquires the lock.

The difference is where the work runs. Multithreading spawns multiple OS threads and lets the OS interleave them. Async runs all your tasks on a single thread, and context switching happens within your program rather than in the OS.

• You define a set of Async tasks.

• Each task has an explicit wait point (the await keyword). But await alone doesn’t hand control back. Control only returns to the system when a task awaits something that actually suspends it: a network call, a disk read, or a timer.

• The system here is a single event loop that manages every task. It runs one task until that task suspends, then switches to another and runs it until it suspends, and so on.

• While a task is waiting, the event loop isn’t idle; it’s watching for when waiting tasks become ready (e.g., their network response arrives) and resumes the matching task exactly where it left off.

• All tasks run on a single thread. Nothing is competing for the lock.

  • Which also means a blocking call from a task stalls everything.

In short: multithreading gives you real OS-level concurrency, at OS-level cost, with interruption points you don’t control.

Async gives you concurrency at, in theory, lower overhead, as it runs all your tasks on a single thread, with interrupt points you can control.

Multiprocessing

For CPU-bound tasks, we can do multiprocessing in Python.

The multiprocessing module is Python’s interface for running CPU-bound code in parallel. When initialized, the module spins up new OS processes, each with its own resources and, importantly, its own Python interpreter.

How each process gets that interpreter varies. Python offers a few start methods. Some clone the interpreter that’s already running, which is fast. Others launch a fresh Python process from scratch: re-import your modules and rebuild your state.

There will be N separate Python interpreters, each with its own GIL, completely unaware of the others. Multiprocessing gets around the GIL by simply not sharing one.

N processes, N interpreters, and N GILs. The OS is now free to place each process’s interpreter on its own core, allowing each process to execute Python bytecode simultaneously.

However, the price for true parallelism shows up in two places:

• First, starting a process is heavier than starting a thread.

• Second, sharing the data requires extra effort. Processes can't hand objects to each other directly. Python's default is to pickle everything through a Queue or Pipe, so you pay serialization on every message. Shared memory avoids that, but only for flat data (bytes and numeric arrays), not arbitrary Python objects.

If your task can be handled faster with more CPU resources (CPU-bound), the multiprocessing module is your choice in Python.

Outro

In this article, I shared what I learned about computer threads and processes, multithreading and multiprocessing, and concurrency and parallelism. Then I discussed how multithreading and multiprocessing relate to concurrency and parallelism.

Finally, I moved on to discuss these concepts in a Python context.

Thank you for reading this far. See you in my next article.

Reminder: I’m offering a limited-time 50% discount on the annual plan:

50% OFF FOREVER

Once you claim it, the discount will be applied forever.

Now, with only $5/month, you will have access to:

• 200+ deep-dive data engineering articles

• practice-spark: 65 LeetCode-style problems to practice Spark SQL/DataFrame

• learn-spark/dbt/airflow: CLI tools to master Spark/dbt/Airflow

If you’re a Vietnamese user, please DM me for an upgrade due to payment issues

Intro

Many junior or graduate students, or even those who work in the data engineering field for 3-4 years, reach out and ask me something like this:

“I feel I lack fundamentals; I don’t know what I'm missing and don’t know where to begin “

The reasons are usually:

• “Vendors are doing a great job in abstracting the complexity behind the scenes; data engineers now just need to interact through the interface.”

• “AI could do anything, so I don’t have motivation to learn”

• “Too many things to learn““

—

I have been thinking about this problem for a while.

I wrote about the data engineering roadmap here, but I think it’s quite overwhelming for those who want to gain the fundamental knowledge fast and efficiently.

—

From first principles, data is just bytes: 0 and 1 written to your disk. When working with data, you do one of these, or a combination of them:

• Storing the data in 0 and 1.

• Bringing those 0 and 1 into RAM for CPU processing.

• Moving those 0 and 1 to other machines

That’s it.

No matter how complex the pipeline, how big the data platform, or what the business requirements look like, any data physical actions can be observed with only these three.

Thus, if one can gain knowledge of these three, I believe they would have 80% of the necessary data infra fundamentals; all other things can be learned from there.

And, to make the learning more concrete, I chose three technologies/frameworks to learn:

• Parquet

• Arrow

• Spark

In the rest of the article, I give you exactly what you should learn from Parquet, Arrow, and Spark, and how you can leverage them to scale your learning to nearly all other data systems.

In the scope of this article, I only go through each one and list out which concepts/topics you should learn so you can build your learning based from those topics; in other words these concepts have the highest ROI: when you learn them, you can definitely learn nearly anything related to data infrastructure.

Reminder: I’m offering a limited-time 50% discount on the annual plan:

50% OFF FOREVER

Once you claim it, the discount will be applied forever.

Now, with only $5/month, you will have access to:

• 200+ deep-dive data engineering articles

• practice-spark: 65 LeetCode-style problems to practice Spark SQL/DataFrame

• learn-spark/dbt/airflow: CLI tools to master Spark/dbt/Airflow

If you’re a Vietnamese user, please DM me for an upgrade due to payment issues

Read more

Reminder: I’m offering a limited-time 50% discount on the annual plan:

50% OFF FOREVER

Once you claim it, the discount will be applied forever.

Now, with only $5/month, you will have access to:

• 200+ deep-dive data engineering articles

• practice-spark: 65 LeetCode-style problems to practice Spark SQL/DataFrame

• learn-spark/dbt/airflow: CLI tools to master Spark/dbt/Airflow

If you’re a Vietnamese user, please DM me for an upgrade due to payment issues

Intro

I’ve had the chance to work with distinguished senior colleagues. They’re distinguished not because they know a lot of technical stuff (although technical knowledge is a factor).

They try to solve higher-impact problems.

You won’t find any illustrations (except for the landing image) like in my previous articles; that is intentional. I want your 100% focus on the text because things I will share with you are the one I wished someone tell me sooner.

You might also notice that I've repeated ideas I've discussed in my previous articles. I believe important things should be repeated over and over again.

Value

Companies hire us (the DEs) to help build a robust data foundation so others can leverage that to extract business insights.

Although the details of job descriptions differ, the ultimate goal will always be the same: store, process, manage, and serve data efficiently — the data foundation.

Any company hires an employee, assuming he is good at “X,” so he can help the company with that. For data engineers, we are hired because we’re good at “data engineering,” and the company believes we can use that to build their data foundation.

You’re not hired solely for your ability to write Spark jobs, debug Airflow errors, or model a star schema. You’re hired because the company believes you can leverage all those things to help them store, process, manage, and serve data.

Thus, every single task you do and every decision you make must deliver some business value (directly or indirectly).

Business value must be the number one priority.

To answer what value you could help create, you need to be aware of the true responsibility of a data engineer. Joe Reis and Matt Housley define this very well in the Fundamentals of Data Engineering: Plan and Build Robust Data Systems:

Data engineering is the development, implementation, and maintenance of systems and processes that take in raw data and produce high-quality, consistent information that supports downstream use cases, such as analysis and machine learning. Data engineering is the intersection of security, data management, DataOps, data architecture, orchestration, and software engineering.

A data engineer manages the data engineering lifecycle, beginning with getting data from source systems and ending with serving data for use cases, such as analysis or machine learning

— Joe Reis, Matt Housley.

However, simply putting all that information into your mind is not enough. Observe what you’re doing and what others do in your company; try to figure out your responsibility yourself, or consult the manager or senior colleagues.

At the end of the day, your responsibilities come down to what Joe Reis and Matt Housley defined. However, figuring them out on your own makes you much more aware of them.

The more business-value-oriented you are, the more chances you have to grow and contribute. A signal that tells you you’re on the right path: you focus more on the “boring“ things: data modeling, security, or governance.

Higher-impact problems

Reminder: I’m offering a limited-time 50% discount on the annual plan:

50% OFF FOREVER

Once you claim it, the discount will be applied forever.

Now, with only $5/month, you will have access to:

• 200+ deep-dive data engineering articles

• practice-spark: 65 LeetCode-style problems to practice Spark SQL/DataFrame

• learn-spark/dbt/airflow: CLI tools to master Spark/dbt/Airflow

If you’re a Vietnamese user, please DM me for an upgrade due to payment issues

When you understand that your responsibility and business value are the 1st priority, you will start reflecting on what you’re doing.

—

I used to measure my success by the number of tasks I finished. The more tasks I do in a day, the better I feel.

Finishing tasks matters.

It isn’t wrong; you’re even praised and paid for it.

But you have to slow down, observe, and start looking for work that matters… more.

—

When you start your career, the world is small.

An Airflow task error, a Python script that’s not working, or a slow Spark job.

You are standing on the ground, ensuring every single detail works as expected.

However, to grow and promote, you need to go “higher” to understand:

• Why you’re doing this: Some tasks add value; others don’t.

• The bigger picture: you’re not just coding a pipeline that someone else designed forever. You have to step up to design, plan, and be responsible for work that matters more.

When looking at the higher place, the problem isn’t always as straightforward as it appears in your Jira ticket. It will be more vague:

• Why don’t users trust our data?

• Finance says revenue is $1M, while marketing says it’s $1.2M — both pulled from the data warehouse. Why is there a discrepancy?

• Is there a way to improve the productivity of Spark and Airflow development on our team?

It’s vaguer than ‘how to run your Spark job’ because the problems occur at a larger scale. They no longer live only in your IDE; they involve your whole team, other teams, or even the entire company.

This explains why solving these problems creates more impact. And creating impact is good for your growth (and reputation)

When it involves more people, you have to communicate, and human language is not binary like your code; it’s fuzzy. That makes understanding and solving the problem even harder.

—

What I'm trying to say is that every task and every problem matters. Solving a Jira ticket is valuable, but some work matters more—the kind of work that has a much larger impact.

Outro

Here is the summary:

Having a mindset that prioritizes business value will motivate you to seek higher-impact problems.

I'm offering a limited-time 50% discount on the annual plan:

50% OFF FOREVER

Once you claim it, the discount will be applied forever.

Now, with only $5/month, you will have access to:

• 200+ deep-dive data engineering articles

• practice-spark: 65 LeetCode-style problems to practice Spark SQL/DataFrame

• learn-spark/dbt/airflow: CLI tools to master Spark/dbt/Airflow

If you’re a Vietnamese user, please DM me for an upgrade due to payment issues

Intro

I’ve had the chance to learn so many things through my career. From data engineering stuff like BigQuery, Spark, Airflow, and Dagster, to actually developing a backend application or a CI/CD pipeline.

I feel lucky.

However, one piece of knowledge I truly appreciate having gained early.

It’s Docker—or, more generally, the ability to containerize any application.

To continue my process of unlearning (first with Parquet here), I relearned Docker and am sharing what I discovered in this article.

The goal is not to cover every corner of Docker; instead, I aim to provide enough material to help you understand and apply what you learn here to real-world workloads.

Note: What I want to cover here is primarily containerization. However, there is currently no containerization technology more dominant than Docker, so most of this article still focuses on Docker. Whenever the opportunity arises, I'll zoom out and discuss containerization concepts more broadly.

What is it?

In the past, if a company wanted to run a new application, the IT department needed to buy a new server. Then, the developer would deploy the application on it.

However, the downside is that a server back then could only handle a single application; if another app required an isolated runtime, it must be run on a different server.

Predicting the application’s utilization is indeed a challenge: if the server doesn’t give enough resources, you must buy more hardware; if the application doesn’t use all the resources, you accept the waste, as a server can only handle one isolated application at a time.

Virtual machines fixed this.

With a hypervisor, one physical server can host multiple applications, each in its own VM, securely and in isolation.

It was a big deal. (You could imagine)

However, VMs have overhead.

Every VM needs its own full operating system, and that OS consumes CPU and RAM that could have been used for your application. The OS needs patching, monitoring, and sometimes a license fee.

Then, a container is introduced to solve the problem.

From the outside, a container looks like a VM. Your application still runs in isolation from everything else.

The difference is that it doesn’t bring its own OS; all containers on a machine share the host’s kernel. No separate OS means no OS overhead: the application can use more RAM and CPU now, and the container's startup time will be faster than a VM's.

We can’t talk about containers without mentioning Docker.

Container technology existed long before Docker. However, it was complex and not so user-friendly at first.

It wasn’t until Docker came into town. When you hear people mention “Docker”, there is a high chance they are referring to the Docker technology, not the company Docker, Inc.

Docker is the software that creates, manages, and orchestrates containers. It’s built from various tools from the Moby open-source project. Docker allows users to focus on applications without worrying much about the underlying infrastructure.

For the rest of the article, I will use “Docker“ to refer to the Docker software, not the Docker, Inc company.

Before diving into it, let’s clarify one thing: Why should data engineers learn Docker, or more generally, containerization?

Why learn Docker?

September 23rd, 2019. My first day at my first (data) job.

I had just graduated, and I walked in expecting to touch the cool stuff: Spark, Hadoop, Kafka, etc.

But life is life; nothing as expected.

My first task was to do something I had never heard of: Docker. I struggled for two weeks. Now, looking back, I realize the main reason for the struggle was skipping the “why” and jumping straight into remembering the commands and getting things running.

Almost seven years later, I can say that Docker has quietly become one of the most important skills in my career. It does nothing to make my applications, queries, or data transformations faster, but it provides software characteristics that are just as important as performance: reproducibility and isolation.

Having these two means you can do a lot of things in data/software engineering:

I’m offering a limited-time 50% discount on the annual plan:

50% OFF FOREVER

Once you claim it, the discount will be applied forever.

Grab it now and become a paid member to get access to:

• 200+ deep-dive data engineering articles

• practice-spark: 65 LeetCode-style problems to practice Spark SQL/DataFrame

• learn-spark/dbt/airflow: CLI tools to master Spark/dbt/Airflow

If you’re a Vietnamese user, please DM me for an upgrade due to payment issues

Read more

With only $7/month (billed annually), you can access all the materials you need to grow from junior → senior DE.

• 200+ deep-dive data engineering articles

• practice-spark: 65 LeetCode-style problems to practice Spark SQL/DataFrame

• learn-spark/dbt/airflow: CLI tools to master Spark/dbt/Airflow

Upgrade with 7$/month

• If you’re a student with an education email, use this 50% ANNUAL DISCOUNT

• If you’re a Vietnamese user, please DM me for an upgrade due to payment issues. As compensation for the inconvenience, you’ll get 50% OFF the annual plan.

Intro

It has been a long time since I analyzed data engineering use cases from big tech companies.

Last week, I came across an article from Figma about how they updated their data pipeline to improve data freshness. The blog includes many interesting technical details we can learn from.

So, here we are. In the rest of the article, you will read my notes and observations after reading Figma's blog post, "From multi-day latency to near real-time insights: Figma’s data pipeline upgrade."

Disclaimer, all the technical details are come from this blog post; I only deliver my understand as well as the analyzes and the observations.

About Figma

Figma is a design and collaboration platform primarily used for creating websites, mobile apps, and user interfaces.

In short, when you need to design a front-end UI, Figma is always a top choice; millions of users have trusted the platform.

And you know what? Large user bases usually come with massive amounts of data and a high demand for using that data to better serve users.

Problem

To perform analytical workloads and extract insights, Figma must find a way to sync data from OLTP databases to OLAP databases.

Their legacy synchronization process, built in 2020, simply exports data from the OLTP databases (RDS PostgreSQL) via a cron job to S3; the data is then imported into Snowflake.

However, this approach has a downside; the higher the data volume, the longer the process. Exporting with SELECT * will be slower as the table grows.

By 2023, the synchronization process took Figma roughly 6 hours to complete. The delay in synchronization means the delay in business insight delivery.

Incremental synchronization

Fully exporting the table is not a sustainable way.

Incremental synchronization is. And you might have your guess now.

It’s Change Data Capture (CDC).

When mentioning CDC, the most common and efficient approach is log-based CDC.

The method operates by reading changes directly from the database’s transaction “log”. It is implemented differently depending on the DBMS. However, the high-level idea is quite the same.

The “log“ serves as an ordered, durable, and complete record of every committed transaction. A CDC tool with a log-based reader taps into this stream of log records, parses them, and propagates them to downstream systems.

The log-based approach provides many advantages:

• Minimal Performance Impact: Because the process reads from log files rather than executing queries against tables, log-based CDC enables continuous change capture with minimal performance overhead.

• Completeness: The “log” is the ultimate source of truth for all data modifications. Log-based CDC can therefore capture all types of changes.

• Low Latency: By reading changes as they are committed to the log, this method provides near-real-time updates to consuming systems.

The most obvious downside of this method is the complexity; a lot of components to manage: the message system, the message consumer, and the data merger. The more tables that need CDC, the more management overhead there is.

In addition, it often requires specific database configurations, such as enabling logical replication in PostgreSQL. It may also require elevated system permissions to grant the CDC process access to the transaction log files, which can be a challenge in strictly controlled environments.

Back to the Figma case, they decided to go with the CDC approach, but to handle the company's data scale, they chose to build the whole solution themselves rather than buy from a vendor to achieve flexibility and scalability while saving costs.

With only $7/month (billed annually), you can access all the materials you need to grow from junior → senior DE.

• 200+ deep-dive data engineering articles

• practice-spark: 65 LeetCode-style problems to practice Spark SQL/DataFrame

• learn-spark/dbt/airflow: CLI tools to master Spark/dbt/Airflow

Upgrade with 7$/month

• If you’re a student with an education email, use this 50% ANNUAL DISCOUNT

• If you’re a Vietnamese user, please DM me for an upgrade due to payment issues. As compensation for the inconvenience, you’ll get 50% OFF the annual plan.

The new pipeline

One thing to note is that, to enable CDC, you must first have the current state of the table, the snapshot; from there, changes come as the log message will be merged with the initial snapshot to capture the change from the source table.

For the initial snapshot, Figma simply leverages RDS to export to S3.

For the CDC setup, they leverage Kafka Connect, Snowflake Connector, and Amazon Managed Streaming for Apache Kafka (MSK).

To merge delta changes, Figma built custom Snowflake stored procedures.

For any new table that requires synchronization, the following things happen:

• A new table in Postgres is automatically captured by the Figma CDC automated service; the change stream is then published to Kafka’s per-table topics.

• The table’s latest snapshot will be exported to S3.

• Then they import the data to a Snowflake table.

• Snowflake Sink Connector will consume data from the associated Kafka topic and write it to a CDC table.

• Periodically, the merge store procedure runs to merge data from the base table and the CDC table.

And that’s not all; to ensure robustness, Figma must validate the data.

For each onboarding CDC table, Figma will clone its source table and compare it against the source table. The goal is to ensure they have the exact data. The compared output is then integrated into Figma’s monitoring and alert system.

Besides scalability, data integrity is definitely Figma’s ultimate goal.

Automation

To scale, the onboarding and validation process must be automated.

For this purpose, Figma uses AWS Step Functions.

In Step Functions, there is a concept called workflows, which facilitate automated processes or orchestrate microservices.

To automate CDC processes, Figma develops two levels of automation, each with an associated set of workflows:

• First-level: this includes workflows they can trigger manually. Those workflows execute bootstrap or validation processes by providing only the table name. No manual intervention except when there are alerts: a software bug or the validation fails.

• Second-level: this includes workflows designed to trigger first-level automations based on specific conditions and schedules. After the first level is complete, the second level checks the current states to determine whether it needs to trigger another first-level automation. For example, a workflow that periodically checks for new tables to perform onboarding.

To test these automations, Figma re-onboards all tables in the staging environment every week for simulation and to detect any issues. They actually identified an issue that could cause 20 minutes of downtime if it reaches production.

With only $7/month (billed annually), you can access all the materials you need to grow from junior → senior DE.

• 200+ deep-dive data engineering articles

• practice-spark: 65 LeetCode-style problems to practice Spark SQL/DataFrame

• learn-spark/dbt/airflow: CLI tools to master Spark/dbt/Airflow

Upgrade with 7$/month

• If you’re a student with an education email, use this 50% ANNUAL DISCOUNT

• If you’re a Vietnamese user, please DM me for an upgrade due to payment issues. As compensation for the inconvenience, you’ll get 50% OFF the annual plan.

Result

By moving to incremental synchronization, Figma sees a lot of improvements:

• Data freshness: data used to be 30 hours old or more; now it is reduced to 3 hours or could be lower based on user freshness configuration.

You might wonder; because change stream arrive near real-time, why it take 3 hours, the answer lie in the merge process. Figma set the default default merge frequency to every three hours. User can override the frequency, the more frequent the merge process, the more compute cost.

• Scalability: no more periodic full exports, which means Figma can comfortably handle large tables, even if some are ten times larger than those used in the legacy synchronization process.

• Productivity: insights can now be extracted sooner; CDC processes are now automated and abstracted by the CDC team.

Outro

In this article, we see how Figma improves data freshness by implementing an incremental approach using CDC to sync data, rather than the full-table export used in the past.

The lesson here is that, at a small scale, full exporting works well, even the ideal choice given its simplicity. However, when the data grows, full exports are painful.

Incremental synchronization is inevitable. It surely helps with the data scale and freshness. However, it comes with complexity.

As you can see, Figma invested heavily in onboarding and validation processes and in abstracting and automating them via AWS Step Functions.

At the end of the day, any technical decision has its trade-offs.

Thank you for reading this far. See you in my next article.

Reference

[1] Yichao Zhao, From multi-day latency to near real-time insights: Figma’s data pipeline upgrade (2025)

Reminder: I’m offering a limited-time 50% discount on the annual plan:

50% OFF FOREVER

Once you claim it, the discount will be applied forever.

Now, with only $5/month, you will have access to:

• 200+ deep-dive data engineering articles

• practice-spark: 65 LeetCode-style problems to practice Spark SQL/DataFrame

• learn-spark/dbt/airflow: CLI tools to master Spark/dbt/Airflow

If you’re a Vietnamese user, please DM me for an upgrade due to payment issues

Intro

I caught this quote not long ago:

It is impossible for a man to learn what he thinks he already knows. — Epictetus

That quote has been living rent-free in my mind for days.

It was totally wrong to me at first.

On a Saturday morning, I decided to reflect on my journey, from having been a data engineer for 6 years to being the author of this newsletter for 3 years.

I’ve gradually come to understand the meaning behind that quote.

I’ve spent a lot of time researching and learning about data engineering. Over time, I started taking my knowledge for granted. I developed the illusion that I knew everything, making me arrogant and less receptive to new ideas (“I already know that”/“It’s just a new way to talk about an old idea“)

“Sh*t, that’s bad; that will make me stop learning someday“

—

Lately, I’ve been practicing unlearning something I know (or thought I knew). That process is harder than it sounds: forget everything you know about something and take the most naive look at it, again.

—

One of the first things I pick up is the one I've talked about endlessly in this newsletter: the Parquet file format.

That’s why I’m writing this article.

Columnar storage

In 2026, the understanding of “columnar storage” is not the same as it was in 2019.

Back then, when the term was mentioned, many people thought that each column’s data was stored separately (including me).

That’s true, but only for some cases.

Recently, thanks to many high-quality blog posts, video sharing (and AI), columnar storage has been increasingly well understood.

When a file format says it stores data in columnar fashion, you must dive deeper to see how the “columnar“ is actually implemented.

—

We know that, in most cases, row-wise format is good for OLTP and bad for OLAP.

And, column-wise is good for OLAP and bad for OLTP.

This is because the two serve different kinds of workload:

• Point query vs. history scan: A backend database (e.g., PostgreSQL) usually serves queries that require only a few records at a time. A data warehouse service (e.g., BigQuery) usually serves queries that require scanning historical data with billions of rows.

  

• Whole row vs. some columns: In an OLTP system, a query usually needs to retrieve the whole row to get all the information. In an OLAP system, when GROUP BY is used on some columns, bringing all columns into memory is unnecessary.

  

• Fast write vs batch load: OLTP requires fast data writes to reflect changes in the new world as quickly as possible, since these OLTP databases are the state for many services that contribute to business flow.

  

  • For example, when a user subscribes to Netflix, the backend must persist that record so that when the user refreshes the UI, they immediately see they can access all the movies.

  • On the other hand, the dominant use case in data warehouses is batch loading; the cadence can be hourly, daily, or weekly.

• Latest state vs. history data: People usually care about the latest state of the record in the OLTP world: latest balance, latest order, latest address. In OLAP, to gain insight, the latest state alone is not enough; historical data must be retained for days, months, or years; storage space will be much larger; data must be encoded/compressed to prevent your storage bill from exploding.

  

—

OLTP databases with row-wise formats store data from the same row close together. This means that if we have 5 records, record A is stored first, then record B, then record C, and so on.

This works well when the system accesses the entire record most of the time. With the help of indexing (e.g., BTree), the system can quickly locate a record; then, it simply loads the entire record into memory for consumption.

Organizing records right after one also makes writing fast. The system first locates an available slot for new data, then writes the entire record continuously without jumping around.

Thus, you know why the row-wise format fits OLTP workload.

—

However, the row-wise format doesn’t work well with OLAP.

The whole record and fast write benefits are not attractive here; when doing aggregation that needs 3 columns on billions of rows, with a row-wise format, the system must bring all column data into memory to extract 3 columns; with a table of 100 columns, you waste bandwidth loading 97 columns. When you multiply that by a billion rows, the waste and overhead are huge.

—

Then people started thinking about storing data in a columnar format: data from each column is stored continuously and separately. This reduces I/O costs, as you can load only 3-column data now, instead of the whole record as with row-wise loading. Also, data from the same column tends to be homogeneous and repetitive, which makes it more efficient to encode and compress.

—

However, storing data in that way has some downsides.

The most obvious is that the data writes require touching multiple column segments; writing 100 columns requires the system to jump around 100 places to write all the data for a single record.

In addition, when queries involve multiple columns, the system must reconstruct the records from separate segments. The cost of this reconstruction increases with the number of columns involved in the query.

—

The hybrid format steals the best from both worlds.

The format first splits the data horizontally into portions, each containing a subset of rows. In each portion, data for each column is stored next to each other.

This format still provides the benefit of the columnar format, while keeping data from the same row close together; it is, of course, not as close as in the row-wise format, but the row’s data is now kept in the same portion, reducing the cost of writing data and reconciling a record, as data from the same record is distributed only within the horizontal portion.

—

When a format claims to be columnar, it is, 80% of the time, a hybrid format. From the internal format of DuckDB, Snowflake, BigQuery, ORC, and the one we discuss in this article: Parquet.

A small number of formats, such as ClickHouse or Redshift's internal format, are purely columnar.

In the rest of this article, I will use the hybrid and columnar formats interchangeably.

—

Parquet first groups data into “row groups,” each containing a subset of rows.

Within each row group, values from a column are stored together. Each row group’s column is called the column chunk. A chunk has a set of pages.

A page is the smallest data unit in Parquet, used for encoding and compression. There are several types of pages, including data pages (which contain the actual data) or dictionary pages (which contain dictionary-encoded values).

Reminder: I’m offering a limited-time 50% discount on the annual plan:

50% OFF FOREVER

Once you claim it, the discount will be applied forever.

Now, with only $5/month, you will have access to:

• 200+ deep-dive data engineering articles

• practice-spark: 65 LeetCode-style problems to practice Spark SQL/DataFrame

• learn-spark/dbt/airflow: CLI tools to master Spark/dbt/Airflow

If you’re a Vietnamese user, please DM me for an upgrade due to payment issues

Read more

With only $7/month (billed annually), you can access all the materials you need to grow from junior → senior DE.

• 200+ deep-dive data engineering articles

• practice-spark: 65 LeetCode-style problems to practice Spark SQL/DataFrame

• learn-spark/dbt/airflow: CLI tools to master Spark/dbt/Airflow

Upgrade with 7$/month

• If you’re a student with an education email, use this 50% ANNUAL DISCOUNT

• If you’re a Vietnamese user, please DM me for an upgrade due to payment issues. As compensation for the inconvenience, you’ll get 50% OFF the annual plan.

Intro

Uber, famous for their high-volume data processing, encounters the same Spark Out-Of-Memory (OOM) errors that many of us face in production.

I got a chance to read an article they shared about how they deal with it.

It’s very interesting, and I believe you will enjoy it as well.

So, here we are.

What you're gonna read next is what Spark OOM is, why it happens, and how Uber deals with it.

I won’t spend too much time explaining how Spark works in this article; for that purpose, you can visit my previous article here:

How Spark works?

If you want to run Spark, you must have a cluster of machines that provide the resources for the Spark cluster.

A Spark cluster is a set of JVM processes, including a Driver and Executors. Those processes run on the cluster of machines (with communication with the Cluster Manager).

Every Spark cluster is associated with a Spark application.

Below the application is the Spark job. A job represents a series of transformations applied to data: the entire workflow from start to finish. The series of transformations (e.g, filter, map…) can be triggered only by an action (e.g., show, count,…). We can say that a job is associated with an action. An application can have multiple jobs.

A job is split into different stages when a transformation requires shuffling data across partitions. (e.g., groupBy, join). A stage is a job segment executed without data shuffling.

A stage has a set of tasks. A task is the smallest unit of execution within Spark. Each stage is divided into multiple tasks, each handling a partition, a portion of data from an external source or from the upstream stage.

At a given stage, tasks can run in parallel; the parallelism depends on the executor’s CPUs. You can understand that tasks are handled in parallel in an executor using the multithreading paradigm. By default, a task is handled by an executor core (controlled by the “spark.task.cpus” setting); if the executor has 4 cores, 4 tasks can run in parallel within the executor.

With only $7/month (billed annually), you can access all the materials you need to grow from junior → senior DE.

• 200+ deep-dive data engineering articles

• practice-spark: 65 LeetCode-style problems to practice Spark SQL/DataFrame

• learn-spark/dbt/airflow: CLI tools to master Spark/dbt/Airflow

Upgrade with 7$/month

• If you’re a student with an education email, use this 50% ANNUAL DISCOUNT

• If you’re a Vietnamese user, please DM me for an upgrade due to payment issues. As compensation for the inconvenience, you’ll get 50% OFF the annual plan.

Why do OOM errors happen?

However, the executor memory cannot be clearly separated like the cores; you might be aware of this: in multi-thread processing, all threads share the memory. With task-processing in an executor, all tasks share the executor’s processing memory.

Assuming an executor with 4 cores and 8 GB of RAM available for processing and storing data, the executor can handle 4 tasks in parallel, and each task is expected to use up to 2 GB of RAM.

If a task needs 5GB of RAM to handle, it causes an OOM.

Thus, there are two main reasons why OOM errors happen:

• A task requires more memory to handle a partition.

• The memory portion of a single task shrinks.

For the first one, many factors could cause this: a task is assigned a skewed partition, the input data volume increases, an operation, such as aggregation, requires memory to hold aggregated values, and it is somehow exploited.

For the latter, the most common root cause is increasing the parallelism of an executor but keeping the memory the same (increase the spark.executor.cores but not change the spark.executor.memory)

An annoying thing is that the OOM might not be deterministic. Back to the example above, a task that handles 5GB of data causes OOM when running alongside three other 3 tasks that only need at most 2GB of memory.

There is a case when the OOM won’t happen: when the heavy task runs alone or with at most a single normal task at the same time in the executor:

• If the heavy task runs alone, all 8GB of memory is available for it, and it only needs 5GB to do well.

• If the heavy task runs with only one normal task, both only require 7GB of data, still below the 8GB cap.

If the Spark cluster has two executors, each has 4 cores, this means 8 tasks can run at the same time.

If the stage’s total tasks are not divided by 8, for example, 19 or 20, there is a point in time where the total 3 or 4 tasks are run at the same time; each executor could handle 1 or 2 tasks at a time.

If the heavy task somehow ends up in these runs, OOM won’t occur.

Spark Scheduler behavior

You might wonder, “Why doesn’t the Spark scheduler check the partition’s size to allocate it more cleverly, e.g., run the heavy task alone? “

This is because the scheduler wasn’t designed like that. It is built to maximize the Spark cluster resource utilization.

When a ready-to-run task set is available, the scheduler examines available task slots (executors’ cores) and fills them as quickly as possible, without using the partition’s size information.

Back to the above example with an executor with 4 cores and 8GB of memory. The scheduler sees 4 open slots and assigns 4 tasks. 3 tasks are normal, one of them has a skewed partition and needs 5GB on its own.

The scheduler’s assumption about resource usage is fine when all tasks require the same CPU and memory. However, when a small percentage of tasks require 3x or 5x as much memory as the rest, the assumption no longer holds.

Neither the scheduler nor the executor knows about this beforehand. So all 4 tasks start running together, sharing the 8GB memory pool. The heavy task keeps asking for more. At some point, the pool runs out. The task failed first, then the executor.

When a task fails, if the number of failures is still within the threshold (spark.task.maxFailures; default: 4), the scheduler resets the task’s state and places it in the pending queue. Later, the failed task is executed again, but the problem is not resolved if it’s still run with other tasks. If you are lucky, the task is re-run when the number of tasks is exhausted.

As I mentioned above, if you have two executors with 4 cores each, 8 tasks can run in parallel. If the total number of tasks is not divisible by 8, such as 19 or 20, some of the last tasks will run with only 3 or 4 tasks at a time instead of 8. This means that if you retry the task, there’s a high chance it will succeed.

This is why the same job can pass on Monday and fail on Thursday. A different scheduling order will give a different outcome.

That’s the root of why Spark OOM errors sometimes feel unpredictable.

How does Uber deal with OOM?

After understanding the nature of the OOM error and the scheduler behavior, let’s find out how Uber deals with it.

Uber tries to bring predictability to the table here.

They adjust the scheduler so it now tracks the tasks run on an executor. The target is to identify the memory-intensive tasks, the ones that cause OOM.

To make things simple, Uber marks all tasks that run on the failed executor as memory-intensive (although one of those tasks is the root cause)

With the target tasks in place, things are straightforward from here.

As discussed in the previous section, OOM occurs because the assumption that all tasks in an executor require the same amount of CPU and RAM no longer holds.

An executor with 4 cores and 8GB of RAM.

Three require 2GB of RAM.

One requires 5GB of RAM.

The whole executor will go down.

Dynamic Executor Core Resizing marks all these 4 tasks memory-intensive, then simply retry, with a twist here: each task will have the resources of the full executor

The three tasks that require 2GB of RAM each (of course) run fine on the 4-core, 8GB-RAM executor.

For the big one that requires 5GB of RAM, because it doesn’t need to share the executor’s resources with others now, it can have 5GB of RAM as it wishes.

Uber trades off a bit of resource utilization here in exchange for reliability: the feature doesn’t require human intervention to adjust the executor’s resources after an OOM occurs; the scheduler will automatically retry in a “smarter” way.

This helps Uber avoid relying on luck when retrying to get over the OOM; the task that caused the OOM is rerun on the executor alone, so Uber forces it now.

In return, the executor’s resource utilization is lower in Spark’s original design: back to the example above, when a task requiring 2GB of RAM runs alone on the executor, the remaining 3 cores and 6GB of RAM sit idle.

Not leveraging all of the executor's resources at a time could mean the end-to-end duration of a Spark job increases.

One thing to note here is that although memory-intensive tasks now get the full executor, they can still fail; in the example above, if the “big” task requires 10 GB instead of 5GB of RAM, the executor’s resources are still not enough.

At least Uber has identified the “big” task here and can apply optimizations, such as salting.

Outro

In this article, I share my understanding of what a Spark OOM error is, why it happens, and, finally, how Uber deals with it using a simple engineering approach while providing much greater reliability.

Thank you for reading this far.

See you in my next article.

Reference

[1] Kalyan Sivakumar, Dynamic Executor Core Resizing in Spark (2023)

As you might know, besides writing articles, I also spent time creating tools to make learning data engineering more cost-efficient and fun:

• practice-spark: 65 LeetCode-style problems to practice Spark SQL/DataFrame

• learn-spark/dbt/airflow: CLI tools to master Spark/dbt/Airflow

btw, you need to be paid subscriper to access these tools.

I created these tools based on my own observations.

However, I realized the motivation behind them should come from you guys: the readers of this newsletter :d

So, I want to hear about your challenges.

Feel free to comment on anything that's preventing you from learning data engineering effectively. Here are a few examples to get you thinking:

• I don’t have “big” data

• Too many things to learn

• The platform X is too expensive

• AI can do anything; it demotivates me

• Fake data doesn’t reflect real-world scenarios

• …

Your sharing will help me sharpen the ideas for my next tools.

I don’t promise to solve all your problems with a ton of tools here, but at least I can ensure the next ones could actually solve real problems.

Have a nice weekend.

Vu Trinh

With only $7/month (billed annually), you can access all the materials you need to grow from junior → senior DE.

• 200+ deep-dive data engineering articles

• practice-spark: 65 LeetCode-style problems to practice Spark SQL/DataFrame

• learn-spark/dbt/airflow: CLI tools to master Spark/dbt/Airflow

Upgrade with 7$/month

• If you’re a student with an education email, use this 50% ANNUAL DISCOUNT

• If you’re a Vietnamese user, please DM me for an upgrade due to payment issues. As compensation for the inconvenience, you’ll get 50% OFF the annual plan.

Intro

When it comes to the single-node processing engine, there are two strong candidates here: DuckDB and Polars.

Last time, we learned about DuckDB, and I think we should also bring Polars into the discussion to gain a more comprehensive understanding of these two popular single-node data processing systems.

In this article, we examine what Polars is, its motivation, design decisions, and use cases.

Note: Although Polars support distributed processing, this article only discusses on its original form: a single-node data processing engine.

Motivation

In 2020, Ritchie Vink was stuck at home because of the COVID pandemic. He spent time on some Rust pet projects. A few were done. But he had an itch. He observed that Rust still lacked a proper DataFrame library (as in Python with Pandas). So he tried something small: a simple Rust crate that does that. (In Rust, a crate is a unit of compilation)

From there, things got out of hand (in a positive way).

Less than a year later, Polars 0.6.5 was already beating Spark and pandas by 4 to 6x on the H2O.ai groupby and join benchmarks.

The motivation for Polars also stemmed from the hardware: Vendors kept adding cores, but tools like NumPy and Pandas still had to be single-threaded (because of Python’s design). Ritchie observed that his own laptop had 12 cores and that Pandas was using less than 10% of the available compute.

Then Polars stopped being a side project.

In 2023, Vink and Chiel Peters, colleagues for five years by then, incorporated Polars and raised $4M. At the time, Polars had around 6 million total downloads and 20K GitHub stars. And when I checked the download statistics for the last 30 days, it's nearly 60 million.

That’s how Polars was created: an initial pet project to combine the famous data manipulation interface (DataFrame), the high-performance language (Rust), and the implementation of decades of database research: logical plans, cost-based rewriting, predicate pushdown, plus all other things.

That is indeed the product-market fit in which data professionals rely heavily on the DataFrame API, but the performance of Pandas and NumPy remains a serious bottleneck.

How It Actually Works

With only $7/month (billed annually), you can access all the materials you need to grow from junior → senior DE.

• 200+ deep-dive data engineering articles

• practice-spark: 65 LeetCode-style problems to practice Spark SQL/DataFrame

• learn-spark/dbt/airflow: CLI tools to master Spark/dbt/Airflow

Upgrade with 7$/month

• If you’re a student with an education email, use this 50% ANNUAL DISCOUNT

• If you’re a Vietnamese user, please DM me for an upgrade due to payment issues. As compensation for the inconvenience, you’ll get 50% OFF the annual plan.

Read more

With only $7/month (billed annually), you can access all the materials you need to grow from junior → senior DE.

• 200+ deep-dive data engineering articles

• practice-spark: 65 LeetCode-style problems to practice Spark SQL/DataFrame

• learn-spark/dbt/airflow: CLI tools to master Spark/dbt/Airflow

Upgrade with 7$/month

• If you’re a student with an education email, use this 50% ANNUAL DISCOUNT

• If you’re a Vietnamese user, please DM me for an upgrade due to payment issues. As compensation for the inconvenience, you’ll get 50% OFF the annual plan.

Intro

This is a guest post from Minh Pham, the hardest-working and most enthusiastic data engineer I’ve ever known. Minh will share his learning on data engineering best practices via a multi-part hands-on project. This is the second part, which is about the transformation layer.

For the first part, you can read here.

If you enjoy it, please react to the article or share it so we know you like it. Plus, Minh is more motivated to write the next part. As you might know, writing an article is hard, but writing an article to guide a hands-on project is 10x harder, as you have to prepare for a lot of things.

—

This is the second part of my series in which I build end-to-end data pipelines, following best practices I learned at Insurify. In part 1, we built the ingestion pipeline - scraping airline reviews, staging them in S3, and loading them into Snowflake.

Now we pick up where that left off: transforming raw data into analytical models, setting up proper access controls for a small team, and building a CI/CD pipeline that rebuilds only what changed.

The Transformation Pipeline

Overall architecture details can be found here.

Tech stacks:

• dbt (dbt-snowflake): Transformation layer - Kimball star schema

• Snowflake: Data warehouse - all RBAC managed by Terraform

• Terraform: Infrastructure as Code - Snowflake + AWS resources

• GitHub Actions: CI/CD - slim CI on PRs, defer/favor-state CD on merge

• AWS S3: Artifact storage - manifests, run results, dbt docs

• AWS CloudFront: CDN for hosting dbt docs

• AWS IAM OIDC: Keyless authentication for GitHub Actions

• SQLFluff: SQL linting - lowercased keywords, trailing commas, explicit aliases

• Apache Airflow (Astronomer, cosmos): Orchestration

Data Model

I want to create a simple star schema model that has a fct_reviews table alongside multiple dimensions to support slicing and dicing. I also follow the Kimball process when creating this data model

• Define the business process: analyzing customer reviews

• Define the grain: each review a user submitted

• Define the dimensions: Aircraft, customer, airline, date, location

• Define the facts: review_id, is_verified, seat_type,…..

Prerequisite

I would expect you to have completed Part 1, so you have a Snowflake account with raw data loaded. You’ll also need:

• An AWS account (free tier is more than enough)

• The terraform-admin AWS CLI profile from Part 1

• Python 3.12+

• Git

• Docker (for local Airflow)

If you haven’t set up AWS or Terraform, follow the guides in Part 1, Steps 3-4.

Step 1: Clone the repo and set up your environment

The repo link is here: Clone it and set up a Python virtual environment:

git clone <https://github.com/MarkPhamm/skytrax_reviews_transformation.git>
cd skytrax_reviews_transformation

python -m venv dbt_venv
source dbt_venv/bin/activate
pip install -r requirements-dev.txt

requirements-dev.txt includes dbt-snowflake, sqlfluff, pandas, and other dev tools. The production CI only uses requirements.txt (dbt + sqlfluff).

With only $7/month (billed annually), you can access all the materials you need to grow from junior → senior DE.

• 200+ deep-dive data engineering articles

• practice-spark: 65 LeetCode-style problems to practice Spark SQL/DataFrame

• learn-spark/dbt/airflow: CLI tools to master Spark/dbt/Airflow

Upgrade with 7$/month

• If you’re a student with an education email, use this 50% ANNUAL DISCOUNT

• If you’re a Vietnamese user, please DM me for an upgrade due to payment issues. As compensation for the inconvenience, you’ll get 50% OFF the annual plan.

Step 2: Set up Snowflake Infrastructure with Terraform

Before we can run dbt, we need to create the Snowflake database, schemas, roles, users, and warehouses. Everything is defined in terraform/snowflake/main.tf. I actually find this easier than just running the create command in the UI. As more and more analysts enroll in the project, I would just need to change my Terraform file and run terraform apply.

What Terraform creates for us

The Snowflake Terraform code is split across 7 files, each handling one concern:

terraform/snowflake/
├── providers.tf     # How Terraform connects to Snowflake
├── variables.tf     # Input definitions (passwords, account info, defaults)
├── main.tf          # Shared locals (warehouse sizes, schema maps)
├── warehouses.tf    # Compute warehouses (XSMALL → XLARGE)
├── databases.tf     # Database + all schemas
├── roles.tf         # Roles + role hierarchy
├── grants.tf        # All privilege grants + ownership transfers
├── users.tf         # Users + role-to-user assignments
├── outputs.tf       # Values printed after apply (for CI/CD config)
└── terraform.tfvars # Actual secret values (never committed)

providers.tf - Snowflake connection

Tells Terraform how to authenticate with Snowflake. It uses ACCOUNTADMIN because it needs full power to create roles, users, and grants. All credentials come from variables (defined in variables.tf, values in terraform.tfvars) so nothing sensitive is hardcoded.

variables.tf - Input definitions

Declares every input Terraform needs: Snowflake org/account, admin credentials, user passwords, database name, and warehouse auto-suspend timeout. Passwords are marked sensitive = true so Terraform redacts them from all output. Variables with a default (like database_name = "SKYTRAX_REVIEWS_DB") are optional - the rest are required and must be provided in terraform.tfvars.

main.tf - Shared locals

Contains locals blocks - internal variables reused across other files. Three maps:

• warehouse_sizes - list of warehouse sizes to loop over (XSMALL through XLARGE)

• prod_schemas - the 5 production schemas that the transformer role needs access to

• dev_schemas - per-developer schemas (adding a new analyst here automatically grants them permissions via the loops in grants.tf)

warehouses.tf - Compute warehouses

Creates 5 warehouses of increasing size using a for_each loop over warehouse_sizes:

All warehouses have auto_suspend = 60 (shuts down after 60 seconds of inactivity) and auto_resume = true (wakes up when a query hits it). This keeps costs near zero when nobody is running queries.

databases.tf - Database and schemas

Creates the database SKYTRAX_REVIEWS_DB and 9 schemas inside it:

Why separate schemas for each environment? At Insurify, we follow the same pattern. Production models go to SOURCE/INTERMEDIATE/MARTS, CI runs write to STAGING (flat, gets wiped after each PR), and each developer gets their own DEV_* schema so they can run dbt locally without stepping on each other’s toes.

roles.tf - Roles and hierarchy

Creates three project-scoped roles and wires them into a hierarchy:

ACCOUNTADMIN
  └── SYSADMIN
        └── SKYTRAX_ADMIN (full control over project database)
              ├── SKYTRAX_TRANSFORMER (read/write on production schemas)
              └── SKYTRAX_ANALYST (read-only on MARTS + write on own dev schema)

The TRANSFORMER role is what dbt uses in CI/CD and production; it can create/replace tables and views across all production schemas. The ANALYST role is for humans - they can only read from MARTS and write to their own dev schema. Wiring SKYTRAX_ADMIN up to SYSADMIN follows Snowflake best practices, so all custom roles are accessible from the top.

grants.tf - Privileges and ownership

The largest file - handles all permission grants. It uses for_each loops over local.prod_schemas and local.dev_schemas, so adding a new schema to those maps in main.tf automatically propagates permissions. The grants break down into:

• Warehouse grants - TRANSFORMER gets USAGE + OPERATE, ANALYST gets USAGE only, ADMIN gets full control

• Database grants - TRANSFORMER gets USAGE + CREATE SCHEMA, ANALYST gets USAGE only, ADMIN gets all privileges

• Schema grants - TRANSFORMER gets USAGE + CREATE TABLE/VIEW on all production schemas; ANALYST gets USAGE on MARTS only + full read/write on dev schemas

• Future grants - Automatically apply permissions to any tables/views created in the future (critical for dbt, which creates new objects on every run)

• Ownership grants - Transfers ownership of all current and future tables/views in production schemas to TRANSFORMER. This is needed because dbt uses CREATE OR REPLACE, which requires ownership

users.tf - Users and role assignments

Creates 5 users and assigns each one a role:

Each user gets a default warehouse (XSMALL), default role, and default namespace (database.schema). The two service accounts (PROD_DBT and DBT_CICD) both use the TRANSFORMER role but are kept separate so you can audit who did what and revoke one without affecting the other. PROD_DBT is for the Airflow scheduler running daily production builds. DBT_CICD is for GitHub Actions - it runs in the STAGING schema during PRs and deploys to production schemas on merge.

outputs.tf - Post-apply values

Prints useful values after terraform apply - database name, warehouse names, usernames, role names, and schema names. These are handy for configuring dbt profiles, CI/CD pipelines, and BI tool connections.

Configure your variables

cp terraform/snowflake/terraform.tfvars.example terraform/snowflake/terraform.tfvars

Edit terraform/snowflake/terraform.tfvars:

snowflake_organization_name = "MYORG"
snowflake_account_name      = "MYACCOUNT"
snowflake_admin_user        = "your_username"
snowflake_admin_password    = "your_password"

prod_dbt_password        = "choose_a_password"
cicd_user_password       = "choose_a_password"
gina_analyst_password    = "choose_a_password"
vicient_analyst_password = "choose_a_password"

You can find your org and account from your Snowflake URL: https://MYORG-MYACCOUNT.snowflakecomputing.com.

Plan and Apply

cd terraform/snowflake
terraform init
terraform plan
terraform apply

Type yes when prompted

Verify in Snowflake

Log in to the Snowflake UI and verify:

• You should see SKYTRAX_REVIEWS_DB with all the schemas

• Under Admin → Users & Roles, you should see the 4 new users and 3 new roles

• Under Admin → Warehouses, you should see the 5 compute warehouses

Step 3: Set up AWS Infrastructure with Terraform

The AWS module creates resources for CI/CD artifact storage and dbt docs hosting. Everything lives in terraform/aws/.

What Terraform creates for us

The AWS Terraform code is split across 7 files, each handling one concern:

terraform/aws/
├── providers.tf     # How Terraform connects to AWS
├── variables.tf     # Input definitions (region, project name, bucket name, GitHub repo)
├── main.tf          # Shared data sources (AWS account ID)
├── s3.tf            # S3 bucket for dbt artifacts
├── cloudfront.tf    # CloudFront CDN for dbt docs hosting
├── iam.tf           # GitHub Actions OIDC provider + IAM role
├── outputs.tf       # Values printed after apply (for GitHub Secrets)
└── terraform.tfvars # Actual values (never committed)

Let me walk through what each file creates.

providers.tf - AWS connection

Tells Terraform how to authenticate with AWS. It uses the terraform-admin CLI profile (set up in Part 1) and tags every resource with Project, Environment, and ManagedBy so you can track costs and ownership.

variables.tf - Input definitions

Declares every input Terraform needs: AWS region (defaults to us-east-1), environment name, project name prefix, S3 bucket name, and the GitHub repository (in owner/repo format) for OIDC scoping. Only artifacts_bucket_name and github_repository are required - the rest have sensible defaults.

main.tf - Shared data sources

A small file that looks up the current AWS account ID via data "aws_caller_identity". This avoids hardcoding your account ID anywhere - other files reference it as data.aws_caller_identity.current.account_id.

s3.tf - S3 bucket for dbt artifacts

Creates the S3 bucket that stores three types of artifacts: S3 Bucket (skytrax-reviews-dbt-artifacts-<account_id>) - Stores three types of artifacts:

s3://skytrax-reviews-dbt-artifacts-<account_id>/
├── manifests/manifest.json       # Production state for defer/favor-state
├── run_results/run_results.json  # Last deploy results
└── docs/                         # dbt docs site (HTML + JSON)
    ├── index.html
    ├── catalog.json
    └── manifest.json

cloudfront.tf - CDN for dbt docs

Serves dbt docs globally via CloudFront CDN. Contains three resources:

• Origin Access Control (OAC) - allows CloudFront to read from S3 without making the bucket public. CloudFront signs every request to S3 using SigV4, so the bucket stays private.

• CloudFront Distribution - the CDN itself. Points to the docs/ prefix in the S3 bucket, serves index.html as the default root object, caches for 5 minutes (default_ttl = 300), and redirects HTTP to HTTPS.

• S3 Bucket Policy - two policy statements: one allows CloudFront to read any object via OAC, and one allows public read on the manifests/* prefix (so developers can curl the production manifest for local defer builds).

Why CloudFront instead of EC2? I actually built the EC2 + nginx approach first (the code is still in disabled/ec2.tf.disabled and disabled/vpc.tf.disabled).

It worked, but it cost ~$8/month, required OS patching, needed a cron job to sync from S3, and required a whole VPC setup (subnet, internet gateway, route table, security group).

CloudFront is $0 on the free tier, fully managed, instant updates, and needs only 3 Terraform resources. The EC2 approach was more educational, but for a static site like dbt docs, CloudFront is the right tool.

iam.tf - GitHub Actions OIDC and IAM role

This is the key security piece. Instead of storing long-lived AWS access keys in GitHub Secrets (which can leak and need rotation), we use OpenID Connect (OIDC) so GitHub Actions can assume an IAM role directly. The file creates three resources:

• OIDC Identity Provider - registers GitHub’s OIDC issuer (token.actions.githubusercontent.com) with AWS. The client_id_list is set to sts.amazonaws.com (the audience claim), and the thumbprint_list is GitHub’s stable TLS certificate fingerprint. This is a one-time setup per AWS account.

• IAM Role (skytrax-reviews-github-actions-role) - the role GitHub Actions assumes. Its trust policy uses a StringLike condition on the sub claim scoped to repo:<owner>/<repo>:*, so only workflows from your specific repo can assume it. No other repo can use this role.

• IAM Role Policy - grants three permissions: s3:GetObject/PutObject/DeleteObject on the artifacts bucket (for uploading manifests, run results, and docs), s3:ListBucket (for sync operations), and cloudfront:CreateInvalidation (for busting the docs cache after deploy)

GitHub Actions Runner
  │
  ├─ 1. Request OIDC token from GitHub's token endpoint
  │     (includes repo, branch, and event in "sub" claim)
  │
  ├─ 2. Call aws-actions/configure-aws-credentials
  │     (passes OIDC token + role ARN to AWS STS)
  │
  ├─ 3. AWS STS validates token against registered OIDC provider
  │     - Checks audience = "sts.amazonaws.com"
  │     - Checks subject matches "repo:MarkPhamm/skytrax_reviews_transformation:*"
  │
  └─ 4. STS returns temporary credentials (15 min default)
        (workflow can now call S3 and CloudFront APIs)

outputs.tf - Post-apply values

Prints the values you’ll need for GitHub Secrets configuration: the S3 bucket name, CloudFront distribution ID and domain name, the GitHub Actions IAM role ARN, and the OIDC provider ARN.

Configure your variables

cp terraform/aws/terraform.tfvars.example terraform/aws/terraform.tfvars

Edit terraform/aws/terraform.tfvars:

artifacts_bucket_name = "skytrax-reviews-dbt-artifacts"
github_repository     = "MarkPhamm/skytrax_reviews_transformation"

The bucket name will automatically be suffixed with your AWS account ID.

Plan and Apply

cd terraform/aws
terraform init
terraform plan
terraform apply

After it finishes, Terraform outputs the values you'll need for GitHub Secrets:

Outputs:

github_actions_role_arn       = "arn:aws:iam::XXXXXXXXXXXX:role/skytrax-reviews-github-actions-role"
artifacts_bucket_name         = "skytrax-reviews-dbt-artifacts-XXXXXXXXXXXX"
cloudfront_distribution_id    = "XXXXXXXXXXXX"
cloudfront_domain_name        = "XXXXXXXXXXXX"

Save these - we'll need them in Step 5.

Step 4: Run dbt locally

Now the Snowflake infrastructure is ready. Let’s connect dbt to it and run our models.

Set environment variables

Set these based on your Snowflake user. Each developer gets their own dev schema:

export SNOWFLAKE_ACCOUNT=your_snowflake_account
export SNOWFLAKE_USER=your_user
export SNOWFLAKE_PASSWORD=your_password
export SNOWFLAKE_ROLE=SKYTRAX_ANALYST
export SNOWFLAKE_SCHEMA=DEV_your_name  # e.g., DEV_MARK

Add these to your ~/.zshrc or a .envrc so you don’t have to set them every session.

Verify connection and run models

cd dbt
dbt deps --profiles-dir ./     # install dbt packages (dbt_utils, dbt_expectations)
dbt debug --profiles-dir ./     # verify Snowflake connection
dbt run --profiles-dir ./       # run all models (writes to your dev schema)
dbt test --profiles-dir ./      # run data quality tests

The profiles.yml lives inside the dbt/ directory and uses environment variables for all credentials - no hardcoded secrets.

Understanding the data model

The transformation follows a classic staging → intermediate → marts pattern:

Staging (stg__skytrax_reviews) - A 1:1 view on top of the raw source table. Adds a review_id via row_number(). Minimal transformation, just mirrors the source.

Intermediate (int_reviews_cleaned) - This is where the business logic lives. Null handling with coalesce(..., 'unknown'), column renaming (verify → is_verified, aircraft → aircraft_model, review → review_text), and type standardization.

Marts - The star schema following Kimball methodology:

All dimensions use dbt_utils.generate_surrogate_key for deterministic surrogate keys. The fact table joins to all 5 dimensions - with role-playing dimensions for dates (submitted vs. flown) and locations (origin, destination, transit). It also calculates average_rating across all non-null rating columns and a rating_band (bad/medium/good).

Local defer builds (against production)

Once there's a production manifest in S3 (of which there’s not yet), you can run only your changed models locally while referencing production for everything else:

mkdir -p dbt/prod_state
curl -o dbt/prod_state/manifest.json \\
  <https://skytrax-reviews-dbt-artifacts-203110101827.s3.amazonaws.com/manifests/manifest.json>

cd dbt
dbt run \\
  --select state:modified+ \\
  --defer \\
  --favor-state \\
  --state prod_state \\
  --profiles-dir ./

This is the same pattern the CD pipeline uses - only rebuild what you changed, reference production for everything else. The manifest is publicly readable from S3 (the bucket policy allows s3:GetObject on the manifests/* prefix).

However, at this time, since we haven’t even set up the CI/CD pipelines just yet, there would be no artifacts, nor would an S3 bucket exist. In this case, you can just cd into the dbt dir and run dbt_run, which will run all your models locally and output to your dev_schema.

Step 5: Set up CI/CD with GitHub Actions

This is where everything comes together. We have two workflows:

• deploy_main.yml - Continuous Deployment on merge to main

• pr_checks.yml - Continuous Integration on pull requests

Configure GitHub Secrets

Go to your GitHub repo → Settings → Secrets and variables → Actions. Add these secrets:

dbt-ci-init/action.yml - reusable in both CI and CD

action.yml is a shared function that both workflows call to avoid repeating the same setup 7+ times. Every job in pr_checks.ymland deploy_main.yml runs on a fresh VM — nothing is pre-installed. So every job needs to:

1. Install Python

2. Create a venv and pip install dependencies

3. Run dbt deps

4. (Optionally) download artifacts from a previous job

Instead of copy-pasting those 15+ lines into every single job, you write it once in action.yml and call it with one line:

name: ci-init
uses: ./.github/actions/dbt-ci-init

# =============================================================================
# dbt CI Init -- Reusable Composite Action
# =============================================================================
# This is a shared "function" that every job calls to avoid repeating
# Python/venv/dbt-deps setup 7+ times across the workflow.
# Callers invoke it with:  uses: ./.github/actions/dbt-ci-init
# =============================================================================

name: "dbt CI Init"
description: "Checkout, Python venv, dbt deps, and optional artifact downloads."

# ---------------------------------------------------------------------------
# Inputs (parameters callers can pass in -- all optional with defaults)
# ---------------------------------------------------------------------------
inputs:
  python_version:
    description: "Python version for the venv"
    default: "3.12"
    required: false
  download_base_manifest:
    description: "Download base_manifest artifact to ./base_state"
    default: "false"
    required: false
  download_changed_models:
    description: "Download changed_models artifact"
    default: "false"
    required: false
  download_prod_manifest:
    description: "Download prod_manifest artifact to ./prod_state (for dbt clone)"
    default: "false"
    required: false

# ---------------------------------------------------------------------------
# Steps -- runs directly in the caller's job (composite, not Docker)
# ---------------------------------------------------------------------------
runs:
  using: "composite"
  steps:
    # Install Python (needed because dbt is a Python tool).
    - name: Setup Python
      uses: actions/setup-python@v5
      with:
        python-version: ${{ inputs.python_version }}

    # Create a virtual environment and install all Python packages from requirements.txt
    # (dbt-snowflake, sqlfluff, etc.). shell: bash is required in composite actions.
    - name: Create venv and install dependencies
      shell: bash
      run: |
        python -m venv dbt_venv
        source dbt_venv/bin/activate
        pip install --upgrade pip
        pip install -r requirements.txt

    # Install dbt packages declared in packages.yml (e.g. dbt_utils).
    # --profiles-dir ./ tells dbt to find profiles.yml in the dbt/ directory, not ~/.dbt/.
    - name: dbt deps
      shell: bash
      working-directory: ./dbt
      run: |
        source ../dbt_venv/bin/activate
        dbt deps --profiles-dir ./

    # --- Conditional artifact downloads ---
    # Artifacts are files uploaded by Job 1 (setup_env) and downloaded by later jobs.
    # Each job runs on a fresh VM, so artifacts are how jobs share data.

    # Download the list of changed model names (text file from Job 1).
    - name: Download changed_models artifact
      if: inputs.download_changed_models == 'true'
      uses: actions/download-artifact@v4
      with:
        name: changed_models

    # Download the merge-base manifest for state:modified comparison.
    - name: Download base_manifest artifact
      if: inputs.download_base_manifest == 'true'
      uses: actions/download-artifact@v4
      with:
        name: base_manifest
        path: dbt/base_state

    # Download the production manifest so dbt clone knows where prod tables live.
    - name: Download prod_manifest artifact
      if: inputs.download_prod_manifest == 'true'
      uses: actions/download-artifact@v4
      with:
        name: prod_manifest
        path: dbt/prod_state

CI

Link to the CI pipeline here.

When you open a PR against main, pr_checks.yml runs. This is slim CI - it only lints, compiles, runs, and tests the models you actually changed. This is intentional, cause when dbt model scales with a long-running model, the CI process can significantly lower development time. The CI process will have its dedicated dbt-project.yml

The CI pipeline has 7 sequential jobs:

• Setup & Detect Changes: It computes the merge-base SHA between your PR branch and main, checks out the base code, runs dbt parse on both, and uses dbt ls --state base_state --select state:modified state:new to find which models changed. The merge-base approach is important - it compares your PR against the point where your branch diverged from main, not against the latest main. This avoids false positives from other PRs that merged while you were working.

  • It also detects macro changes - if a macro file changed but no models were detected as modified, it selects all models (since macro changes can affect any dependent model).

• Lint SQL - Runs sqlfluff lint on only the changed .sql files. Uses git diff --name-only against the merge-base to find them. Our linting rules (configured in setup.cfg) enforce lowercased SQL, trailing commas, explicit column aliases, and shorthand casting (:: instead of CAST()).

• Compile Changed Models - Runs dbt compile on only the changed models. This catches Jinja errors, missing refs, and syntax issues without hitting Snowflake.

• Run Changed Models - Actually executes the changed models against Snowflake in the STAGING schema. Before running, it uses dbt clone to create zero-copy clones of production tables in the staging schema, so that unchanged upstream models remain available without rebuilding.

• Test Changed Models - Runs dbt tests on the changed models in the STAGING schema.

• Run Downstream Models - Runs all models downstream of the changed ones (using the model_name+ selector), excluding the changed models themselves (already run in Job 4).

• Test Downstream Models - Tests all downstream models to make sure your changes didn't break anything further down the DAG.

CD

Link to the pipeline here.

When you merge a PR to main, the CD pipeline kicks off. Here’s the flow:

• Check out code + configure AWS via OIDC and install dbt, run dbt debug to verify Snowflake connection

• Download production manifest from S3 (if exists)

• Run

  dbt build --select state:modified+ --defer --favor-state --state prod_state

• dbt docs are generated → uploaded to S3 → invalidate CloudFront cache

• Upload manifest + run_results to S3 for next deploy

• Send email notification (success or failure) - for monitoring CD process

The key insight is incremental deploys. Instead of rebuilding all models on every merge, we only rebuild what changed using state:modified+ and their downstream dependencies. --defer means unchanged models reference existing production tables. --favor-state means when resolving deferred refs, prefer the production state. This saves significant Snowflake compute credits.

If this is the very first deploy (no prior manifest in S3), it falls back to a full dbt run + dbt test.

The workflow also has concurrency control - only one deploy runs at a time. If a second push happens while a deploy is running, it queues instead of cancelling. This prevents race conditions on the production manifest.

How the manifest flow works

This is the part that confused me before building it. Here's the full lifecycle:

First deploy (no manifest):
  → dbt run (full build) → dbt test
  → Upload manifest.json to S3

Second deploy onward:
  → Download manifest.json from S3
  → dbt build --select state:modified+ --defer --favor-state --state prod_state
  → Upload new manifest.json to S3

CI on PR:
  → Build merge-base manifest locally (dbt parse on main's code)
  → dbt ls --state base_state --select state:modified
  → Only lint/compile/run/test changed models

Each deploy uploads its manifest, so the next deploy can diff against it. The CI pipeline doesn't touch S3 at all - it builds its own baseline manifest from the merge-base code. This means CI is completely independent of production deploys.

Step 6: dbt Docs Hosting

dbt docs are auto-generated and hosted on CloudFront. Every merge to main triggers:

1. dbt docs generate - produces index.html, catalog.json, manifest.json

2. aws s3 sync - uploads to s3://bucket/docs/

3. aws cloudfront create-invalidation - busts the cache

In my case, the docs are at: https://d38l3fc9bckvbz.cloudfront.net

The bucket is private - CloudFront accesses it via Origin Access Control (OAC). This means nobody can bypass CloudFront and hit S3 directly (except for the manifests/* prefix, which is public so developers can curl the production manifest for local defer builds).

Step 7: Set up Local Airflow (Optional)

If you want to schedule dbt runs with Airflow, the dbt-dags/ directory contains an Astronomer project using the cosmos provider.

cd dbt-dags
astro dev start

The Airflow UI is at http://localhost:8083. The skytrax_dbt_transformation DAG runs all dbt models as the PROD_DBT user via the cosmos provider - it automatically converts your dbt project into Airflow tasks.

You’ll need to set up a Snowflake connection in dbt-dags/.env via AIRFLOW_CONN_SNOWFLAKE_DEFAULT (same format as Part 1’s Airflow setup).

Outro

That’s it! You now have a fully working transformation pipeline:

• Star schema with 5 dimensions and 1 fact table, following Kimball methodology

• Slim CI that only lints, compiles, runs, and tests changed models on PRs

• Incremental CD that only rebuilds modified models and their downstream dependencies

• Keyless auth via OIDC - no static AWS credentials anywhere

• Full IaC - every Snowflake and AWS resource managed by Terraform

• Per-user dev schemas for safe local development

• Auto-hosted dbt docs on CloudFront, updated on every deploy

If you followed along from Part 1, you’ve built a complete data pipeline - from scraping raw data off the web, through S3 staging and Snowflake loading, all the way to a production-grade star schema with proper CI/CD and infrastructure as code. Everything is code, everything is reproducible, everything is version-controlled.

The repo is here: https://github.com/MarkPhamm/skytrax_reviews_transformation. Feel free to clone it, break it, and make it your own.

With only $7/month (billed annually), you can access all the materials you need to grow from junior → senior DE.

• 200+ deep-dive data engineering articles

• practice-spark: 65 LeetCode-style problems to practice Spark SQL/DataFrame

• learn-spark/dbt/airflow: CLI tools to master Spark/dbt/Airflow

Upgrade with 7$/month

• If you’re a student with an education email, use this 50% ANNUAL DISCOUNT

• If you’re a Vietnamese user, please DM me for an upgrade due to payment issues. As compensation for the inconvenience, you’ll get 50% OFF the annual plan.

Intro

About 20 years ago, people thought we needed a lot of machines and complex programming abstraction to process and analyze data. Everyone wants MapReduce.

About 10 years ago, people still thought we needed a lot of machines to process and analyze data; however, the programming abstraction must be simpler. Everyone wants Spark.

Recently, people haven’t thought that way anymore.

We can now process and analyze data with only a single machine and a simpler programming abstraction.

Single-node processing tools, such as DuckDB (SQL) and Polar (DataFrame API), are receiving increasing attention.

I believe that when a medium-sized company starts building its data infrastructure, Spark is no longer the only option on its list, given that DuckDB and Polar can handle hundreds of GB of data with unfair simplicity.

—

I analyzed the rise of the single-node processing tools in the past. What I’m trying to do in this article is discuss DuckDB, a unique OLAP system introduced in 2019 that is gradually changing our thinking about “big data”.

Note: This article is NOT sponsored by DuckDB. What you're gonna read is based purely on my own research and understanding.

The motivation

DuckDB is quite unique, not only in the OLAP world but in the database world in general.

Most databases are designed on a client-server model, where the client (you) and the server (the database) are usually on separate compute instances.

DuckDB isn’t like that. It aligns with the philosophy of simplicity and embedded operation, drawing inspiration from the famous SQLite (from the OLTP world).

From the user’s point of view, DuckDB is simply an SQL interface running alongside other applications on your laptop, integrating the (powerful) analytical functionality into the host process.

In other words, when you install DuckDB on your laptop, it runs as a process, just like other applications, such as Chrome or Visual Studio Code.

—

Mark Raasveldt, co-founder of DuckDB Labs, spent his PhD solving a problem.

At a database architecture research group, surrounded by people whose entire job title started with the word “data,” and who still avoided database systems entirely.

Terabytes of CSVs sitting on disk, but they prefer scripting languages like Python or R rather than traditional databases. Mark didn’t stand it. A piece of technology with 50 years of engineering behind it: query optimizers, compression, concurrency control, all the stuff people spend entire careers on, and the people whose job is literally "data" want nothing to do with it.

The challenges are:

• Setting up a client-server database (even something as lightweight as Postgres) is quite overhead, especially with ones that only want to work with data (e.g., data scientists).

  

• Getting data in and out of the database is costly. To process data, they must ingest it into the database. To route the data to other systems, they have to extract it from the database (e.g., port the data to Pandas). However, moving (a decent amount of) data effectively is always a challenge.

  

• And, the dependencies are worth mentioning here. If you pack your analytical logic in a Jupyter notebook or a shared script, a client-server database introduces external state. Now your “self-contained” snippet of code depends on a server (database) running somewhere else.

SQLite solved some of this; it’s embedded, no server, just a library you import. But SQLite is an OLTP. Single-threaded, built for point lookups and small writes.

So product-market fit might look like this: something with SQLite’s embedded philosophy, but built for analytics from the ground up: columnar, vectorized, parallel, easy data integration.

That’s the motivation behind DuckDB.

Architecture

With only $7/month (billed annually), you can access all the materials you need to grow from junior → senior DE.

• 200+ deep-dive data engineering articles

• practice-spark: 65 LeetCode-style problems to practice Spark SQL/DataFrame

• learn-spark/dbt/airflow: CLI tools to master Spark/dbt/Airflow

Upgrade with 7$/month

• If you’re a student with an education email, use this 50% ANNUAL DISCOUNT

• If you’re a Vietnamese user, please DM me for an upgrade due to payment issues. As compensation for the inconvenience, you’ll get 50% OFF the annual plan.

Read more

With only $7/month (billed annually), you can access all the materials you need to grow from junior → senior DE.

• 200+deep-dive data engineering articles

• practice-spark: 65 LeetCode-style problems to practice Spark SQL/DataFrame

• learn-spark/dbt/airflow: CLI tools to master Spark/dbt/Airflow

Upgrade with 7$/month

• If you’re a student with an education email, use this 50% ANNUAL DISCOUNT

• If you’re a Vietnamese user, please DM me for an upgrade due to payment issues. As compensation for the inconvenience, you’ll get 50% OFF the annual plan.

Intro

In recent years, people have called for applying software engineering best practices to data engineering. From CI/CD and testing to environment separation and observability.

The motivation is understandable: if software engineering best practices could ensure software quality, we hope they could do the same for data quality.

I think that’s a great idea.

But now, we have to expand our technical breadth even more. That journey might be confusing.

To clear the mist, I wrote this article to list skills/areas of software engineering that a data engineer must equip themselves with, along with an approach to learning them effectively. Hope this makes your roadmap a bit clearer.

Before jumping into those skills, let's first understand what software engineering is.

This article is written purely based on my experience and observations. If you find I miss anything, feel free to let me know.

What is software engineering?

I have a confession.

For the first few years of my career as a data engineer, I thought software engineering just meant creating working code.

And there's nothing more to say about it; I was wrong. That thought makes me focus only on the writing code aspect and ignore everything else.

Software engineering is the discipline of building systems that keep working. Even when requirements change, when bugs arise, and when the guy who originally created them has left the company. Software engineering makes us think about maintainability, reliability, testability, productivity, and scalability.

For data engineers, seeing software engineering as more than coding is crucial. It is no exaggeration to say that software engineering practice is 90% about what works reliably, especially in contexts where even a small mistake could make people lose trust in the data we provide.

The rest of this article covers the specific software engineering skills I believe we should all have.

Writing code that other people (and future you) can understand

I said coding is not software engineering, but listing it as the first thing to learn here. ¯\_(ツ)_/¯

(Saying this might get me into a lot of trouble →) Writing code is easy.

But writing understandable code is hard. It needs time.

Writing understandable code means expressing your logic clearly enough that another engineer (or even you, six months from now) can understand, modify, and extend it without spending a whole day figuring out what the current code is doing.

I don’t think I need to discuss much about why we must write understandable code here; a very high chance that you’ve already faced a situation where you inherit someone’s code and a week later you still don’t fully understand it, or have the courage to adjust something with 100% sure it won’t break.

Writing understandable code makes you a less selfish person.

Also, it makes you become better at writing code. To do it, you must first clear your thought process, which will make you better at reasoning and expressing things.

—

How to learn?

And to achieve it, start simple. Give variables meaningful names, create functions that do one thing and modules that have a clear purpose, and add comments where code is not enough. Then, pay attention to design patterns (Python-general or data-pipeline-specific) and follow up your programming language best practices or your company’s practices.

To me, the best way to learn is by participating in the coding review process on both sides: having your code reviewed and reviewing someone’s code. This allows you to get feedback from others while learning from how they write code.

Another effective approach is to read open-source code, or even better, contribute to some open-source project. To enable collaboration, those projects must be understandable enough: you will learn a lot by the way they organize the code, naming the variable, expressing if-else, or handling edge cases.

Read it and “steal“ some of their method, if you can, contribute to the project, and you will have a code review session from the project’s maintainers, who have a lot of experience in writing understandable code.

Version Control

It’s more than Git.

With only $7/month (billed annually), you can access all the materials you need to grow from junior → senior DE.

• 200+deep-dive data engineering articles

• practice-spark: 65 LeetCode-style problems to practice Spark SQL/DataFrame

• learn-spark/dbt/airflow: CLI tools to master Spark/dbt/Airflow

Upgrade with 7$/month

• If you’re a student with an education email, use this 50% ANNUAL DISCOUNT

• If you’re a Vietnamese user, please DM me for an upgrade due to payment issues. As compensation for the inconvenience, you’ll get 50% OFF the annual plan.

Read more