I know a lot of drama is starting to go down in the Duck world, with the recent announcement that DuckLabs is getting acquired by AWS. This article is not to pontificate on that one, but I am cautiously optimistic it will turn out way better than when Oracle acquired MySQL

Instead, today’s article will demonstrate how to deploy a DuckLake catalog on EC2 getting served by Quack.

Deploying The Quack Infra to EC2

Like several of my previous articles, I’ve been on a terraform kick lately, since it’s more/less the defacto IaC that most orgs use to deploy infrastructure to the cloud. Sure, you a make an argument about Polumi and/or AWS Cloudformation. I don’t have a dog in that fight, and given I’m ‘good enough’ with terraform, that’s where I’ll stay for now. To deploy the quack server on EC2, we simply run this bash script below that executes the terraform code:

What this does is deploy the following to AWS:

• an S3 bucket to warehouse DuckLake data

• an EC2 instance running Linux that installs the DuckDB CLI 1.5.5

• The EC2 instance has a bootstrapper that installs ducklake, creates a ducklake catalog, and binds it to the AWS S3 bucket

• An AWS EBS volume attached to the EC2 instance to ensure the DuckLake catalog remains durable even in a restart/crash.

…wait a second…no python???

That’s right; we don’t need no stinkin’ python folks. A linux image with the DuckDB CLI gives us everything we need for this one!

Once this is deployed, we need to give EC2 about 5 minutes to ramp up so it’s ready to use. Let’s go look at the console to see if we are locked and loaded:

Rock and roll - status check is all green. Now let’s get cooking:

Deploy SSH tunnel

Since I don’t have a paid domain that I can use as the endpoint for my EC2 instance e.g. quack.yolomatt.com, I'm going to use an SSH tunnel to forward my local host port 9494 to the EC2 instance. To do that, I made this bash script:

Let’s go ahead and run it so we get wired up:

And Now, The Client

Now that we have the infra and the ssh tunnel active, let’s demonstrate how to actually use a client to connect to the EC2 quack server. For that, let’s launch the DuckDB CLI locally. Our connect script is as simple as this:

Notice how there is ZERO mention of DuckLake in that script. It’s now handled behind the scenes with Quack. This unlocks a ton of potential for allowing clients to interact with a DuckLake without having to worry about S3 buckets and whatnot; you will also notice there is no mention of AWS S3 secrets on this connection; again, the quack server handles that; all we need to do is just connect with our super secure token 🤣.

And now that we are connected, let’s demonstrate running some actual queries to create/insert/select data from our DuckLake:

You will notice that we are using the “remote.query” syntax here. Once DuckDB 2.0 rolled out this fall (fall 2026), that remote.query stuff will go away and instead we will be able to run much cleaner scripts like this instead:

And did our remote.query work…yes it did:

Let’s check S3 to see if the files were actually written like Quack and EC2 were configured to do so:

Wow…just wow!

The Teardown

Now that this POC is over, I’m going to nuke all the AWS assets to minimize my bill. To do so, I have simple bash script called destroy.sh:

There is just something so satisfying about running a single script to take out the trash.

Conclusion

This article demonstrated how to get a Quack server on AWS EC2 that serves up a ducklake. The key takeaways of this are:

1. clients no longer need to attach ducklakes or AWS credentials, nor have any knowledge that they are the backend

2. Quack on EC2 can be a cost efficient alternative to AWS Aurora Postgres on hosting your DuckLake catalog

3. Quack allows multiple concurrent clients to interact with and WRITE to the DuckLake…that is huge for concurrency scaling

THE CODE

Thanks for reading,

Matt

A common task I have to wrangle with is reading JSON files from an object store; And from all I have seen over the years, its actually quite memory intensive for engines like Spark and DuckDB to deserialize JSON and read it in; however, I figured I should go ahead and finally build a benchmark and compare the 2; and that’s exactly what this article will cover today.

We will be pitting 2 AWS services against each other to see which one can process the 1K JSON files the quickest and cheapest. While most of the files are 15kb in size or less, I am throwing a common wrench into this test:

• we will have a handful of the files exceed 20MB in size. The purpose behind this is two-fold - It will definitely generate some skew on the parallel processing; however, this also happens in the real world; not every file and/or process follows the happy path. You end up with some easter egg files here and there.

For the 2 AWS services we will be benchmarking, they will be ECS Fargate using DuckDB and AWS EMR Serverless using Spark. Although AWS offers Glue as another serverless spark service (try saying that phrase 5 times fast….), I ultimately chose EMR over glue to get a more “pure spark” setup.

The Fabricated Data

The fabricated data represents retail return chat conversations. Here’s a sample of what it looks like:

The actual script that creates these 1,000 files and slings them all up to S3 can be found here.

The AWS Infrastructure

If you have followed along from some of my earlier articles, you will understand why using Terraform to deploy all the infra makes sense. You can quickly spin up and spin down these things and not incur a sizable bill at the end. What our terraform will deploy is:

• AWS ECS Cluster and Task (Task is the actual ETL job)

• AWS ECR Repository + Docker Image that hosts the code

• AWS IAM roles to execute the ECS task

• AWS EMR Serverless application

• AWS IAM role to execute EMR job

• typical cloud watch log stuff

To deploy the infra, we can run this simple bash script:

And if all goes well, after about 2 minutes, we should see this:

And Now The ETL

…and this is where things might get a little subjective so strap in 🤣.

The general premise of the ETL is that it will perform the following steps:

1. Read in the raw json files from s3

2. flatten it

3. do some aggregation to get things like message counts, order counts, etc

4. write aggregate results back out to a json file on S3

Here’s a link to both the DuckDB ETL script and the Spark ETL script:

• DuckDB ETL

• Spark ETL

One of the things I try to always avoid on ETL jobs is multiple round-trips to the object store. To do so, with DuckDB, I’m creating an in-memory DuckDB database and reading all the S3 files in there once. From there, DuckDB then runs queries to flatten, aggregate, and copy out. With Spark, you don’t really get that benefit of jamming the raw data into a local database. However, Spark “should” be smart enough as it builds logical plans to know what it can cache on its workers and reuse as we build temp global views. Thus, the spark version may or may not make a few trips to S3. But it’s hard to determine unless you go really deep into the logs. Thus, we will just trust that Spark will make the smart decision here on when and how many times it wants to hit S3 vs. caching the data.

Running the Benchmark

To run a benchmark, I have a handy shell script. You can provide it a run date so when it puts the results in S3, it’s saved off in a folder by that run-date+a unique ID. To execute the shell script, we run the following in terminal:

What this does is the following:

• runs the ECS Fargate DuckDB job

• subsequently runs the EMR Serverless Spark Job

To ensure we have some consistency, I ran each benchmark 3 times so we could get multiple runs and have a good idea of the overall time. And to make that easy, I created another bash script called “run_benchmark_batch.sh” which can be found here.

The Results

Below is a summary of the results for all 3 benchmarks:

To isolate start-up times, I have in each ETL script a timer capturing the actual start of the data read from S3 to when it finishes writing the aggregate file back to S3. On average, DuckDB running on ECS fargate had an in-process time of 48.8 seconds, whereas EMR serverless on average took 167.53 seconds. No doubt the skew of the 20MB files played into causing EMR and its Spark plan to have some issues. However, it’s fascinating that DuckDB was able to just deal with all this at a remarkably faster speed.

For those wondering how much CPU/RAM I provided each service, here you go:

• ECS Fargate - 2 CPUs, 4 GB RAM

• EMR Serverless - Max 8 processors and 32GB Ram

  • That is a ceiling of what EMR can use; its serverless model will actually optimize the query plan and flex up/down the needed resources so it might not use all 32GB of RAM nor all 8 processors.

One other thing that was very interesting to watch was the start-up time for both services. ECS fargate was like clockwork at 22-23 seconds to fire up and start running. Spark had a little more variability and definitely took more time; however, to be fair, Spark is having to acquire more resources than ECS Fargate.

The Financials

Now to a more interesting aspect of this benchmark; I wanted to try and get close to exact costs for each service. To do so, I used AWS’s cost explorer (CE) and turned on our benchmark ID in the cost explorer allocation tags. AWS Cost explorer has a lag of 24-48 hours for allocating down to tags; as of this writing, it was able to get the real costs for ECS Fargate; however, the EMR jobs have not propagated the costs to CE yet, thus they are an estimate below:

• The EMR serverless cost estimate is based on the bill vCPU-hour and memoryGB-hour we were able to pull from the EMR logs.

As you can see, the cost to run ECS Fargate is significantly less than EMR serverless; so DuckDB wins on both speed and cost here.

Tearing Down the Infra

This is probably the most satisfying part of the testing. I can run a simple bash script below that nukes all the AWS infrastructure I spun up:

And if all goes well, you should see this message after about 1 minute:

Conclusion

I’m not looking to start a war here with the Spark evangelists; am I using the right tool for the right job? maybe and maybe not; the data is definitely not GB of size; however, I wanted to demonstrate that many workloads today that use Spark to process data like this can probably benefit by transitioning over to the Duck; you will get a decent speed boost and you will also make your employer happy by saving some $$$, especially if you are running hundreds or thousands of daily jobs that fit this size of data. Those 3 cent runs can add up very quickly.

Another thing to remember is you can easily run the DuckDB jobs locally from your laptop; so QA and testing scripts is much easier vs. sending spark jobs up to EMR and praying to the AWS gods that no errors occur after waiting a few minutes.

As always, the code link is below and you are welcome to pull it and test it yourself and run your own benchmarks, if you think mine are sus (learned that word from my kids recently 😄).

THE CODE

Thanks for reading,
Matt

This is a rather bold statement I’m about to make, but I think we have finally hit the holy grail of local Iceberg development on DuckDB to build genuine scripts that can be tested fast and cheap locally and then deployed up to the AWS stack (Or GCP/Azure if you’d like).

If you rewind nearly a year ago, DuckDB’s Iceberg extension just started supporting more than reading data; you could now do simple writing such as creating a bare bones table and doing inserts. At the time, this was a significant shift in the support posture DuckDB was offering for Iceberg, given DuckLabs was also rolling out DuckLake.

Side Note - I’m actually in the midst of authoring the O’Reilly book “DuckLake - The Definitive Guide” with my co-author Alex Monahan. You can get early release chapters at this link if you are interested.

However, fast-forward to today and the DuckDB Iceberg extension now supports all of the following DDL/DML syntax:

• CREATE TABLE AS (CTAS)

• MERGE INTO

• INSERT

• UPDATE

• DELETE

This is a big freaking deal. You now have the ability to author scripts that can easily read files from S3/GCS/Azure, do transformations, and then merge them to an Iceberg target, all with just using DuckDB - no spark necessary.

However, what do we do about local development?

Hello LakeKeeper+MinIO

I’ve heard of MinIO in the past, but never really gave it a try until a few months ago. MinIO basically is a simulated S3 environment, but without you having to need an AWS account. It’s simple to stand up and all runs in Docker.

LakeKeeper is an Iceberg catalog written in rust. When you marry the 2 of these together in a simple docker setup, you get the ability to build and run CRUD scripts on Iceberg tables, but locally and fast. And then, when it’s time to promote the locally tested script up to S3, you can do so by simply toggling a few environment variables. And that’s what the rest of this article will showcase and walk through.

The Docker File

First thing we will do is setup a simple docker compose file that builds our environment for MinIO+LakeKeeper. The full script is about 142 lines long, so I won’t post here, but there’s a link at the bottom of this article to it, as always. One important thing I did for this docker file was put my MinIO bucket name in a .env file; when the docker compose script runs, it will reference the .env file and pull in the minio bucket name, so that it’s not hardcoded throughout the file. The bucket name also gets reused in a a few other scripts.

The Raw Data

As like many of my prior posts, I’m going to use the TPCH dataset and load the orders table to both the MinIO local bucket, as well as the AWS S3 bucket.

The actual script to push to a bucket is as simple as this:

You will notice a “__BUCKET__” placeholder. That gets substituted for the bucket name at run time depending if we are targeting dev or prod.

The Iceberg Workload

Now that our raw dataset is squared away, this Iceberg script demonstrates what is now possible when we run DuckDB against an authenticated Iceberg warehouse. The actual script that wires up environment to target for DuckDB is in a bash script that we will talk about soon:

What this script does is the following:

1. reads from the raw source bucket, which is substituted at run time depending on if we are targeting dev (MinIO) or prod (AWS S3)

2. creates an iceberg schema

3. runs a CTAS

4. creates a temp table and then runs a MERGE statement

5. finally, demonstrates running DELETE/UPDATE/INSERT/SELECT

The script targets the full spectrum of the most common DDL/DML operations one would run on an Iceberg table, less the maintenance operations such as compaction. I’d say this is a pretty significant step for the DuckDB Iceberg Extension

Wiring Up the Environment

To wire up the environment, we first have a bash script that has a reusable function to get our targeted bucket name:

The function resolve_bucket_for_mode will get called in 2 other bash scripts:

• the raw data load

• the ETL/workload

Next, we have the bash script for the raw data load. When we run this script, we can simply pass in as a param either “dev” or “prod” depending on which environment we want to run the script on:

I think starting at line 10 is the secret sauce you’d probably care about on this post the most; it demonstrates how we setup our connection to MinIO if we are running in dev mode.

Finally, here’s the bash script to run the actual ETL:

You’ll notice in this bash script, depending on our environment mode, we will either establish and attach a LakeKeeper connection to DuckDB or an S3 Table Bucket for AWS’s managed Iceberg experience.

Alright, Let’s Run This

Now that we have all the plumbing in place, let’s go actually run this.

First let’s spin up the Docker container:

So far, so good. Now, let’s load the raw data to dev and prod. First dev:

And now the raw load to prod (AWS):

Alright, we have our raw data in both environments. Now let’s execute the actual Iceberg workload. First up - dev:

And now prod:

Wow - so easy a caveman can…wait a second…I don’t work for that company 🤣.

Let’s also go take a peak in Amazon Athena to verify the script isn’t lying to us:

Bingo! All works.

Summary

At this point, I think we have hit a special setup for running Iceberg workloads locally to test, without the need for Spark. DuckDB covers all the DML/DDL I think most would need. And toggling between dev and prod is as simple as altering the secrets and attach statement for the Iceberg catalog.

One other think you will notice is this entire project was done without a single line of python code; the DuckDB CLI has come a very VERY long way and is very powerful out of the gate. That CLI coupled with some bash - and you are good to go.

On the subject of S3Tables vs. traditional AWS glue tables - yes, traditional glue tables with Iceberg support in AWS were out well before S3Tables came about; however, I think S3Tables have had enough runway at this point to hammer out most of the kinks, and overall, it’s a more fluid and easier experience to attach and create tables managed by the S3Table service. You don’t have to specify bucket paths for every table you make, and it fits better with testing against a local MinIO+Lakekeeper setup IMO.

I hope this inspires others to realize they don’t always need that spark sledgehammer, and that they could probably save some significant $$$ going this route instead.

THE CODE

Thanks for reading,
Matt

A month ago, I published a couple articles that talked about how to deploy infrastructure and ETL in AWS for DuckDB workloads. The setup uses ECS Fargate for elastic and serverless compute; it also uses a generic ECR image so it can be bootstrapped to multiple ETL scripts, which IMO is better than deploying an image per script where the script gets bake…

Read more

Last week, I published the longest article I’ve ever written on this platform; it was a full End-To-End walkthrough on how to deploy DuckDB ETL jobs to AWS at scale; the article was well received but it also sparked some open ended questions. I was missing the following things in the article:

1. CICD bash scripts to deploy all the AWS assets

2. Demonstrating deploying multiple jobs

3. Clarity on how to remove all the assets at the end to save some $$

Today, we will answer these questions. I’d recommend though that before you proceed, you take a look at the previous article so you understand what all gets deployed and why.

Separation of Concerns

First, The Foundational Assets

The first thing we will do here is separate out the foundational infrastructure deployment from the actual ETL jobs. The rationale is simple. You need to deploy the host/server/cluster farm to hold all your jobs. You can think of this as being analogous to a databricks workspace or an Airflow instance. Thus, in our foundational deployment, we will do the following:

1. Create the ECS cluster

2. Create the ECR repository

3. Deploy the S3 buckets (Raw and Agg)

4. Create the log group in Cloudwatch

5. Create the slack webhook in the secrets manager

6. Create the IAM roles to execute ECS tasks and pull ECR images

7. Build the Docker Image (Our ETL runner) and push up/register in our ECR repository

To do this, I created this handy bash script here: deploy foundational. Here’s how it looks when we decide to start executing it:

The bash script invokes the terraform to deploy all the AWS assets as well as execute the docker image build and subsequently push it up to ECR.

Next, The Dummy Datasets

Once the foundational AWS assets are deployed, we will then run a script that now deploys 2 separate dummy data sets to S3. I’m deploying 2 because later, we will be deploying 2 separate ETL jobs that process the individual datasets. The script for this one is very straight forward and can be found here:

Side Note - The DuckDB TPCH plugin is awesome; I used to write a ton of scripts to generate dummy data, but now I just use TPCH and in 3 lines of code, I have a full suite of tables. Its very easy and time efficient.

Thirdly?? If that’s even a word - The ETL Jobs

Now that we have our foundational assets and dummy data pre-loaded, it’s time for us to deploy and schedule 2 separate ETL jobs. To make this process scalable and simple, I created a bash script that takes in the following parameters to deploy a job:

• name we want to call the job

• the location of our local python script path (the actual ETL code)

• the target s3 path we want to deploy the script to

• the cpu we want allocated to the task

• the ram we want allocated to the task

• our cron schedule

Below is an example that deploys our orders ETL pipeline. I’ve given it 2 CPU cores and 4GB of RAM and told it to fire daily on a schedule at 5am.

Here are the links to both job’s deployment bash scripts:

• Orders Job

• Customers Job

I hope now you see the scale factor of this setup; with just a few lines of code, we can deploy numerous, independent ETL jobs, each of which can have their own schedule, their own CPU/RAM config, and they all can reuse the same base docker/ECR image. Plus, they are SERVERLESS; thus they only acquire the compute when they are needed to run. This makes deploying hundreds of scripts super easy.

Here’s what the two ECS tasks look like when we deploy them:

You will notice how each has their respective script path as param. This setup is one I truly love and think you will too.

And here are both of their schedules in EventBridge:

The Local Dev/Test Experience

One of the major things I strive for when developing an ETL framework that has to get deployed to the cloud is ensuring I have a seamless local development/testing experience. I have been in numerous situations where I burned HOURS deploying jobs to the cloud; it would look something horrifying like this:

• comit code to git repository

• kick off CICD script, which runs the terraform to deploy stuff (10 minutes)

• login to web console - manually kick off jobs

• wait for jobs to complete and/or error (15 minutes)

• when they error, go scavenge through cloud watch and cloud trail logs (30 minutes)

• Find suspected bug, fix local code, redeploy (20 minutes)

And I’d end up repeating this cycle 4 or 5 times in a day in some cases. Stack that along with meetings in between, and the multiple context switching, and you can see where this is going…it ain’t a fun experience.

Thus, with this ECS setup, I wanted to locally test the scripts before sending them up to ensure I don’t have to get into that nasty debug cycle. The good news is, with the Duck, local testing against cloud assets such as S3 files is brain-dead easy; I setup separate bash scripts below to test both ETL python scripts locally:

• run customer etl locally

• run orders etl locally

Having the ability to run the ETL locally before deploying the jobs to the cloud is a true time saver. I can rapidly debug, ensure it runs correctly, and then simply deploy the scheduled tasks to AWS once I know they are correct.

And If All Goes Well, We Get Nice Slack Notifications

Here’s what happened when both jobs fired successfully on ECS:

All I gotta say about that is…

And Now The Really Fun Part…The Teardown

One of the biggest value-props of setting up this full ETL deployment with Terraform is that destroying all those assets in AWS and minimizing your bill after testing is super easy. I setup a bash script to nuke all the AWS assets and this is what it looks like when you run it:

Here’s also a link to the teardown script

• teardown script

That teardown script ran in about 30 seconds. Instead of me tediously clicking through web screens and hoping I remembered everything to delete, the script just handles it all for you.

Conclusion

I hope this updated article squares away any concerns on the modularity and scalability of this deployment pattern of AWS ETL jobs for DuckDB.

THE CODE

Thanks for reading,
Matt

This one has been a long time coming. I’ve mulled over several months if I had the patience and willingness to write this one and do a full scale tutorial. But I ultimately came to the conclusion that something like this is absolutely needed. For years now, we get sold by this idea from consultants that building ETL in AWS is “easy”. This brings me to this somewhat practical representation of reality:

Memes aside, this article is going to walk us through a full build out of deploying a modular, scalable, and reusable ETL infra in AWS using ECS Fargate and DuckDB.

What is ECS Fargate?

ECS stands for “Elastic Container Service”; the “Fargate” part is just some random word that means more/less ‘serverless’. So why would we want to deploy a DuckDB ETL script in a docker container instead of something like AWS Glue? A few reasons:

1. Glue is purpose-built for spark jobs; it will spin up a cluster of compute resources, which does not fit the bill for DuckDB and single-node execution; it is also generally more expensive because of this

2. Glue shell jobs (their single node execution service for python scripts) can only run on python 3.9, which last time I checked was at EOL as of October 2025; additionally, the more recent versions of DuckDB (1.4 and above) I believe don’t even deploy to 3.9

3. Glue shell jobs have limited CPU and ram; with ECS fargate, we can easily toggle all of that and scale up the CPU and RAM as needed

4. Deploying additional python packages in glue is a nightmare when your org doesn’t allow you to access the PYPI repo directly

5. With Fargate, you get to control how much CPU, RAM, and ephemeral disk space you want to provision with simple parameters

6. With Fargate, you get to control the contents of the docker image including the python version and any other packages you want to bundle with it

30,000 Ft. Architectural Overview

What we will be deploying in this tutorial at the highest level is:

1. a Docker image with python 3.13, DuckDB 1.5.2, and DuckDB extensions httpfs and aws pre-installed

2. An ECS Task, which is analogous to an ETL pipeline; the task takes in a parameter of the S3 location for the script and then executes it against the docker image we have deployed up. This makes our docker image modular and reusable

3. A scheduler to run the job at a specific time of the day; this is done via AWS Event Bridge

4. An alert that posts to a Slack channel if our job succeeds or fails - this is key to monitoring and observability, which many teams neglect until they get yelled at by the business about a table that hasn’t updated in 3 weeks…

You will notice up above I put in bold “pre-installed”; in many organizations, for security reasons, they do not allow jobs in the cloud to reach out to public repos to pull python packages; thus, we will pre-install them in the docker image so at run time, they are already there.

I hope at this high level, what I’m saying makes sense. Now we will dig into the actual low level part of this.

Ground Level Architectural Overview

This part is effectively what I refer to as the “AWS Legos” and this pulls back the curtain so you understand exactly how the sausage is made. To deploy a reusable docker image up to AWS to run DuckDB ETL, this is all we will have to deploy:

1. An AWS ECS Cluster - this is just a logical grouping to hold tasks (its not like a spark cluster…no compute whatsoever)

2. An AWS ECS Task - The actual job

3. An ECR repository - to hold our docker image

4. an EventBridge Trigger - the scheduler to run the job

5. AWS Secrets manager entry for our Slack Webhook - to secure our Slack Webhook URL

6. AWS Cloud Watch Logs - to log the task as it runs; is equivalent to the stdout; helpful for debugging

7. An IAM role that can pull the ECR Image (Docker image)

8. An IAM role that can execute the ECS task

9. an IAM role to run the scheduler from EventBridge

10. The actual ETL script - gets deployed to an S3 bucket

These assets will get deployed via the terraform code for our project located here:

• TF Code

But there is something important to consider when deploying Terraform code; you actually need a separate IAM role that you have already provisioned that has the ability to deploy all those assets mentioned above. I had previously made terraform role that I called my svc-infra-role; the template for it’s IAM policy can be found here:

• IAM Infra Role Template

Additionally, I had pre-created my S3 buckets vs. having the Terraform create them; the reason behind this is 2-fold:

1. This allows me to test the script locally before deploying up to AWS; if the buckets didn’t already exist, the process would fail

2. I needed a “source” S3 bucket to pre-generate some dummy data that our ETL script could pull from.

The ETL Script

Our ETL script is pretty straightforward. It does the following steps:

1. runs an aggregation query against the source parquet file and stores the result in temp space

2. does 3 QA/validation checks: row counts, nulls, and checks where the total_price column is less than 100

   1. if any of these QA checks fail, then the process errors out and sends an error notification to slack

3. If all QA checks pass, then the final file is written to the destination S3 bucket and a success notification is sent to Slack

Here’s what the script looks like in all its glory:

The Source Dataset

For our main ETL script to run, it has to pull its source dummy data. That was a simple DuckDB script that I ran locally using TPCH dataset, which looks like this:

The Helpers Library

You’ll notice in our ETL script that we are importing a module called “helpers”. This is a common pattern that I and many others use for ETL. Many of the reusable functions like sending slack notifications, uploading files to s3, etc., I will pre-bundle with the docker image. All the code from our helpers library can be found here:

• helpers library code

The Docker Image

Our Docker Image is pretty simple. It’s the amazonlinux 2023 image which comes pre-bundled with the AWS CLI; that is important since DuckDB needs to authenticate to various AWS services using the “credential_chain”. You will see that we are also installing python 3.13 on the base image, DuckDB (via the requirements file) and adding DuckDB extensions httpfs and aws; again as a friendly reminder, we are pre-installing the extensions with the assumption that once we deploy this ETL script to AWS, our security policy prevents us from reaching outside of our AWS network to public repositories. Here’s what the image ends up looking like:

A few other important things to note here:

1. you can see we are copying our helpers folder into the main app folder. We then subsequently update the PYTHONPATH environment variable to include the app folder. This ensures that when our main ETL script imports the helpers module, it will find it and not error

2. Our command at the end is calling runner.py. This is the entry point for the container that runs a python script which does the following:

   1. takes in a param for the S3 ETL script path location

   2. uses boto3 to download that ETL script into the docker image

   3. executes the ETL script

The runner.py code looks like this:

Again, the beauty in all of this is that we can now pass into our image any DuckDB ETL script we have uploaded to S3, thus making this a reusable image.

Observability and Notifications

I setup a slack webhook to handle posting notifications of the ETL script to a slack channel; slack web hooks essentially allow you to run an http POST call to an endpoint where you can send many types of messages; i’ve setup the webhook code to handle 3 types of notifications:

• success (green)

• error (red)

• basic info (blue dot)

Below is a screenshot of how each of these types of notifications appear in Slack:

One thing to be careful about though is exposing your webhook url; if anyone gets a hold of it, they can start posting to it; thus, to secure the webhook URL, I simply loaded it to AWS’s secret’s manager as a simple key/val pair like in the screenshot below:

The actual slack notification python code simply pulls down the secured URL from secrets manager at runtime using the boto3 api.

Deployment

To deploy all of these assets, we need to run the following in order:

1. Terraform code to deploy the assets. This is done via in our terminal using terraform apply and logging into our infra aws role that I provided details on above

2. Once the terraform is deployed, we then need to build and deploy our Docker image up to our AWS ECR repository. I have a simple bash script called “build_and_deploy.sh” located here

Assuming everything was all rainbows and unicorns, we should be able to see the following assets in AWS:

The ECR Image

That image is tiny…only 133MB in size

The ECS Task

The IAM Roles

The Scheduler

Ok…This Was A Lot; Now How Do I Run This?

We can either wait for the EventBridge scheduler to kick off tomorrow morning, or we can manually start the ECS task with this simple bash script:

Like I had mentioned before, ECS Fargate is very fast at spinning up and will launch the job in a matter of about 30 seconds. And if all goes well, we will get a beautiful slack notification like the following:

We can also go check our cloud watch logs and we should see some good stdout info messages there as well:

Additionally, we can run this simple validation script that uses DuckDB to read the results that were written to our S3 target from the job:

…And congrats, you have reached the end of the monopoly board and have officially deployed a modular, scalable, reusable ETL infrastructure suite on AWS.

I Know That Was A Lot - There Is A Silver Lining

We just covered nearly everything under the sun that is required for deploying ETL to AWS that gives you a scalable and modular setup. You could have taken numerous shortcuts such as:

1. one IAM role to rule them all that has blanket access to everything

   1. this is known as “god mode” and is highly discouraged due to its security implications if it ever fell into the wrong hands

2. bake the ETL script into the image, thus eliminating the need for the launcher code that downloads the script

   1. this would make the image though built for single use at that point and no longer modular to accept any script we want to run

3. just throw all helper functions and main etl into one giant script and pop into glue

   1. You’d still have to grapple with the ability of the glue job to import python packages if you don’t have open internet access, its scaling issues, and that it runs on python 3.9.

Therefore, I think the setup we just built hits the sweet spot; and for future deployments, you will not have to deploy the Docker Image, ECR repo, nor the ECS cluster; and you might even be able to reuse some of the IAM roles that you had already deployed.

The Financials

Believe it or not, the cost for me to run and deploy this entire demo was just shy of 50 cents; of course, I did destroy the AWS assets after I finish the demo so I wouldn’t keep incurring a bill; but for what its worth, ECS Fargate tasks are relatively cheap compared to Glue; additionally, our ECR image, which was about 133MB in size, would cost less than 10 cents a month to store in ECR.

Further Musing

If you made it this far, you can see that there was a lot involved in this tutorial. And this, again, is why many companies are moving to data management platforms such as Databricks and Snowflake; they offer more seamless integrations and you can spend more time focusing on the business outcomes vs. the infrastructure. However, if you fork the code that I demonstrated today and applied a little elbow grease to fit it to your orgs standards, this will get you a simple and repeatable deployment pattern that scales very well.

ALL THE CODE

• main repo link

• docker stuff

• main etl script

• pre-gen data script

• post run validation

• helpers lib

• terraform

And What You Really Cared About? The Duck Wizard Image Prompt

Let’s face it; if you’ve gotten this far on this article, what you really cared was the prompt I gave ChatGPT to generate the wizard duck at the beginning of this article. Here it is:

Thanks for reading,
Matt

From time to time, I pay homage to the language that started it all; the one that runs most of the world still today - good ole C code. If you started off in your programming journey with a language like Python, then trying to write in C can be somewhat daunting. But once you unpack the language, it’s actually very very simple. I think where most people…

Read more

If you’ve been in data engineering long enough, you’ve probably heard the phrase “predicate pushdown” for query optimization. But what exactly does that mean? How is it getting used in query engines today? And what are the implications of this idea?

Read more

I wrote an article last week that I thought would spark some interest. You can use AWS Athena as a serverless and cheap ETL engine with nothing more than some bash scripts. I’ve actually started using this approach in some projects I’ve been working on that just need to manipulate some json files sitting in s3.

Read more

Something I’ve come to like lately is using CLI’s more than python, especially for AWS and DuckDB. There is just something gratifying about a CLI’s simplicity; With that being said, today we are going to talk about a common problem that rears its ugly head in the ETL world and how to take a ‘poormans’ ETL approach to it using AWS Athena.

The problem we …

Read more

For a while now, the DuckDB Iceberg extension has had great support for AWS S3 Tables (AWS’s managed Iceberg table experience); you can perform basic CRUD operations such as create a table, insert, update, and delete. I wrote about this a few months ago here; but for many that have been on the AWS stack for a while, you all know well that S3 Tables are …

Read more

I think it’s no secret at this point, but if you saw my post the other day or the LinkedIn announcement, you’d know that Alex and I are hard at work on the O’Reilly book titled “DuckLake - The Definitive Guide”. With that being said, when Ducklake first came out in 2025, I was amazed at how easy it was to get up and running; but like most data warehousi…

Read more

I haven’t written on Substack in over a month, and if you saw the announcement this morning via LinkedIn, you’d understand why. I’m currently working on a book with Alex Monahan that we hope to roll out by the end of the year. I’ve never written a book before so this will be a major shift in how I approach projects. I do plan to still write here from ti…

Read more

I’ve been on a roll lately with MotherDuck, who is the sponsor of today’s article:

Read more

We live in a world where nothing is ever perfect; especially in the data engineering space, we are tasked quite often with solving integration problems where we have multiple catalogs and multiple data stores floating around the ether; to further complicate things, these catalogs can also be in different clouds or managed services. I know most of you re…

Read more

We live in a world these days where we have many choices abound for data services; the 3 predominant cloud providers are no doubt AWS, GCP and Azure; and what’s really interesting about the current landscape is that many orgs find themselves in situations where they have built up apps and data stores that span across more than just one cloud; this is us…

Read more

If you have readily followed me on LinkedIn, you will notice from time to time, I drop some nuggets on my rowing journey. I would not consider myself an “elite” rower by any means, but I have logged well over 1M meters now on my Concept 2 rower and have a pretty good clue of what direction is up. Plus, the few times that I have ranked my workouts in the…

Read more

It’s been over a month since I last wrote an article. I’ve been on a bit of a hiatus, but I have some good things coming down the pipeline. Today though, we will be looking at the current state of AWS S3 Tables and Duckdb.

If you rewind the clock a few months back, Duckdb had a monumental release (1.4) which came with the ability to write to Iceberg tabl…

Read more

I’ve been on a spark config kick lately. A week or so ago, I published a post showing how to config Spark for the AWS Glue Iceberg Rest endpoint. Overall, the config is not too complex, but does require a ton of inputs. Comparing that to the duck, you see a stark contrast.

However, after working through that article, I was somewhat disappointed that the …

Read more

Duckdb has been making some major inroads over the past few months to advance their integrations with Iceberg and AWS. One can simply run the following now to map any iceberg tables from an AWS account into duckdb (no fancy s3 tables required):

Read more