We dig into Shehab’s journey from writing C++ as a kid to studying computer science at Berkeley, becoming a founder, and eventually building Sail. We talk about why Spark has remained so sticky, what Rust, Apache Arrow, and DataFusion are changing about data infrastructure, and why rebuilding Spark compatibility turned out to be much harder—and more interesting—than expected.

Most importantly, we explore LakeSail’s bigger idea: what if the data pipelines companies already have could become the foundation for their AI pipelines?

A wide-ranging conversation about Spark, Rust, AI, and where data engineering goes from here.

Open wide. Time to take your medicine; your AI medicine, that is. Remember that pink goop your mother used to shovel down your throat; you millennials know what I’m talking about, who could forget that taste of rubber. Blah. Hopefully this experience will be a little better for you, and me.

I’m trying not to be a glutton for punishment; the pace at which new tools, features, products, models, agents, sparkle-glitter-sunshine SaaS thingy-ma-bobs are lobbed over the fence at us is quite overwhelming. I breathe the same air as you, surf the same LinkedIn and Substack feeds as you, get the same FOMO as you when I see someone I respect using a new thingy-ma-bob that I’m not. What am I missing? Can I keep up? I’m giving it my best shot, week in and week out.

Well, now that I’ve moaned, lamented, and hopefully made that striking connection with you over the invisible tunnel of bits and bytes that separates us, again, I must put my poor calloused hands to the plow and throw myself into the deep end. But not just any deep end, mind you, the Agentic Meta Harness deep end. Yikes. I feel like I’m in Blade Runner 2049; I’m not sure if I’m even human anymore.

So. Meta-harness for agents, namely Omnigent.

Thanks to Delta for sponsoring this newsletter! I use Delta Lake daily, and I believe it represents the future of Data Engineering. Content like this would not be possible without their support. Check out their website below.

We need to talk about Coding Agents.

So … what you on? What do you use? What’s your poison? Have you tried this? Yeah, I’m kinda burnt out on it all too. I don’t want to have a Gas Town, I don’t want to be a loop bro. At some point, you just have to pick, like, your IDE of old, and leave the rest to argue their way into oblivion. Me? I like my good ol’ trusty vim, or if I’m feeling spicy, neovim. Now we’ve moved on to coding agents.

Codex, Claude Code, Cursor, the list goes on forever. Not only that, we have to keep up with /skills, MD files, plugins, MCP Servers, and all the rest. It’s a mess, a literal spaghetti of tools, instructions, and skills, all plugged into various and numerous tools. We want different things at different times; we want these repos to be treated one way with a certain set up rules, this other stuff over here we want it done a certain way, maybe we're doing some high-level project planning where we have a flow, could be this stuff over in the corner needs heavy control for running bash commands.

Like I said, a mess.

If we are honest with each other, we are standing on shifting ground; tools are coming and going; we are unsettled and find useful a various assembly of different plugins, skills, MD files, whatever. We have our own little “agentic workflows,” if I may call them that. I prompt this model a certain way; that one differently; I treat one codebase a specific way, and use a different one another based on past technical debt or just its nature.

This is what Omnigent solves.

It’s as easy as …

curl -fsSL https://omnigent.ai/install.sh | sh

Wala, with this simple command, you have installed your first meta-harness, a piece of software that can sit above your various LLM models, workflows, MD files, prompts, … whatever it is you are using. It’s like the One Ring of Power, in the darkenss of your terminal Omnigent will bind them.

What’s the TDLR of a meta-harness like Omnigent?

Of course, I’m only scratching the surface of what a meta-harness like Omnigent can do, but you or I want to get lost in the weeds and get distracted. Rather, let’s just install and play around with the major Omnigent features to get a sense of what we are dealing with, and maybe it will give you ideas on how such a meta-harness can fit into your workflow and development lifecycle. The truth of the matter is that, yes, many tools can do complicated and space-age things within the agentic coding space, but it’s the simple things that provide end users real benefits and features that win the day.

Ok, so what? Show me the money.

Yeah, yeah, yeah, so enough yip-yapping; let’s get into Omnigent, get it set up, piddle and poke around, and see what we can find under the rug. Personally, I am actually excited to get this meta-harness running and see how it can ease my agentic development process. Some of the features that are top of mind for me and seem very useful are …

• single interface for multiple models and harnesses

• rules and policies I can apply to specific use cases

• multi-agent workflow for high-level systems planning, research, and design.

That’s probably just because of the nature of the work I end up doing. Big, high-level project planning that includes extensive systems architecture and design. It’s really important to me that the planning and research phase goes well before a particular direction is chosen. Also, on a more practical note, I have particular codebases that I deal with differently, in very specific ways. Being able to set up policies to deal with this would be very helpful.

Blah. Ok, so you say the install.

>> curl -fsSL https://omnigent.ai/install.sh | sh

So what happens next? Well, we need to set up our existing tooling(s) and harnesses, like Claude or whatever, to Omnigent; this should be the first step.

>> omnigent setup

After running this command you should be prompted to setup and connect any models and harnesses you already are using. In my case I connected both Claude Code and CoPilot up to Omnigent. The setup usually can pickup automtically and detect any tools you are already using. As you can see below, it found my Claude Code setup, sensed my CoPilot setup but I also had to get a token for CoPilot for Omnigent to use. Either way, not rocket science, just connect your existing tooling. I also use OpenCode but did not connect it.

At this point you couse run Omnigent ontop of a single harness, and get all the features we talked about previously. The UI, policies, session sharing, mobile access, etc. For example, I can run … (note: must have tmux installed)

omnigent claude

Of course, as you can see in the bottom of that screen, we get a local host WebUI on port 6767 that has the session we just started in the terminal.

So what, we have meta-harness sitting ontop of our typical Agent Harnesse(s), now what? Well, of course this UI gives us the ability to do things with our conversations as if we were say using Cursor or any other UI. You can watch what’s happening in a session, make a comment on some of the work that’s happening, upload files, edit files directly yourself. It’s kinda like having a IDE for your Agents. We could go on forever exploring the features Omnigent has. One of the more interesting and powerful features is Policies.

We can add our own policies, or enable the many pre-built policies that ships with Omnigent that are centered around Cost and Safety. For example, let’s add a custom policy through just the Web UI Chat interface that we want our Agents to ask for permission before making git commits and pushing changes up to remote repos. Also, we will have it ask for permission before running bash commands.

Apparently these are only session scoped policies, and I need to be more specific. Either way, after asking it to commit and push the git changes it made to the repo (updated the README). It did ask for permissions to run the bash and to run the git commands.

Of course we would probably want to make policies that run over many sessions and the sky is the limit as to what kind of policies and limits we could put in place. There is a massive list here of built-in policies. A very nice feature indeed when you are working across many different code bases, work on teams where budgets and token costs are a thing, where you are worried about Agents doing unintended things to filesystems and production environments. You could easily build a number of policies to watch out for errant aws cli commands that are dangerous, or maybe even an Agent trying to update some terraform file and sync it.

Polly: the omnigent multi-agent super power.

Probably the one feature I am most excited about is Polly. Again, this probably has alot to do with my day to day work of planning technical projects from idea to implementation. This involves lots of research, trial and error, exploring different avenues, reviewing materials, and generally this a lot of back-and-forth between some Agent and myself. Don’t worry, I still spend time just reading raw documentation for myself, there is no replacement for understanding deeply different technologies and features.

“Polly is a multi-agent coding orchestrator. It breaks your task into sub-tasks and delegates each one to a different AI agent, with cross-vendor code review built in. This tutorial shows you how it looks in practice.” - omni docs

I’m excited about this feature because when using some specific Agent to explore a topic, I generally do a ton of different back and forths, exploring every angle I can. Some of that work could defiently be saves when the meta-harness can do that for me. Personally, I’ve found that simply one-shot prompting is the worse approach to solving any problems, most models might do Ok, but depending on the subject they might do quite bad. I would expect this multi-agent Polly routing to impove results a lot.

It’s as easy as …

omnigent polly

This is what I asked Polly.

In the WebUI for Omnigent I can click on Agents and see multiple Agents running, solving this problem. Pretty easy and for many situtations this gives me peace of mind that multi-agents will do a reasonable job out of the box planning or solving the problem I want to explore.

It worked of course, no surpise there.

Again, I think that the multi-agents for planning and researching is a underutililzed and easy to adopt superpower for most devs. You know me, I’m a fan of learning and exploring yourself, but this sort of multi-agent Polly workflow with Omnigent can help with systems design, architecture, and generally large scale planning that is important you want to cover all bases.

The scary part.

Well, maybe scary isn’t that right word I’m looking for, I’m not sure how I feel about this feature. What is sold as the ability to collborate from anywhere … aka your phone, well that can be good and bad. Good from a technology perspective, of course in theory we would want to be able to continue long running sets of tasks and work, collaborate with co-workers on high-stakes agentic work. Sure, I get it.

But, we should be honest about the possible pitfalls of being able to run Agents from your phone. From your bed. The airport. The school line. On the couch. At your mom’s house.

Handle such things with care, you don’t need to be productive 24/7, even if you want your agents to run 24/7, I have serious questions about the mental health of someone running agents from their phone at 8 pm at night. You need to touch grass, exercise, give your head and mind a break. I’ve learned from experience that burnout creeps up quickly and quietly, make good choices that are healthy for you and your body.

Anywho, because I love you all very dearly, I will go ahead and setup this feature for myself, Omnigent recommends using railway. You will need to setup a free account to do this. Just a few button clicks for the free account. Point Railway at the Omnigent GitHub repo, authorize via GitHub, and you’re off to the races while it deploys.

Ya know, part of me is happy that the Railway deployment of omnigent failed on the first try. Does that mean I don’t have to do it?

Didn’t matter what I tried, Omnigent wouldn’t deploy; it always failed on the health checks. Which, honestly, is fine with me, I didn’t want to be suckered into coding from my phone. No good. Ok, maybe I can try a little harder, let’s look at the logs.

RuntimeError: DATABASE_URL is required (env), or set `database_uri:` 
in the server config. Accepted forms: 'postgresql+psycopg://user:pw@host:5432/omnigent' 
(explicit psycopg3), or the 'postgres://' / 'postgresql://' URLs emitted by Railway,
 Render, etc.

Looks like there should be a DATABASE_URL as part of this deploy, the instructions never mentioned it. I tried adding the default localhost Postgres ENV Var and re-deployed; it didn’t work. That’s about as far I would mess with it; as I said, I don’t really want to run Agents from my phone.

Share

Thoughts on meta-harness.

Well, we learned a little something, maybe; I did anyways. I know we’ve barely scratched the surface of all the bells and whistles of Omnigent, but that’s ok. I learned it’s easy to use and install, easy to hook up my other coding harnesses too (Claude and CoPilot), and I do like the option of switching between the WebUI and Terminal depending on what kinda work I’m doing.

The policies were probably my favorite part of Omnigent. When working on sensitive projects that require me to run aws cli commands, or mess around with terraform, policies are another great back-stop to keep tight control where that control is warrented.

Second favorite, and a close one at that, is the Polly feature of Omnigent. It’s hard to state how important this feature can be for those of us who do a lot of research, planning, and systems design/architecture. I spend a lot of time ensuring I’m covering all bases, taking the right direction, and that things are in tip-top shape. Being able to have multiple agents allowing for “cross-vendor code review”, is next-level stuff.

The final piece of Omnigent that has become clearer to me with repeated use is the … cleanliness.

Anyways, this is one tool I’m looking forward to using on a daily basis, and getting to know it better will probably expose a lot of cool and unsung features that will improve my agentic workflows.

Leave a comment

Want more? Checkout this video.

The vibes are cheap; even if the per-token cost increases 10x or 100x, it doesn’t matter, the vibes are cheap; at least they are cheap from the viewpoint of those who hold the power. The power to hire and fire, to set technical direction, and to point a finger and send the peons running.

Read more

I’ve been watching this story play on repeat for some time now; the death of tools like Spark and Kafka has been heralded from the treetops for a decade now, yet here we are … living in a Spark and Kafka world. Someone once told someone else that it is hard to kick against the goad. Indeed it is. I don’t see this as any sort of failure or shortcoming, though; it’s just part of the circle of life, and tech. It’s natural.

Think about the wicked sharp edges that drove legions of SQL Server developers and DBAs insane and to an early grave over the last few decades. Yet, there are still plenty of SSIS packages humming happily away in the quiet corners of the world.

The truth of the matter is, once a tool gets its tentacles latched onto the corporate minds of CTOs, architects, and the like … it’s there to stay for better or worse, for some time to come. Spark is like that. It’s the bread and butter of the data community at large, has been for a while, and will continue to be that … even as Polars, DuckDB, Daft, and the like … bite and nip at the heels. Truth be told, I hold a special little place in my cold and stony heart when it comes to Apache Spark. I grew up in data in a time when I remember ssh’ing into various nodes, installing Spark, the correct AWS S3 JARs, adding IP addresses to configs, updating the memory allocations. Basically terrible things.

• Go search through my old blog; you're sure to find some Spark goodies in there.

Anywho, back to the matter at hand. Lakesail.This one has been on my list for some time, but you know how things go: busy, life, this, that, AI. But they are making enough waves now that I owe it to myself to do what I’ve been doing for a long time, namely, kick the tires and poke with a stick … see what crawls out of the dark corners.

I mean, what is it? Well, the Lakesail homepage will give it to you right away, straight to the old eye, which is appreciated. People forever beating around the bush, making you figure out what the crap an Agentic Wizz-Bang-Popper is. What is Lakesail? Spark, without the JVM tax. You know what sounds even better to my old ringing ears? Spark rewritten in Rust. Like a fly to light me.

I know some of you unbelievers and luddites are sitting there rolling your eyes at the words “rewrite” and “Rust” in the same sentence … but there is a reason people are doing it, my friend; just as well give in to it and get on the ole’ bandwagon. The ride is fine.

Of course, I already mentioned it, but the idea of going after Spark, because it’s “slow” and a “pain in the butt,” is nothing new. Some of the criticisms are true; some are not. It depends on the context. Most people don’t work with “big data,” aka under 10 TB of data or way less, so theoretically they don’t need Spark. They just use it because it’s there and everyone else is using it. Herd mentality.

I’ve made the mistake myself of trying to prove to people that you don’t need a big cluster to work on large data. Be warned: when you start messing with people's golden calf, expect to be summarily burned at the stake; be ready to be a heretic.

One can see the major difference between someone saying DuckDB or Polars can replace Spark, which they can, and someone saying that isn’t the whole story. We don’t live and work on data projects in a vacuum. It takes a serious amount of commitment for any Staff Engineer or CTO to look at a group of drooling and keyboard-happy engineers and say, “Yes, that’s a great idea; go ahead and rewrite our entire stack and legacy data pipeline(s) into DuckDB/Polars/Daft.” That isn’t the real world. In the real world, we have all sorts of deadlines, product launches, tech debt, and general mayhem. Sure, we of course WANT to do those projects, but we don't often get the chance.

Back to Lakesail. This is the impressive part, if it turns out to be true, and believe me, I’m going to try it out, hook, line, and sinker. This is the difference maker. Just swap out the engine, leave the code as is.

“Spark Connect protocol means your existing PySpark, Spark SQL, Delta Lake, and Iceberg code runs unchanged, natively, not as an add-on.”

This is a much easier pill to swallow for data teams. At least for those running the show. It’s a hard sell, even with cost savings, to tell someone to rewrite all the code; a much more palatable message is to just swap compute or configs and leave the code be.

Ok, so all that as an introduction; I don’t really know a better way to understand a tool than by getting to work. So, let’s do that.

Exploring Lakesail concepts, open source, paid platform, etc.

It’s going to be hard to understand all the features of any tool in the short time we have, but I will do my best to give you a working understanding of what Lakesail offers from both open- and closed-source perspectives. We will start with the concepts, check out the open-source options, and finish by setting up an account on the Lakesail Platform to see what that is all about.

Ok, what is Lakesail?

It is

• “… distributed multimodal compute engine …”

• built on Apache Arrow

• built on Apache Datafusion

• Spark users can “… switch to Sail without making code changes.”

• Built with Rust.

This probably makes sense to most of you, I would imagine, besides maybe the word multimodal (you can read more about this on the buoyantdata website). The rest of these are kinda obvious and well known to most of the data community. Apache Arrow is the new go-to in-memory data format; Apache DataFusion has been around for a long time, and many tools are built on it. The idea that Lakesail is built with Rust is a given, but the most interesting part is that we could port existing Apache Spark code to run on Lakesail compute, which is enticing indeed.

How can this be?

The handshake between Lakesail and Spark is Spark Connect.

I have written about Spark Connect before, you can check that link out below. Recently, I’ve also used Spark Connect when building Agentic Chatbots and Agents, etc, as it allows for simple communication between anything non-Spark … and well … Spark.

Let’s jump to the TDLR of Spark Connect, because it’s an important part of the Lakesail conversation. Here is that summary taken from the article I wrote about Spark Connect.

What, pray tell is Apache Spark Connect?

“… a decoupled client-server architecture that allows remote connectivity to Spark clusters using the DataFrame API …”

and

“It is a thin API that can be embedded everywhere: in application servers, IDEs, notebooks, and programming languages … “

So, this is key to understanding how something like Lakesail can truly be a built-in replacement for compute/processing for existing Spark pipelines. It’s just a protocol; Lakesail understands this protocol and can translate a Spark query into a Lakesail query. Easy peasy, not really.

Subscribe now

Playing locally with Lakesail.

What I want to do is play with Lakesail locally, just to see and feel what it’s like; then I would like to set up a Lakesail account, connect it to my AWS account, and maybe run the same setup to see what’s crackin'.

uv init testingLakesail
cd testingLakesail 
uv add pysail pyspark-client==4.2.0

Apparently, we could use the command line with sail to drop into the classic interactive Spark shell, with the Sail server running behind it. But I’m of the opinion that most folk don’t use the Spark command line anymore; most people are in Notebooks. You can thank Databricks for that.

sail spark shell

Either way, let’s write some PySpark code that uses Sail; in theory, it should be boring, because … Sail is just running behind the scenes and taking our normal Spark code and executing it against its own engine built on Rust, Arrow, and Datafusion. This will not be a scientific test at all, but I’m new to this, so give me a break. We will run the same normal PySpark script in a Databricks Notebook on Serverless compute, observe the runtime, then run the same script with Sail. This will all be without a classic Spark cluster, in a sense, on small data, but I just want to see what happens.

If I’m feeling spicy later, I can connect my personal AWS account to a new Lakesail account and run a larger dataset with a classic Spark cluster.

Luckily, kind people like Backblaze provide an open-source hard drive dataset as a queryable Apache Iceberg table on S3 or compressed CSV files.

We will use the raw, compressed CSV files, since they should pose no problem for Spark. I can get them into my personal AWS account in an S3 bucket; then we will have the data against which to run our tests of vanilla Spark and LakeSail Spark.

• 4.54 GB (4,536,210,401 bytes) across 365 objects

• 120,191,685 rows (~120.2M)

Now, I know everyone is going to say this is not fair; we should test on some massive dataset to give vanilla Spark a fighting chance. Also, Serverless; who knows what’s going on behind the curtain? Oz is back there pulling levers. I hear you, but I ain’t got the time, and I’m the one who has to spend the money. Easy for you to say Sunny Jim.

The Spark version running in a Databricks Notebook against Serverless is easy enough.

import datetime as dt
import time

import pyarrow.csv as pacsv
import pyarrow.fs as pafs
from pyspark.sql import functions as F

SRC = "s3://confessions-of-a-data-guy/drivestats/csv/*.csv.gz"
DST = "confessions-of-a-data-guy/drivestats/results/failures_by_model_day_serverless.csv"
REGION = "us-east-1"
START = dt.date.today() - dt.timedelta(days=730)

t0 = time.perf_counter()

failures = (
    spark.read.option("header", True)
    .csv(SRC)
    .select(F.to_date("date").alias("date"), "model", F.col("failure").cast("int").alias("failure"))
    .where((F.col("failure") == 1) & (F.col("date") >= F.lit(START)))
    .groupBy("date", "model")
    .agg(F.count(F.lit(1)).alias("failures"))
    .orderBy(F.col("date").desc(), F.col("failures").desc(), "model")
)

table = failures.toArrow()

fs = pafs.S3FileSystem(
    access_key="xxx",
    secret_key="xxxxxxxx",
    region=REGION,
)
with fs.open_output_stream(DST) as sink:
    pacsv.write_csv(table, sink)

print(f"{table.num_rows} rows -> s3://{DST} in {time.perf_counter() - t0:.1f}s")

display(table)

And the results.

2588 rows -> s3://confessions-of-a-data-guy/drivestats/results/failures_by_model_day_serverless.csv in 35.6s
pyarrow.Table
date: date32[day]
model: string
failures: int64 not null
----
date: [[2026-03-31,2026-03-31,2026-03-31,2026-03-31,2026-03-31,...,2025-04-01,2025-04-01,2025-04-01,2025-04-01,2025-04-01]]
model: [["ST8000NM0055","HGST HUH721212ALN604","TOSHIBA MG10ACA20TE","ST16000NM001G","TOSHIBA MG08ACA16TEY",...,"ST8000DM002","WDC WUH722222ALE6L4","HGST HUH721212ALN604","ST8000NM0055","TOSHIBA MG08ACA16TA"]]
failures: [[5,4,3,2,2,...,2,2,1,1,1]]

Ok, for the Sail swap-out, in the Databricks Notebook we need to do this.

%pip install "pysail==0.7.0"
%pip install "pyspark-client==4.2.0"

Here is the same code, with Sail. You can see the differences are only due to starting a Sail Spark Connect server and then using that. Obviously, in the Production setting, we would have a dedicated Sail Spark Connect Session running and serving from some permanent place. And in theory, we wouldn’t have to change the core code for the Spark pipelines, just what they connect to.

# run these first, in their own cell:  %pip install pysail   then   %restart_python

import datetime as dt
import os
import time

import pyarrow.csv as pacsv
import pyarrow.fs as pafs
from pyspark.sql import SparkSession
from pyspark.sql import functions as F

from pysail.spark import SparkConnectServer

SRC = "s3://confessions-of-a-data-guy/drivestats/csv/*.csv.gz"
DST = "confessions-of-a-data-guy/drivestats/results/failures_by_model_day_sail.csv"
REGION = "us-east-1"
START = dt.date.today() - dt.timedelta(days=730)

# Must be set before the server starts.
os.environ["SAIL_SPARK__SESSION_TIMEOUT_SECS"] = "3600"
os.environ["AWS_ACCESS_KEY_ID"] = "xxxx"
os.environ["AWS_SECRET_ACCESS_KEY"] = "xxxxxxxxxxx"
os.environ["AWS_REGION"] = REGION

t0 = time.perf_counter()

server = SparkConnectServer("127.0.0.1", 0)
server.start(background=True)
host, port = server.listening_address
# The notebook's own `spark` is the Databricks session; this one is Sail.
sail = SparkSession.builder.remote(f"sc://{host}:{port}").create()

failures = (
    sail.read.option("header", True)
    .csv(SRC)
    .select(F.to_date("date").alias("date"), "model", F.col("failure").cast("int").alias("failure"))
    .where((F.col("failure") == 1) & (F.col("date") >= F.lit(START)))
    .groupBy("date", "model")
    .agg(F.count(F.lit(1)).alias("failures"))
    .orderBy(F.col("date").desc(), F.col("failures").desc(), "model")
)

table = failures.toArrow()

fs = pafs.S3FileSystem(
    access_key=os.environ["AWS_ACCESS_KEY_ID"],
    secret_key=os.environ["AWS_SECRET_ACCESS_KEY"],
    region=REGION,
)
with fs.open_output_stream(DST) as sink:
    pacsv.write_csv(table, sink)

print(f"{table.num_rows} rows -> s3://{DST} in {time.perf_counter() - t0:.1f}s")

display(table)

sail.stop()
server.stop()

This is where I started to have problems. I could see from the Logs in the Notebook that the Sail server was starting, but it was churning for minutes running in Serverless compute with no end in sight. My guess is that Databricks Serverless is not the place to test the Sail version of this code.

[2026-08-18T00:19:00Z INFO sail_python::spark::server] Starting the Spark Connect server on 127.0.0.1:37959...
[2026-08-18T00:19:00Z INFO sail_session::session_manager::actor::handler] creating session 9c1eba5a-3765-4572-81cf-63201d9a2d0d

Well, in the end it did finish, just much slower. 197.7s

[2026-08-18T00:19:00Z INFO sail_python::spark::server] Starting the Spark Connect server on 127.0.0.1:37959...
[2026-08-18T00:19:00Z INFO sail_session::session_manager::actor::handler] creating session 9c1eba5a-3765-4572-81cf-63201d9a2d0d
2588 rows -> s3://confessions-of-a-data-guy/drivestats/results/failures_by_model_day_sail.csv in 197.7s
pyarrow.Table
date: date32[day]
model: string
failures: int64 not null
----
date: [[2026-03-31,2026-03-31,2026-03-31,2026-03-31,2026-03-31,...,2025-04-01,2025-04-01,2025-04-01,2025-04-01,2025-04-01]]
model: [["ST8000NM0055","HGST HUH721212ALN604","TOSHIBA MG10ACA20TE","ST16000NM001G","TOSHIBA MG08ACA16TEY",...,"ST8000DM002","WDC WUH722222ALE6L4","HGST HUH721212ALN604","ST8000NM0055","TOSHIBA MG08ACA16TA"]]
failures: [[5,4,3,2,2,...,2,2,1,1,1]][2026-08-18T00:22:17Z INFO sail_session::session_manager::actor::handler] removing session 9c1eba5a-3765-4572-81cf-63201d9a2d0d
[2026-08-18T00:22:17Z INFO sail_python::spark::server] Shutting down the Spark Connect server...
[2026-08-18T00:22:17Z INFO sail_python::spark::server] The Spark Connect server has stopped.
[2026-08-18T00:22:17Z WARN sail_server::actor] aborting 4 task(s) for SessionManagerActor

I mean, both scripts worked; in s3 I can see the results from each process.

Even the results of each file match, which we would expect.

I guess I wasn’t expecting Databricks Serverless to beat Sail on speed, but that wasn’t really the point. We did prove that we can take a vanilla Spark data pipeline, point it at a Sail Spark Connect instance, and run the same exact code without touching it, and get the same exact answer as vanilla Spark, except we ran it through a different engine.

Don’t get me wrong, I could have been doing something totally dumb and backward; you don’t know what you don’t know when playing with new tools.

Out of pure curiosity, I ran the same Sail Spark code from the Notebook locally on my machine to see what would happen. That way I could ensure beyond a doubt that the Spark Sail code was indeed using that as the engine, because who knows what’s happening on Databricks Serverless; it could be playing nice or not. I mean, from the Sail logs, it appears like it went through that way, but I’m not 100% sure.

Well, it took even longer locally. 202.0s

 2588 rows -> s3://confessions-of-a-data-guy/drivestats/results/failures_by_model_day_sail.csv in 202.0s

I don’t really take this as a bad sign; I’m just playing around for the first time. Anyone in their right mind knows that for many datasets, Rust + Datafusion + Apache Arrow is WAY faster than Spark for most data workloads. Maybe until you get to the PB level.

Subscribe now

LakeSail as a Data Platform.

I figured since the above didn’t work out that well, I would poke at the SaaS data platform LakeSail offers. My first impressions were very good; they have a clean website and a straightforward, non-confusing onboarding workflow. You would be amazed at how many people get this wrong. LakeSail clearly put some work into this.

When creating a LakeSail stack in AWS, you can see during the setup what is going on behind the scenes, aka, what technologies they are using to wrap Sail up in way we can use it from their SaaS.

• EKS clusters

• VPC and networking (subnets, security groups, load balancers, VPC endpoints)

• IAM role creation for EKS infrastructure and workload roles

• S3 bucket management for workspace data and job artifacts

• Glue Data Catalog management for default catalog

• KMS key management for EKS secrets encryption

• CloudWatch Logs for EKS control plane logging

• CloudFormation stack discovery (read-only, tagged resources only)

Dang, you know what kinda money I spend on you hobbits??!! This is why, if you are a free subscriber to this Newsletter, give me an upgrade and pay the piper.

Look, you can watch my AWS bill go up.

Look Ma, I got a LakeSail account now.

The UI is very nice and clean, and now that we have connected to my AWS account, let’s go ahead and set up some “Compute” and then I would like to re-run the code we wrote against this more production-like Sail system in one of their Notebooks, and see how smooth the experience is.

When I went about setting up a Cluster to use, naming it, selecting my AWS account, region, etc … it wouldn’t let me go forward without filling in IPv4 CIDR Range for it to use in AWS. This was kinda strange; I've never experienced this before on other compute platforms like Polars Cloud, MotherDuck, Databricks, etc.

I mean, clearly LakeSail has access to my AWS account to set up the previous infra; this probably should have been taken care of during that process. I am no AWS expert, especially in networking, but that leaves me little choice but to scroll through my AWS VPCs and pick some unused IPv4 CIDR range?

While in the middle of doing this, apparently my timing is bad. The site was undergoing some maintenance. So, had to wait for things to get stable and come back again.

Ok, now that we are back in business, I can get to creating this Cluster, of course selecting the Budget option.

As a side note, I wasn’t really planning on reviewing LakeSail’s cloud data platform in depth, but since we are already here, notice what menu options, and therefore features, they provide.

We have Jobs, Notebooks, Compute, Catalogs, and Queries … all of which need no explanation for most of us using modern SaaS data tooling. We would expect such features as the minimum groundwork on which to build robust data platforms.

Ok, after a little wait, it looks like your Cluster is deployed and running. 15 Pods! Holy Mother of Pearl. Better get this test done and everything torn down before I go broke.

Next, I need to make a new Compute Profile (I couldn’t create a Notebook without it). We will use Tiny for size.

After that, we can get to making a Notebook from which to run our previous code.

Next, we wait for the Notebook to come alive, which is a strange feeling. Makes sense when you read the note about waiting for a Marimo pod to emerge.

Waiting for Marimo pod to become ready (namespace: lakesail-compute-zkmjymkcwb)…

Now, finally, we can run our previous Sail code and see what's crackin'. I’m only half interested in the runtime, but I’m more interested in this code's ability to run out of the box without issues. Remember, we are just trying to get a feel for working with Sail itself and LakeSail, the data platform as a whole. I mean, we picked a T3 Small for AWS compute, with a whopping 2GB of RAM. So, the fact that Sail can run this code with that small of compute says something in itself. That’s what you call cost savings.

( I had to change from display(table) to print(table). )

It looks like we got about the same runtime, 177.7s, which is great, considering we used the smallest compute on the market.

Ok, I think it’s time to stop our testing of Sail and LakeSail, if they are one and the same, and just make a few comments about what we’ve seen so far.

Share

The TDLR on LakeSail and the future of Spark.

Wow, what can I say? I literally knew nothing about Sail when I opened my Chrome browser to start writing this … and now I feel like I still know nothing about Sail! Hahaha! I sort of say that tongue-in-cheek, but it’s hard to give the proper time and space to brand-new tools like this in only a day or two. But we played enough to discuss some high-level concepts.

Firstly, it’s so interesting that you and I live in the age of the upstarts, all coming yapping and nipping at the heels of Apache Spark. There is nothing inherently wrong with Apache Spark; it’s been the backbone of data and analytics for a decade at this point. But it shows how far that data community has come, and how much the data landscape has changed. Today's data requirements are not the same as they were 10 years ago.

LakeSail puts those needs at the forefront and solves them. We’ve seen these shifts across the tech landscape at large.

• Moving away from the JVM toward Rust.

• Apache Arrow underpinning everything.

• Crunching more data with less compute.

• Reducing compute costs.

• Reducing complexity.

Like all good humans, data folk want their data faster, cheaper, and right now!

LakeSail, the SaaS platform itself, I thought, was wonderfully nice and easy to use. You would be surprised how many people get this wrong. The onboarding experience, the simple UI, and the ease of use and navigation. Providing a Data Platform experience that developers enjoy and find to be a low barrier to entry is priceless. It means LakeSail is doing things and going places.

Sail, as a technology in and of itself, I don’t have much to comment on, other than that it is brilliant. The idea to support the Spark Connect Protocol was a genius move. It takes all the arguments about migration away. There is no code rewrite; lift-and-shift has always been a problem, and Sail solved it. I mean, it’s so easy to use; it makes sense to at least test workloads on LakeSail and see if money can be saved.

• Remember, this is just my opinion; I have no affiliation with LakeSail. I’m allowed to say something is good … when it is good.

The only question left nagging in my mind is the performance. I’m sure that has more to do with me than Sail. I said it before, and it’s true: for most datasets, Rust, Datafusion, and Arrow are going to beat vanilla Spark. I think maybe that says more about Databricks Serverless running Spark, how good that is, if you want to know my opinion. I can like two tools at once.

Let me know what you think in the comments.

Leave a comment

Yet another day passes with me casting a lazy eye over the endless dribble of marketing-produced content on the once-glorious but now fallen LinkedIn; at the intersection of Agentic/LLMs and Data, if you allow me to call it that. The winds of change are blowing hard, and it’s difficult to decipher the real from the classic hello-world demo tricks. A wise man once said that “all things are possible, not all things are profitable.”

What we really need, what I need, is more real-life examples of building quasi-agentic pipelines (data) that walk the thin red line down the middle of the road. Most of us have no desire for the Gas Town and Loop your Prompt Bros that hammer at us incessantly at every waking hour. There is also a very real need for Data Engineers and data practitioners generally to accept the world has changed, and we need to provide useful solutions to the business; some of those include providing the ability to add LLMs/Agents into our existing data workflows.

I wanted to show and talk about two things today that may or may not be interesting to you. Mostly as a way of providing a third point of view into the morass of Agentic discourse, once that is grounded more in reality, while slowly trying to incorporate new things into old tools. A practical approach to agents/LLMs inside data workflows and pipelines.

1. Using the same old tools … Databricks and Apache Airflow to solve new Agentic problems that the business wants solutions to.

2. Talking about determinism in data pipelines and bridging the gap between what the business wants you to do and how you do it as an Engineer and Architect.

One will be moderately technical, the other philosophical and *gasp political.

Thanks to Delta for sponsoring this newsletter! I use Delta Lake daily, and I believe it represents the future of Data Engineering. Content like this would not be possible without their support. Check out their website below.

The Agentic “problem” in a world that hasn’t caught up.

We are going to start backward today. When introducing today's plan and discussion, I listed the whole “… determinism in data pipelines and bridging the gap between what the business wants…” as part deux, but I’m going to start from the bottom up and talk about this first because it sets the stage for the more technical solution that I’ve used to solve this very real business request. First, it isn’t uncommon in the Brave New World that we live in to get some version of this request from “The Business” or Product … or the C-Suite; it doesn’t really matter. We do what we are told.

I mean, as I said, it’s a new world we live in where our most gracious and wonderful business counterparts have learned via ChatGPT and Claude that the sky is the limit. If they can dream it up, apparently we can build it. The issue with this dream of randomly injecting some AI into the middle of an otherwise normal workflow is not new, but is fraught with difficulty and has bred an entire army of consultants and new “AI Tool(s)” to do this for us.

So what are you waiting for? Done yet?

With everything that has changed … one thing has not. The last %10 of that “easy” project is where the blood is spilled onto the ground.

• We can’t buy and implement a new AI tool for every new use case.

• We HAVE to work within the constraints of our existing infrastructure and systems.

• We need to deliver reasonable results that meet expectations on a reasonable timeline.

The answer to every question involving AI isn’t simply another tool, package, subscription, some new backflip, and throwing salt under the old oak tree at midnight. We can’t redesign the entire stack to send rockets to the moon because someone simply wants an LLM/Agentic feature to be added at some random spot. This is not the way. We need to find the right balance of being problem solvers inside an engineering and business context. This means we are thoughtful, considerate of cost and time, and focused on good-enough solutions that are a great starting point and don’t involve signing up for some new year-long SaaS contract just to call an endpoint that has an LLM behind it.

Apache Airflow + Databricks for LLM integration.

Ok, so I just wanted to cover this topic, which won’t be that exciting, but does contain some interesting talking points that we will get to. Recently, at the day job, I’ve been working on integrating LLMs into your average, everyday data pipelines in a reasonable manner, without trying to boil the ocean or start over. As an engineer, when I start to solve a problem, I first turn to the tools that I have in front of me. I’m reluctant to add new tools and infra to an existing stack without good reason.

Core tenets are …

• simplicity

• dealing with non-determinism

• use existing tooling until it doesn’t work

This a real data flow inside a Production Airflow DAG. It’s a mix of normal, Python-centric data tasks and API calls to an LLM that may or may not return what you expect, knowing that downstream tasks depend on these non-deterministic tasks. Sorta like taking your life in your own hands, ya know. “Claude, don’t screw this up, no mistakes.”

Anywho, not sure if you keep up to date on the happenings at Databricks; it can be hard to keep pace. One of the most unsung heroes and cool features on Databricks is modeling serving endpoints. Databricks does things well, like it or not, and they make it super simple to create and serve API endpoints for LLMs that you can call with a simple TOKEN.

For example, serving an LLM behind an API sounds super complicated, right? This is a sample from a production Terraform file/repo that runs the LLM behind the API calls above, which go to a Databricks Modeling Serving Endpoint. All batteries and bolts included. It’s amazing that in a few short years we went from the cool kids playing with LLMs to a few lines of code that will spin up a model-serving endpoint ready for use. Simplicity is key; architecturally, implementing features that aren't a burden to the overall system allows us to move on and start providing value for our business.

I won’t bore you with the Python DAG and Python function details; they are far from special. Calling a Databricks Model Serving Endpoint from an Airflow DAG is like calling any other API … the only fizzbuzz being that you aren’t ever really going to be sure what comes out the other end; you have to design for that ambiguity and non-determinism.

Again, the point is that you don’t need some fancy new tool(s) to add “Agentic” capabilities to your data pipelines. Sure, if you think you need to rebuild self-healing pipelines run and managed by agents, then yeah, you probably do need to go buy some SaaS. But if you have Databricks and Airflow, two of the most well-known tools known to human-data-kind, then yes, with a little fiddling around, you can add LLM calls into your data pipelines. The biggest problem posed with doing this, as I mentioned a few times above, is how to deal with the non-determinism.

Most of us, alot of us, deal in Directed Acyclic Graphs. We have units of work, tasks; they have relationships with other tasks, dependencies, etc. This is, in theory, in direct opposition to inserting LLMs into the mix. What, at the core, is a DAG?

• Directed: Lines have arrows that show a one-way path / strict dependency from one node to another node.

• Acyclic: There are no loops … aka feedback cycles.

• Graph: A collection of tasks connected together.

We have to build some sort of verification into the system; at least we should, unless we are going to close our eyes and let ‘er rip, come what may. Which isn’t the worst idea in the world. I mean who knows what we are asking the LLM to do, it could be any number of things. Clean data, check data, make data, look for this, find that, combine this. Whatever. But, unlike the rest of our DAG, which is most likely a very strict set of code that will return a very specific thing … we can’t really say for certain an LLM will do what we ask. At least not 100%

Indeed, a brave new world.

We aren’t the only ones thinking about this.

To be sure, even if you don’t hear talking heads talking about this one, we aren’t the only ones mulling over what it means to combine Agents/LLMs with classic data tooling. Apache Airflow itself has deemed it necessary to release a Common AI Provider.

They provide the ability to make single Task Decorator LLM calls.

from pydantic import BaseModel
from airflow.providers.common.compat.sdk import dag, task

@dag
def my_pipeline():
    class Entities(BaseModel):
        names: list[str]
        locations: list[str]

    @task.llm(
        llm_conn_id="my_openai_conn",
        system_prompt="Extract named entities.",
        output_type=Entities,
    )
    def extract(text: str):
        return f"Extract entities from: {text}"

    extract("Alice visited Paris and met Bob in London.")

my_pipeline()

Even multi-task Agent steps.

from airflow.providers.common.ai.toolsets.sql import SQLToolset
from airflow.providers.common.compat.sdk import dag, task

@dag
def sql_analyst():
    @task.agent(
        llm_conn_id="my_openai_conn",
        system_prompt="You are a SQL analyst. Use tools to answer questions with data.",
        toolsets=[
            SQLToolset(
                db_conn_id="postgres_default",
                allowed_tables=["customers", "orders"],
                max_rows=20,
            )
        ],
    )
    def analyze(question: str):
        return f"Answer this question about our data: {question}"

    analyze("What are the top 5 customers by order count?")

sql_analyst()

And possibly the scariest of all: LLM Branching, which lets the Agent decide which downstream tasks to run. Yikes!!

@task.llm_branch(
    llm_conn_id="my_openai_conn",
    system_prompt="Classify the support ticket priority.",
)
def route_ticket(ticket_text: str):
    return f"Classify this ticket: {ticket_text}"

Heck, they even added Human-in-the-Loop gates!

LLMOperator(
    task_id="summarize_report",
    prompt="Summarize the quarterly financial report for stakeholders.",
    llm_conn_id="my_openai_conn",
    require_approval=True,
    approval_timeout=timedelta(hours=24),
    allow_modifications=True,  # reviewer can edit the output
)

Wow, things get a little complicated fast, I methinks. Who knows what the future holds; time will tell. Even though the world of AI seems old, everyone is Clauding their way to glory, as far as REAL Agentic work that gets to be integrated into our existing tech stacks; well, I would argue that we are still in the infant stage of that. Lots of new tooling is being released into the data world for agents; only time will tell what sticks and what doesn’t.

I personally think that big question isn’t …

“How do we insert LLM/Agents into pipelines.”

But rather …

“How do normal determanstic pipelines integrate with LLM/Agents that are non-determanstic.”

What I mean by this is that it’s one thing to have a purely LLM/Agent pipeline running, because all the tasks are deemed non-deterministic and encapsulated together. It’s something else to mix the two. Like I did. Kept a guy up a night, ya know. It’s walking the fine line between trusting an inherently untrustworthy entity and what were once trustworthy, deterministic pipelines. No more. This is even more important in a Data World. Trust in data is the cornerstone of many a business these days, and the downfall of many. The unknowns of LLMs/Agents getting their hands on the data pipelines definitely raise some interesting questions.

Have you been playing with LLM/Agentic tasks inside your existing data pipelines? How has it gone? What tools are you using? What do you think the future holds?

Leave a comment

A few things have not changed in the age of AI, where the LLM has replaced 3/4 of the brain power of a whole generation of engineers who once found their meaning in the code they wrote. I mean, I hate to sa…

Read more

I’ve seen this topic arise a few times, sometimes shouted from a Substack, other times running as a strong undercurrent among some unhappy technologist. Either way, it boils down to the same problem that an entire generation of software engineers and other technical folk are wrestling with ... It’s even taken on the hues of a life-and-death struggle between two factions.

Being a person myself who was and is a self-taught programmer, having never taken a CS class in my life, I find the choice between the red and blue pill an incredibly difficult one, depending on the day. God knows what the right answer is; I don’t know. Most likely it doesn’t matter, and I will be long gone to the sky or retirement home before the die that has been cast is actually settled. To better help you understand the epic struggle I find myself battling inside myself every day when I open the terminal to do some work or exploration, and in that single split second decide whether I will type “Claude” or something of my own choosing, let me take you back to young me.

Subscribe now

The early years.

I come from a family of tinkerers, first mechanical and then digital. Growing up in the heart of the Midwest, beaten beneath that unforgiving sun, or frozen stiff in the cold north winds of the winter; it would not be uncommon circa 1998 to be found in two distinct places on a Saturday morning.

{ BTW. I write about those experiences on a separate Substack. }

First …

More often than not, I would find myself amusing myself amidst a vast pile of old and decaying cars piled to the sky, otherwise known as a junkyard. Often we would be on the search for a specific make and model, whos general vicinity was hand waved by some mysterious old man with a cigarette hanging out of his mouth standing next a burn barrel spewing forth the blackest smoke. Unless my skinny arms and young fingers were needed to wrench on some hidden corner of an engine, I would happily collect the spare change from the cigarette trays of 100 old cars. Then we would leave.

Second …

Many a time we would end up at my uncle’s farm, a lonely spot on the undulating prairie of the North, whose vocation as a shop teacher held many delights for a young boy. He was a most unusual shop teacher for his time; not only could he explain and tear into anything mechanical with delight and knowledge, but he also had a love for computers. It turned out to be a bug that affected a fair number of others, cousins, brothers ... and the like.

Today’s sponsor is Estuary. Without them, this content isn’t possible. The best way to support this Substack is to click the links below and give Estuary a try.

Pipelines built by an agent, orchestrated by you

Estuary Agent Skills lets your AI assistant draft production-grade data pipelines from a plain-language description, right inside Claude Code, Cursor, or the tool you already use. The output is config in your repo, not a black box. You review the diff, approve it, version it, and roll it back like any other infrastructure. Same runtime as every other Estuary pipeline. No separate “agent mode.”

Install the skill and ship your first pipeline →

I caught that bug young. Sitting in front of his Mac II, or later in front of Netscape, I felt that dial-up tone do something to my heart and mind I cannot explain. If you know, you know. What followed in the years before and after was an ever-increasing obsession with computers and all that they could do.

Sure, I would leave his house and drive the dirty gravel roads another five miles to my grandmother’s farm, where I would promptly seat myself on the top of the D17 Allis Chalmers (a tractor 🚜) with either a shotgun or fishing pole cradled in my lap, depending on the season. This was my experience growing up, and still is to some extent today.

Early on, I learned about Perl from a cousin I worshipped and became enamored with the terminal. From an older brother, I learned of something called PHP. The rest, as they say, is history. I spent my high school and college years working with the LAMP stack, picking up Objective-C, Droomla, hand-rolling servers, MySQL, CSS, HTML, PHP, and my beloved Perl. Mostly as a hobby, one of many, but I never thought of it as a career.

How do we save our souls from the Agentic Devil(s)?

So, with that long-winded and highly short view to the past, from the days before we even had StackOverflow to tell us we were absolute idiots not worthy to touch the keyboard for asking a question; we come to the crux of the position programmers find ourselves in. In some respects I envy the new breed of developers who come of age when Claudius Maximus reigns supreme; they don’t know what they don’t know; those little buggers probably sleep like logs.

Not me.

The existential crisis we find ourselves in, much like Neo, is: do we accept what we don't like, or do we fight back against the Machine? Like Cypher do we sell our birthright for a bowl of porridge, or in his case, a nice juicy steak?

Chew on that AI goodness, that sloppy code spewed out, just close your eyes and tab your way to a PR.

Look, we programmers have to eat. That’s what the luddites and witch hunters of old who refuse to dain to bend the knee to the AI gods are missing. It’s the context that matters. Some of us are gainfully employed by companies who, fully within their rights, have decided that the use of Agents in coding is the way forward and the future. If you refuse to drink from the cup, you are sealing your own fate. We can’t simply bite the hand that feeds us. We don’t have that luxury. We have mortgage payments, kids, cars, gas, healthcare, Big Slurps from the corner store; ya know, we don’t want to live in a van down by the river. At least not yet.

You can’t put the cat back in the bag. Indeed, at least in the short term, there are many aspects of the Software Lifecycle that benefit greatly with the use of LLMs.

• Infra as Code

• Documentation

• Testing

• Research

• Finding edge cases

• Exploring architectures and options

• Generating boilerplate

• Projecting planning

Even for those people that hate LLMs for coding can’t reasonably deny the use of Agents in these basic concepts. An LLM paired with a Senior+ Engineer is a sight to behold; the speed of what can be researched, planned, and built… is amazing. Yet, the very elixer we drink can posion us.

Yes, of course, with a few decades of experience and Claude, I can single-handedly output unthinkable amounts of code, features, research, plans, etc. But I do think all is as it seems. We have known, collectively, for some time that software solutions are more than just code. Software involves end users, engineers, product-market fit, cost, infra, on-call, alerts, monitoring, data, analytics, teams, inter-team communication, stakeholders, and the list goes on

The core issue(s) with The Vibes.

There is a time to vibe, and a time not to vibe. We have to be realistic about the business we operate in and the Vibes that call out to us from the terminal. A person/team can get into serious trouble without strong boundaries and leadership in the age of The Agent. You can very quickly vibe yourself into a corner that is hard to get out of. All of a sudden, those sweet tokens that were once an offering of incense sent up to Saint Anthropic will become a bitter pill.

Here are some extermely obvious gotchas.

The problem is that Agentic Coding is a new frontier, pun intended; we don’t know, and won’t know the full effects until years from now. We know that it’s not going anywhere; businesses everywhere are scrambling to get on the Agentic Bandwagon without much thought for the unintended side effects … if there are some.

I can feel the change in my bones, can you?

After weeks of the Vibes planning a new project, researching, MVPs, spewing out code that works, understanding what went wrong, trying to tweak a complicated piece I never wrote, incorporating more business logic … I can FEEL the difference. I can feel the difference in the solution; I can also feel the difference inside myself when I start on the next project.

I’m not as sharp. My mind is overcrowded, foggy, dull.

Maybe it’s the speed at which we are now required to attempt to comprehend complex solutions and their various impacts on the business, people, and projects. Technical and non-technical processes that took days and weeks collapsed down into ten minutes of tokens. How can you really understand the impacts of what is happening, the side effects, the best choice? Talk to the relevant stakeholders and read between the lines of what they need and want. You can’t.

Share

How to not lose your technical skills when Vibing.

There is a time to vibe, and a time not to vibe. We have to be realistic about the business we operate in and the vibes that are expected of us. We have to pay The Piper that pays us; do what they want. But you do have skin in the game. You have your technical skills and abilities hanging in the balance. One must remember that most of us are just little pawns acting within a cosmic game of chess. You should be wise and understand that your technical skills gleaned over years and decades are what make you … you. They are what make you valuable in the marketplace, set you apart.

Do you really want to walk away from all actions that caused you to learn and grow, just so you can meet a token budget and get the Vibes out 6 hours faster than your coworker? That is short term thinking that will bit you in the !@#$ later.

Say what you want, if you write no code yourself, and simply review code, barely, for weeks, months … a year; will you still be the same programmer a year later? No, you will not. It’s not magic. Practice makes perfect; if you don’t practice, you will lose the skill(s) over time. All that research, talking to stakeholders, writing the minutiae of code, thinking about tradeoffs, MVPs, POCs; when you put your hand to the plow, you get calluses that help you the next time around.

5 steps to keep your skills sharp.

Remember, you are your own best investment; putting time into you will give the best ROI. No one else is sitting around thinking about the you that will exist 10 years from now, what your professional life and skills will look like. Will you be stronger, or atrophy into oblivion?

• Here is what I do to keep my technical skills sharp.

No rocket science involved, or double backflips; just the simple exercises of your skills and mind that will keep it sharp and on point. Not dulled down by King Claude after handing over some of the most valuable skills you have. We learn through experience, through struggle … Agents will steal this from you if you are not careful. You will hand over your critical thinking skills, research skills, desire and willingness to learn new languages and concepts, and communication with others. Stop doing things at your own peril. You will truly become a mindless tab engineer, no matter how pretty your prompts are, if you stop learning and growing. Don’t let the LLM learn for you.

1. Write some code by hand.

• Take the time to write some code by hand, even if it is on the weekend. It will force you to think, read, fail, try again, and remember the fundamentals. Things you forget about when you no longer write code.

2. Read and consume good technical materials.

I know there is a lot of “AI Slop” out there, some good, some bad; either way, reading and consuming material about other problems people are working on and solving is something that pays back in spades later on down the road. All of the information is like money in the bank that you can draw on later. I read a lot, but a few Substacks like these will keep you informed about what is going on in the greater data and software community.

• and the Data Engineering Weekly is a must.

• The and The Pragmatic Engineer is gold.

• and Intersting Links in the Data and AI World.

Of course there are many more; I wrote this article on other blogs I follow if you want some more ideas.

It’s fair to say that you can’t read enough, and that is the truth. It expands your horizons, opens you up to new technologies and solutions, and what other people are working on; it gets you out of the box that you work inside.

3. Learn new tools and languages. Anything new.

When I mean anything, I mean anything. The reason it’s important to learn new things is that there is usually a fair amount of struggle and work involved. You can’t ask for a better teacher than something hard that you have to earn. It could be a whole new programming language, maybe a new package you haven’t tried; sometimes the simple act of solving a “new-to-you problem”, even in a caveman-like way, is going to keep your wits sharp.

The worst place youd could possibily be is waking up 1 year from now, with a wonderful Agentic workflow, using the same old prompts and same old tools. Realizing you … I mean YOU … haven’t hard to struggle to learn a new concept or tool once.

Fight against that tendency to be lazy; push yourself to be uncomfortable; find yourself in a position where you don’t know what you are doing or what the right answer is. This is the best place you could possibly be. Others will be growing lazy and overfed on their generous allowance of tokens, unable to program their own way out of a box within a short time.

Subscribe now

4. High-level project planning + critical thinking.

The other important part of software projects that is only quasi-technical but core to the way we can deliver good solutions, on time, on budget, that meet our end-users' needs … is project planning … and the closely tied skill of critical thinking. Project planning in software requires not only technical know-how, but a wide array of soft skills that reach far into other areas and groups inside whatever business you find yourself in.

• Understanding and obtaining user requirements and desires.

• Understanding the C-Suite and how to play their games.

• Thinking about the best solution that can be delivered and how to say no.

• How to communicate with others, how to lead by example, and how to bring others along.

• Making critical tradeoffs in technical areas and business areas, weighing options and making decisions.

• The list goes on.

I have used Claude many times to help me plan and suss out the details of many large-scale projects involving multiple people. Sure, it feels good at first; you feel like you’re getting somewhere without much effort … the only problem being YOU didn’t put the time or effort in to think through the options and plan yourself, so you get to watch it slowly unravel, show cracks, and generally feel ill-prepared for the future ahead or answering questions and solving problems that inevitably come.

5. Write and communicate with LLMs.

I want to let you in on a little secret. Shhh … don’t tell anyone; it’s a superpower. Did you know that when trying to communicate an idea or message yourself, when you actually write it out yourself or speak it, you actually have to understand it at a deeper level than if an LLM does the work for you?

Shocker, I know.

For example, when I wrote to you about Cloudflare as a Data Platform, I knew exactly what you did beforehand. Nothing. Zilch. Nada. You know what I knew when I finished? More than I did when I started. Also, alot more than some talking head on Reddit who’s never actually DONE ANYTHING without Cloudflare. Suprise.

If you’re writing a Slack message to a co-worker or boss, and you have to explain why you did something the way you did … it makes you stop, think, consider. Does this communicate what I want? How will they take it? Do I need to do something else to make this clearer? There are few people more valuable in the marketplace than those who can communicate clearly and are good at it from practice.

At the end of the day.

Do what you want at the end of the day; you are your own boss, and you get what you put in. We must just learn to ride the winds and waves of popular opinion in the world of CTOs and do what we are told like good little hobbits. But we must remember that if we turn over the minutiae of how we go about our work day to day, we might wake up and find ourselves worse for wear. I suspect in a year or two, many a once mighty Senior Engineer will find themselves slipped back into the Slough of Despond to wrestle and battle with the Juniors.

You are smart; you know how you learned and gained experience to land where you are today; it’s not a secret. You are the one who did it after all. Consider those things. Remember your first love. Code. Read. Learn. Grow.

I know, probably giving whiplash: one week I'm pontificating on some new tool, the next, like this, writing a thesis on the philosophical question of “Is data modeling dead?”

I've been thinking about this more recently, after talking with Andreas and hearing the strong reactions he has to his belief that data modeling is (dead) overrated. Maybe he's right. But we also have the indelible finishing a book on data modeling, and he's the premier thinker of this data decade.

Truth be told, differing opinions in the data space about pretty much any topic are common and widespread. We all are just the victims of the technologies and jobs/work we’ve been a part of. Data Modeling is as old as the “data profession”

I mean, if the classic “relational model” for data was being written about in 1970, then we are indeed handling a topic that has withstood the sands and flying bytes of time and space. It makes one wonder if in another 50 years, on Mars, someone will be writing another article about the death of data modeling.

Anyways, back to today.

Clearly, there has been an increased conversation around the topic of data modeling over the last handful of years; it seems like COVID got deep into the brain and made us question everything. It seems like the conversation around data modeling has not been how to do it, but whether it is even relevant.

Todays sponsor is Estuary. Without them this content isn’t possible. The best way to support this Substack is to click the links below and give Estuary a try.

Pipelines built by an agent, orchestrated by you

Estuary Agent Skills lets your AI assistant draft production-grade data pipelines from a plain-language description, right inside Claude Code, Cursor, or the tool you already use. The output is config in your repo, not a black box. You review the diff, approve it, version it, and roll it back like any other infrastructure. Same runtime as every other Estuary pipeline. No separate “agent mode.”

Install the skill and ship your first pipeline →

Is data modeling dead? Overrated? Will it ever return to its former glory as in the days of yore? Was it ever really alive? These questions have been making the rounds for a few years now; maybe it was COVID that did it.

Even in recent years on Reddit this has come up. Ah … Reddit, methinks it’s the new StackOverflow. A place to get raked over the coals and hangout inside an echo chamber. Yet, I do find myself often scrolling that mucky muck.

Why is the question even being asked?

I don’t know if data modeling is dead or not. Sorry, if that is what you came here trying to find, might as well go back to scrolling Instagram or TikTok, maybe YouTube Shorts if you’re a gray-bearded millennial like me. I will just attempt to present and explore the idea of data modeling, and if it is dead, and let you make up your own mind. Maybe do a survey at the end.

Let’s start by asking the simple question of “Why is this even being asked?”

There has to be some logical explanation, somewhere, of why this has become a topic to even discuss, and seemingly a hot one at that. I believe the answer to this question is fairly obvious. As per this Reddit comment about the reason for data modeling being seen as dead … it’s just become a skill that hasn’t been valued, or at the very least, less valued than it once was.

Well, skills rise and fall in time with the popular technologies of choice. Where have we come from the last few decades? SQL Server, MySQL, Oracle, the data warehouse toolkit. Data was SQL. Data was data warehouse. Kimball. OLTP and OLAP. You couldn’t really be a “data professional” in the early 2000’s and before unless you were an expert in data modeling.

The data didn’t live in S3. It wasn’t found in Parquet or JSON files. The data was, by in large, found in relational databases.

Then the data lake happened; the lake house soon followed. Spark arrived on the scene. We wanted better programmers, systems designers, APIs, streaming; the relational databases were, and are, still here, but they moved from the top of the popularity stack to the bottom … and so did those skills … like data modeling. I’m not saying this was right or wrong; I’m just saying this is what most likely happened. People spent time on LeetCode writing Linked Lists and doing palindromes… and less time talking about the finer points of 3rd normal form.

I would suppose this is fairly normal, as technology and abstractions change and morph, the required skills to be successful, or perceived required skills, are going to change as well.

What we are trying to figure out is this.

What does the data actually say?

The only way I know of if we are all just victims of the talking heads convincing us one way or the other, probably because they have a bent one way or the other, if data modeling is dead … would be to look at the average Senior Data Engineering job requirement we can find online, just take a random sample, and found out if compaines are requesting skills like data modeling or data model. I am not saying that all companies are competent and even know what they need, but at the end of the day, we get hired to do a thing. It’s as good a place to start as any other.

• So, I went and got myself 30 data job descriptions off of LinkedIn for Senior Data Engineer, or close to it.

I recently did an exercise where I proved to myself I’m not a complete idioit and went back to the old days of writing Rust without an LLM (video and audio processing at that) … since that is how I learned it in the first place. Let’s do the same here to find out out of 30 data jobs, how many require data modeling as a skill.

cargo init datajobs
cd datajobs
cargo add glob

We should be able to write a simple Rust program that …

• list a directory of files

• read a text file

• look through text for a word(s)

• accumlate a counter for jobs that have that word

Ok, since I’m a Python guy at heart, we will use the glob create.

use glob::glob;

fn main() {
    let files = glob("data jobs/*.txt").expect("Failed to get list of files");
    for file in files {
        println!("{:?}", file);
    }
}

Yah, Yah, I hear ya, quit messing around and get to the point.

danielbeach@Daniels-MacBook-Pro-2 datajobs % cargo run       
   Compiling datajobs v0.1.0 (/Users/danielbeach/code/datajobs)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 3.53s
     Running `target/debug/datajobs`
Ok("data jobs/1.txt")
Ok("data jobs/10.txt")
Ok("data jobs/11.txt")
Ok("data jobs/12.txt")

Ok, let’s get to it.

use std::i32;
use std::path;
use std::vec;
use std::fs;
use glob::glob;


fn main() {
    let files = glob("data jobs/*.txt").expect("Failed to get list of files");
    let looking_for = "data modeling";
    let mut job_counter:  i32 = 0;
    for file in files {
        let text = read_txt_file(file.expect("bloody bugger"));
        let in_there = contain_words(text, &looking_for);
        if in_there {
            job_counter += 1
        }
    }
    println!("Out of 30 jobs, {} asked for {}", job_counter, looking_for);
}

fn read_txt_file(path: path::PathBuf) -> String {
    let contents = fs::read_to_string(path).expect("Dang file, dang it");
    return contents
}

fn contain_words(text: String, pat: &&str) -> bool {
    let contained = text.to_lowercase().contains(pat);
    return contained;
}

Nasty little piece of work if I do say so myself. Unlike my friendly local Anonamyous Rust Dev, no on pays me to write Rust. Dang, not what I expected to be honest. All this, including source data is on GitHub if you think I’m lying.

warning: `datajobs` (bin "datajobs") generated 1 warning (run `cargo fix --bin "datajobs" -p datajobs` to apply 1 suggestion)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.17s
     Running `target/debug/datajobs`
Out of 30 jobs, 16 asked for data modeling

Over 50% of the jobs asked for data modeling, that is a lot more than I expected, and is a very good sign for all the folk who started out this article by saying in their mind … “You bloody fool, of course data modeling is not dead.” For all you on the other end of the specrtum, it is what it is, not exactly scientific, but hard to argue with. At the very least, one would have to say that if you are in the market for a data job, and you don’t think data modeling is a required skill, than you are severally limiting your pool of jobs. To cut out over %50 of something before you even start is usually not a good sign.

Subscribe now

Polling the LinkedIn data professionals.

To make sure I was covering my bases and giving both sides of the argument a fair shake at coming out on top, I decided to post and informal poll, well, not even a poll, but ask the question of my 20k+ LinkedIn followers along with the 37k+ followers on Substack what they think about the topic. I kinda wanted to simply lick my finger, put it in the air and see which way the winds of popular opinion were blowing. I figured it would skew one way or the other, and that is at the very least another good bone to throw on the pile.

Here is the link to the LinkedIn post, and here to the Substack post. I can so that it’s another 80/20 split, most folk affirm that data modeling is an important skill that it is not dead or obsolete.

That’s probably not a suprise to most people, but it was interesting to see the response of folk who leaned maybe more towards what I would call the data modeling is sorta obsolete in the Lake House era. At least from what I could tell reading between the lines. The arguments in this vector seem to stem from a few totally reasonable areas and schools of thought.

• New tools are very fast and “cover up” the need for good data modeling.

• Small data with no data modeling isn’t that problematic.

• Distributed compute with something like Spark and OBT (One Big Table) works well enough.

To be fair, we must give this school of thought some serious attention, because if nothing else, there is a prevading sense in the data community that data modeling is not taken as seriously as it once was. Things like Medallion Architecture are often passed off as “good enough” for a data model in the Lake House, no more work needed.

I have a few thoughts and takes on why we are even having this dicussion, and I think most discussion are valid and worth having. Sure, from what we’ve seen so far, data modeling as a technical skill is far from dead. Over %50 of data job listings ask specifically for that skill, and the powers and thought leaders of our time seem to lean towards the view that data modeling is indeed crucial.

The origins of the “Data Modeling is Dead Theory”

I think to explore both sides of this argument fully, we have to park on the data modeling is dead theory for a little bit and try to understand that point of view, and where it comes from. Either way, we can better argue for our viewpoint if we are fully versed on both sides, rather than just propping up our strawman.

I’m going to list some bullet points that I read in online comments to my posts, and others, that are more in favor of the data modeling is dead theory. You can take them as you will.

• The last “bible” of data modeling (Kimball’s Tookit) was published in 1996.

• 15 years ago we lived in a SQL Server, Postgres, Oracle, MySQL dominated relational database world in data land. That is no longer the case.

• Many relational database data modeling techniques were designed for performance bottlenecks that no longer exist.

• Data technology and compute has changed siginficantly; think DuckDB, Polars, Datafusion, Spark, blah, blah.

• The hot choice for data storage is now the Lake House … aka Iceberg and Delta Lake which are file storage systems.

• As compared to two decades ago, data has become more complicated in both size, volume, and velocity.

• Many people are running Lake Houses with a hodge podge of data modeling, or none at all, and it works*.

Again, you can have a totally different viewpoint on these bullet points, but we must deal with that at some level, argue against them, just try to generally understand them and how to fit into our metal model that data modeling does indeed hold an important part of the data platform. Back in the day, our Data Warehouse exsisted on SQL Server and if our data model sucked, our queries didn’t run, the reports were slow, and you would get fired by the CTO.

Today you can have One Big Table sitting in parquet files on GCS, point Big Query at them, and get the answers you need well enough. It might not be cost effective, but it’s possible the CTO as such an organziation could care less, as long as that line item stays in the same spot of the budget and doesn’t get out of hand.

Does the data model need to change with tecnology?

This is a valid question that comes up often as well. Does the data model; Kimball, Imon, dreamed up decades ago need to be adjusted for the new world we live in? I’m not saying yes or no, I’m just asking the question. I’ve lived through these data changes, I cut my teeth on data teams running the Data Warehouse on SQL Server clusters, where we tuned every index and query, and were manical about the data models at every layer.

We didn’t have a choice, the whole ship would slow to a halt if we screwed up the data model.

No one has become the data modeling messiah of the 2000’s. It’s been an empty void of talking heads regurgitating the OLTP and OLAP bullet points we learned from our forefathers. When is the last time I read an article with some data influncer going over the finer points of Third Normal Form? Yeah … doesn’t happen. So, if we want data modeling to be taken seriously, then the data community needs to teach it and apply it to the Lake House archteciture so the data engineers are not left in the dark to feel their way about the bits and bytes finding their own way.

Subscribe now

What if data modeling is more about the business (than the technology)?

For someone like me who grew up with the Data Warehouse Toolkit next to my literal Bible on my nightstand, I feel like it’s possible the entire data modeling conversation, in all its technical glory around DuckDB, Spark, Databricks, Snowflake … or whatever, is missing a core componet.

This componet rarely gets brought up in the argument from the “Data Modeling is Dead” crowd. That part is … The Business. I would like to quote who’s written extensively on this topic and is the defacto expert of our age.

This has always been the problem of engineers who focus to much on the technical aspect of any topic, not that any of those technical things are not important. It’s just that to create good Data Platforms and code, we must remember we are here to serve “the business.” Everything we do happens within a context. All those annoying data models are only half for tools, the other half are for the … PEOPLE. Us. Me. You. The collective WE.

We model our data after business logic to reduce errors, help humans and AI trained on humans, better understand context, answer questions, and make sense of the unknown using good data models.

Is data modeling overrated?

Beauty is in the eye of the beholder, it depends on the context of you, the company you work at, the tools you find yourself using every day. I think it’s a sliding scale. Maybe you find yourself in Postgres or SQL Server, than of course you’re going to have a high view of data modeling. Maybe you’re working at some marketing company who dumps stuff into JSON files on GCS and just throws BigQuery on top and is off to the races. Maybe they don’t want you to spend time data modeling.

I guess you have your answer then.

But, maybe you’re working on a Databricks or Snowflake platform, your data is large, there is a new wave of agentic solutions that need to be given access to a complex and vast array of data. Than the data modeling is probably going to make or break your data platform success.

Can you get away with not data modeling “very much.” Yeah, if you are using Spark ontop of Iceberg with just a handful of tables, yeah … you will probably survive … but not thrive. Maybe you have 100GB of data and you are using DuckDB and Polars ontop of Parquet files in cloud storage, and it’s just easier to and faster to get the answers the way the data landed.

• Sure, it probably works, as long as you have 5 reports that no one looks at.

If you spend time to curate your data, understand it, massage it into meaningful formats and models, yeah, you’re probably in the top 10% and you get returns on your data modeling investment. I myself eat my own words, today, I work in a place were we made the concious decision to run a Kimball style data modeled Lake House with Facts and Dimensions as if it was SQL Server form 30 years ago. That was probably just my pre-concieved notions and background playing into it.

I recently had a bit of a self-identity crisis; of course, it was tied to spending way too much time on social media. I do try to avoid the worst of it, got rid of Twitter (X) this year, and mostly stick to Substack, LinkedIn, YouTube, occasional Reddit, and Hacker News. Probably still too much.

Anyways, I’ve always been one to keep as close to the bleeding edge as possible throughout my career, at least from my view, without wandering over the edge. I’ve always seen it as a form of self-preservation, probably due to growing up the youngest of five in the middle of the prairie next to the river, in the only county without a working stoplight.

I try to be pragmatic and never dogmatic, except when someone says “Production” and “Notebook” in the same sentence.

Again, I digress; what I’m trying to say is that I miss writing code. Myself. Without a machine looking over my shoulder. Rust, I miss that nasty little borrow checker and compiler. I miss struggling.

If life writing code for two decades has taught me anything, it’s that struggle and exercising one’s mind are critical to success, personal and professional, and I’ve felt it sorta creeping up on me, slowly and sneakily.

• Don’t get me wrong: spending time on systems design, architecture, business requirements, project planning, the big picture, nitpicking Claude solutions, and providing value through software solutions … it’s important work.

But I can only accomplish those tasks well insofar as my current mind is continually exercised and acquainted with the intricacies of software used to solve data problems and provide value. Building systems that run well, are cost-effective, and are generally a pleasure to work with.

Customer-facing analytics that ship in days, not quarters

Customer-facing analytics are slow, the warehouse bill keeps climbing, and every new chart turns into an engineering project. Cube is the agentic analytics platform built for embedding: drop governed dashboards and an AI analyst into your product in days, with built-in caching that keeps queries fast and warehouse costs down. Pelago, Constant Contact, GoFundMe, and 400+ other SaaS companies ship customer-facing analytics on it.

Get started for free.

Check out Cube today!

I, for one, find it surprising how quickly those finer details seem to fog over in the Cold Claude War we find ourselves in. Sure, it’s hard to erase 20+ years of code and experience, but it can get a little opaque.

So today, I’m here to do a simple thing with you. Bust out the terminal, vim, Rust, cargo … and solve a problem I deal with daily and find annoying. The most important part is that my own two fingers, with a few Google searches, are going to do this work.

WARNING! No LLMs were used in generating the following code.

The problem.

So, what is this itching problem I deal with way too often each week? Well, I’m sure, or I hope you are, that, along with the written word I produce for you, more than I should, includes a lot of Video/Podcasting content. Substack Podcasts, YouTube, etc.

The problem is simple: I need the transcripts from my video and podcast recordings. For all sorts of reasons. I need the transcript to summarize and find …

• What I/we talked about, aka topics covered

• figure out a good title and description

• etc

There are a lot of AI tools I can pay for to do this for me, but I want my own, My Precious. So, we are going to whip out Rust and bang our heads against a wall until we figure this out.

cargo init transcription-rs
cargo add unbundle

I’m thinking we just want to accomplish a few simple things …

• command-line CLI to run this tool

• reduce dependencies

• rip out audio media from a video file

• … try to transcribe the audio file

I’m excited to write some Rust code without any LLMs, to think through the problems and struggle on my own, like the good old days.

• I found a Rust crate that uses the powerful and apparently popular FFmpeg, which has been used since the beginning of time.

Re-learning Rust without LLMs.

Let’s start our journey by using this crate to do something simple, knowing it probably won’t be our final solution. But can we open an MP4 video file, a clip from one of my podcasts, and have an audio file ripped from it?

use unbundle::{self, MediaFile, AudioFormat};
use std::env;

fn main() {
    let args: Vec<String> = env::args().collect();
    let input_file = &args[1];
    println!("Starting to work on {input_file}");
    let video_data = unbundle::MediaFile::open(&input_file).expect("Failed to open video file");
    process_video(video_data);
}

fn process_video(mut video_data: MediaFile) {
    video_data.audio().save("output.wav", AudioFormat::Wav).unwrap();
}

We will probably need to use clap for proper CLI features at some point, but raw arg input parsing will work for now. So far, it’s pretty straightforward, at least in proving out our idea.

• It’s a good reminder of Rust's core principles.

When it comes to mutable and immutable, we can think of our data and variables in such terms. This is far from the land of Python, where everything can be anything and do whatever. Also, another Rust beauty is borrowing: can we borrow a copy of a pointer to some data or thing? For example, our argument and variable for input_file should be immutable and borrowable. No need for it to change.

This is in contrast to our video_data, which holds the data from our MP4 file; it is mutable because we need to extract the audio data from it.

< Always thinking about data and how it’s used. >

• Immutable data …

  • A file_name would/should have no need to ever change.

  • We should probably always just borrow the file_name if we need it

• Mutable data …

  • If you scroll down a little, you will see we eventually declare video_data as mutable, because we need to rip the audio off it.

One thing I noticed was that this solution depends on the machine having a proper installation of FFmpeg. I would prefer a pure Rust implementation, but maybe we can deal with that later. Let’s run our little bitty pieces of Rust and see what happens.

danielbeach@Daniels-MacBook-Pro-2 transcription-rs % cargo run -- test.mp4
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.06s
     Running `target/debug/transcription-rs test.mp4`
Starting to work on test.mp4

• Well, our hacky little Rust script works; here is our output.wav audio file ripped from a video MP4.

Next, transcription: hackey is fine, but we need to pull text from the audio file.

Transcription with whisper-rs.

One thing I enjoy about hand-rolling code is not handing off the thinking part to our benevolent master Cladius Maximus. Working on scratching out a version of what I want requires me to do my own research into what others have found useful for solving problems.

The problem with The Vibes is that we don’t question the LLM’s choices enough. We tab and move on. What causes the invisible gremlins, the weights trained in some dark evil frontier lab, to pick one tool over another for solving a problem?

• Those reasons won’t be the same for a human.

OpenAI has a model, Whisper, that is a general-purpose speech recognition. Look, maybe we will turn back to the models, but not for our code.

Looks like we have some pretty small Whisper models available, which is great; we can probably use anything from tiny to base.

Of course, we could find the Rust wrapper for this tool, basically the Rust bindings to whisper.cpp

Ok, so let’s see if we can add some more hacky Rust code to get whisper-rs to extract a transcript from our audio file. Don’t worry, we will revisit the code at some point to do some refactoring, but I’m mostly just thinking about getting the end-to-end working at the moment.

• Doing things “right” can come later.

Ok, let’s download one of the Whisper models onto our machine.

mkdir -p models
curl -L -o models/ggml-small.en.bin \
  https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-small.en.bin

Easy enough, now for the Rust bindings.

use unbundle::{self, MediaFile, AudioFormat, AudioHandle};
use std::env;
use whisper_rs::{WhisperContext, WhisperContextParameters, FullParams, SamplingStrategy};


fn main() {
    let args: Vec<String> = env::args().collect();
    let input_file = &args[1];
    println!("Starting to work on {input_file}");
    let mut video_data = unbundle::MediaFile::open(&input_file).expect("Failed to open video file");
    let audio_data = process_video(&mut video_data);
    
    let path_to_model = "models/ggml-small.en.bin";
    let audio_samples = pull_audio_samples(audio_data);
    pull_transcript(path_to_model, &audio_samples);
}


fn process_video(mut video_data: &mut MediaFile) -> AudioHandle<'_> {
    return video_data.audio();
}

fn pull_audio_samples(audio_data: AudioHandle<'_>) -> Vec<f32> {
    let samples: Vec<f32> = audio_data
      .sample_iter().expect("Failed sampling")
      .flat_map(|chunk| chunk.unwrap().samples)
      .collect();
    return samples;
}


fn pull_transcript(path_to_model: &str, audio_sample: &Vec<f32>) {
    let ctx = WhisperContext::new_with_params(
        path_to_model,
        WhisperContextParameters::default()
    ).expect("failed to load model");

    let params = FullParams::new(SamplingStrategy::BeamSearch {
        beam_size: 5,
        patience: -1.0,
    });

    let mut state = ctx.create_state().expect("failed to create state");
    state
        .full(params, &audio_sample[..])
        .expect("failed to run model");

    let mut transcript = String::new();
    for segment in state.as_iter() {
        transcript.push_str(&segment.to_string());
    }
    println!("{transcript}");
}

You can see that not much has changed, just the addition of whisper. When we process_video we return the raw audio, then we chunk that raw audio data out into a Vector, this allows us to pull_transcript with whisper.

Running the command.

PATH="/opt/homebrew/bin:$PATH" \
  PKG_CONFIG_PATH="/opt/homebrew/opt/ffmpeg/lib/pkgconfig" \
  cargo run -- test.mp4

This is hilarious.

whisper_full_with_state: decoder  4: score = -5.57022, result_len =   4, avg_logprobs = -5.57022, entropy =  1.38629
whisper_full_with_state: best decoder = 0
single timestamp ending - skip entire chunk
seek = 36985, seek_delta = 2141
 I live in the capital of Twerba and Trutar. [growling] I am a woman. I am... my wife. 
[sigh] [growling] [growling] [growling] [growling] [growling] [growling]
 [growling] [growling] [growling] [growling] [growling] [growling]
 [growling] [growling] [growling] [growling] [growling] [growling] 
[growling] [growling] [growling] [growling] [growling] [growling]
 [growling] [growling] [growling] [growling] [growling] [growling] 
[growling] [growling] [growling] [growling]

So, apparently, we have something running end-to-end, although I can tell you this is not how the conversation went. Maybe with the exception of “my wife. [sigh].”

Clearly, we have a frigging in the rigging somewhere, but I see this as a positive: the fact we have ~50 lines of Rust code and can take a video file and get something that is supposed to be a transcript out … well, that’s fun!

I’m no audio or video expert, but one can imagine this has something to do with the audio or video format, probably the audio being pulled from the video. From what I can tell, the Whisper model requires “16 kHz sample rate, 16-bit samples for audio.”

• Great, now I, who know nothing, need to figure out how to resample audio data.

Of course, a quick Google search shows rubato, a small Rust package we can use to resample. The rubato docs show a simple example of how to do the resampling, so hopefully this shouldn’t be too bad.

One thing we should also do is add a check to get the current sample rate, then convert it to the 16Hz required for Whisper to work. I download video files from all sorts of sources: Riverside.fm, Substack, etc. Ok, let’s update our pull_audio_samples method to determine the sample rate of the chunks we are working with.

fn pull_audio_samples(audio_data: AudioHandle<'_>) -> (Vec<f32>, u32) {
     let mut sample_rate = 0u32;
     let mut samples = Vec::new();
     for chunk in audio_data.sample_iter().expect("Failed sampling") {
         let chunk = chunk.unwrap();
         sample_rate = chunk.sample_rate; // same on every chunk
         samples.extend(chunk.samples);
     }
     (samples, sample_rate)
 }

Then, the nasty old complicated resampling; it’s ugly to look at, too verbose for my liking, but it works.

fn resample(input: &[f32], from: u32, to: u32) -> Vec<f32> {
    let mut resampler = Fft::<f32>::new(
        from as usize, // input rate
        to as usize,      // output rate (16000)
        1024, // chunk size in frames
        1,                  
        1,   // channels = mono
        FixedSync::Both,
    )
    .expect("resampler construction failed");

    let in_len = input.len();
    let out_len = resampler.process_all_needed_output_len(in_len);

    let input_buf = InterleavedOwned::new_from(input.to_vec(), 1, in_len)
        .expect("input buffer construction failed");
    let mut output_buf = InterleavedOwned::new(0.0f32, 1, out_len);

    let (_nbr_in, nbr_out) = resampler
        .process_all_into_buffer(&input_buf, &mut output_buf, in_len, None)
        .expect("resample failed");

    let mut out = output_buf.take_data(); // mono => frames == samples
    out.truncate(nbr_out);
    out
}

Can you imagine some Rust wizard sitting in a dark basement somewhere who thought it was fun to write a Rust crate for resampling audio? Good Lord.

Anyways, with these two update/new methods, let’s re-run our script and see if we can get a real transcript back, and not “I live in the capital of Twerba and Trutar. [growling].”

Well, look at that! We got our own decent video-to-text transcription tool! Of course, we could have used a slower, more accurate Whisper model, but this looks good enough for what I need! All I need to do is maybe add another command-line argument to write the transcript to a text file.

use std::fs;

let ouput_file = &args[2];
...
let transcript = pull_transcript(path_to_model, &samples);
fs::write(ouput_file, transcript).expect("Should be able to write to {output_file}");

Thoughts and more.

Well, that was fun, wasn’t it? A little wandering down memory lane to the good old days of fighting the borrow checker and compiler. Trust me, there was plenty of that going on that I didn’t show you.

If you want, as always, you can get this codebase on GitHub.

To be honest, it was good to have those old feelings of not being able to figure something out right away. It took me two days, off and on, during my free time to hack this code together. Remember to exercise my brain about data, mutable or not, borrow it or not, data types … you know … just programming stuff.

Sure, I bet Claude could write this Rust code way better than I, much prettier, catch all the rough edges, who knows what path it would have chosen. That’s ok. I enjoyed the process of researching and thinking myself. Not worried about anyone else but me.

It was a nice reminder of minimalist code: not everything needs a ton of boilerplate, “to be done right,” or anything fancy. Seems like Rust is like riding a bike; it comes back awful quick. Maybe I will make it a habit.

In this episode of the Data Engineering Central Podcast, I sat down with Hugo Lu, founder and CEO of Orchestra, to talk about what “agentic data engineering” actually means beyond the buzzwords. Rather than another conversation about AI replacing engineers, we dug into the infrastructure that’s still missing before autonomous data platforms become reality.

• Hugo shares his unlikely path into data engineering, from investment banking to helping build data systems at Juul, before eventually founding Orchestra.

What started as an effort to simplify orchestration has evolved into a broader vision where AI agents don’t just generate code, but can safely execute work, observe the results, validate changes, and iteratively improve pipelines inside secure environments.

• That ability to observe outcomes, what many are calling “closing the loop,” may be the missing ingredient preventing today’s coding agents from becoming truly autonomous.

We also explore why data engineering has not experienced the same AI disruption as traditional software engineering. While AI can produce application code remarkably well, production data systems introduce a completely different set of problems. Branching production data, validating schema changes, testing transformations against realistic datasets, and understanding business semantics all remain difficult challenges that require much more than simply generating code.

The conversation naturally turns toward the future of the profession itself. We discuss whether junior engineers are losing the traditional apprenticeship path, how senior engineers are shifting from writing code to reviewing AI-generated work, and why decades of experience debugging production systems may actually become even more valuable in an AI-first world. Rather than eliminating engineering expertise, AI may simply be changing where that expertise is applied.

Finally, we dive into the rapidly changing data infrastructure landscape. From DuckDB and Polars to serverless compute, Iceberg, semantic layers, AI-native orchestration, and the growing concern over rising LLM token costs, we discuss where the industry appears to be heading and which trends are likely to stick long after today’s hype cycle fades.

If you’ve been wondering what comes after Orchestration, how AI agents will actually manage production data pipelines, or whether data engineering itself is about to undergo its biggest transformation in a decade, I think you’ll enjoy this conversation.

In this episode we discuss

• What “agentic data engineering” actually means.

• Why AI still struggles to fully automate data engineering.

• The importance of closing the loop with production feedback.

• Why orchestration may become the operating system for AI agents.

• The changing role of data engineers in an AI-first world.

• Why junior engineers face a very different career path than previous generations.

• DuckDB, Polars, serverless data platforms, and where modern infrastructure is heading.

• Whether today’s dependence on proprietary LLMs will create tomorrow’s vendor lock-in.

• How Orchestra is building infrastructure for AI-native data platforms.

You might think I, your friendly neighborhood Anonymous Rust Dev, am having some dyslexic trouble with the alphabet, but alas, no mistake was made. Arrow’s answer to cross-platform database connectivity, the Arrow Database Connectivity API is, to quote Arrow’s docs:

What makes this API special, and why would anyone use it over any other available tool? Let’s find out together as we look under the hood of this thing.

Overview

Arrow’s blog introduced ADBC in early 2023. At the top, in the first sentences, they offer two compelling reasons you might care:

• ADBC is a columnar, minimal-overhead alternative to JDBC/ODBC for analytical applications.

• In other words, ADBC is a single API for loading and unloading Arrow data across different databases. (emphasis mine)

I’ll admit, when I first heard about this, I was hoping it was ODBC-compatible. Being able to point any of my existing applications at it simply by modifying the connection string seems appealing to me, but that’s not quite how it works. Though technically, I guess that’s not really a good way to think about it, anyway; consider the FAQ on some of the benefits of the ADBC approach:

JDBC uses row-based interfaces like ResultSet. When working with columnar data, like Arrow data, this means that we have to convert the data at least once and possibly twice...

In keeping with Arrow’s “zero-copy” or “minimal-copy” ethos, we would like to avoid these unnecessary conversions. [...] ODBC uses caller-allocated buffers (which often means forcing a data copy), and ODBC specifies data layouts that are not quite Arrow-compatible (requiring a data conversion anyway).

Thanks to Delta for sponsoring this newsletter! I use Delta Lake daily, and I believe it represents the future of Data Engineering. Content like this would not be possible without their support. Check out their website below.

Hey, as a Rustacean, I love zero-copy. Still, all I’ve seen so far is what ADBC isn’t. Let’s keep reading:

ADBC is a client API specification. It doesn’t define what goes on between your client and the database, just the API calls that you make as an application developer. Under the hood, a driver must handle those API calls and communicate with the actual database.

...an ADBC driver can be written for a database purely as a client library.

Maybe I’m just dense; I’m still not quite getting it. There’s a lot of energy spent in the FAQ comparing ADBC to Arrow Flight SQL, but that still doesn’t quite click for me. It’s not until I encounter the following that I think I understand what’s going on:

What is the ADBC SQL dialect?

So, in a nutshell, it seems ADBC’s job is to abstract the work of running a query and returning a result set in a consistent (Arrow-based) form. It seemingly doesn’t care about the query itself, but rather primarily marshals your request to its destination and back.

Getting Started

Rust

I’m gonna start by looking at Rust’s quickstart, since that’s what I know; here’s some copy-pasta demonstrating an ADBC query against DataFusion:

// These traits must be in scope
use adbc_core::{Connection, Database, Driver, Statement};

let mut driver = adbc_datafusion::DataFusionDriver {};
let db = driver.new_database().expect("Failed to create database handle");
let mut conn = db.new_connection().expect("Failed to create connection");

To run queries, we can create a statement and set a query:

let mut stmt = conn.new_statement().expect("Failed to create statement");
stmt.set_sql_query("SELECT 1").expect("Failed to set SQL query");

We can then execute the query to get an Arrow RecordBatchReader:

let reader = stmt.execute().expect("Failed to execute statement");
for batch in reader {
    let batch = batch.expect("Failed to read batch");
    println!("{:?}", batch);
}

If you don’t know Rust well, that’s fine, it’s more the flow of the application I’m interested in than the language itself. As with ODBC/JDBC, it’s as simple as establishing a connection, executing a query, and traversing the resulting recordset. It’s a pretty familiar formula for anyone who’s done garden-variety SQL queries against a database.

Preceding this query, we have the one-time setup necessary to get this working:

This is because ADBC depends on drivers, which are provided via Rust’s Cargo ecosystem.

Share

Python

If you’re looking to do things via Python, it’s similarly easy. I’ll pick on DuckDB this time...

Note: According to the ADBC docs, DuckDB work is still in progress; however looking at the referenced PR it seems like it’s been fully implemented.

First, you’ll need to install lib duckdb. The DuckDB docs will point you in the right direction, which for Python consists of:

$ pip install duckdb

Feel free to wipe the sweat from your brow, you’ve earned it. Now, let’s dig into the ADBC documentation for DuckDB. More installation stuff:

$ pip install adbc_driver_manager pyarrow

It should be pretty apparent by now, but because our recordset comes back in Arrow format, we have a dependency on Arrow. After this, it’s ready to go; shown below is their example:

import adbc_driver_duckdb.dbapi
import pyarrow

data = pyarrow.record_batch(
    [[1, 2, 3, 4], ["a", "b", "c", "d"]],
    names = ["ints", "strs"],
)

with adbc_driver_duckdb.dbapi.connect("test.db") as conn, conn.cursor() as cur:
    cur.adbc_ingest("AnswerToEverything", data)

Other languages

ADBC driver support for various platforms is available here. Of note, the stable platforms, which generally have PyPI packages available, currently include:

• Apache Arrow Flight SQL

• DuckDB

• PostgreSQL

• SQLite

• Snowflake

How it works

Just a forewarning, the ADBC API Standard documentation focuses primarily on the following languages: C/C++, Java, and Go.

Other languages, such as Python’s adbc_driver_manager, wrap around the C API and offer low-level bindings.

The glossary is also helpful at understanding what the moving parts are. In particular, the driver, driver manager, and wire protocol terms are helpful to understand when dealing with ADBC; in a nutshell: you use a driver manager to handle the selection of drivers (the libraries responsible for connecting to a particular database engine), which themselves implement the correct wire protocol (the implementation details of connecting and transferring data specific to a given database platform).

Or, to put it another way, the driver manager abstracts away the work of interfacing directly with the driver; see the following from the How Drivers and the Driver Manager Work Together document:

...the overall recommended approach is to use the ADBC driver manager. This is a library that pretends to be a single driver that can be linked to and used “like normal”. Internally, it loads the table of function pointers and tracks which database/connection/statement objects need which “actual” driver, making it easy to dynamically load drivers at runtime and use multiple drivers from the same application...

As seen in the Rust and Python examples, the workflow is similar to many OLTP workflows (e.g. ADO or JDBC): establish a connection, prepare a query, execute or fetch results, and enumerate data over a reader interface.

While it’s tough to find in the documentation, my impression is that for row-based data (e.g. PostgreSQL), the ADBC driver is performing a conversion into columnar Arrow data. Google’s AI doesn’t really seem to echo this, though:

My suspicion is that the conversion is happening somewhere down the line, but the AI overview separately describes the Arrow format as a distinct selling point.

Subscribe now

Why ADBC?

As shown above, Arrow's zero-copy nature provides direct access to the dataset returned by the drivers. One of Arrow’s big promises is that it’s an in-memory format that can be transferred between layers as-is... I mean, it’s literally on the Arrow landing page:

Apache Arrow defines a language-independent columnar memory format for flat and nested data, organized for efficient analytic operations on modern hardware like CPUs and GPUs. The Arrow memory format also supports zero-copy reads for lightning-fast data access without serialization overhead.

This is the lingua franca of ADBC. Where another typical database driver might need to fetch data using a C library, and then convert it for use in another language like Python, with Arrow it’s merely a matter of passing a memory pointer around and letting the Arrow library for a given language make sense of it.

Another selling point is the consistent API. In practice, each language has its own concerns and may take some liberties, but just like Arrow itself, if you can master the ADBC API in one language, that knowledge transfers over to the others.

I’d also remind you of a recent interview between Dan and Wes McKinney, one of the principal authors of Arrow.

Michael Drogalis chose a different path.

After helping build technology in the Kafka ecosystem, founding a startup that was ultimately acquired by Confluent, and leading product for stream processing, he walked away from big tech to see if one person could build a serious B2B software company.

• The result became ShadowTraffic, a product that helps engineering teams generate realistic production traffic for testing, demos, and development.

In this conversation, we talk about much more than streaming systems. We discuss why most engineers underestimate the importance of understanding customers, how AI is changing software development without replacing experienced engineers, what it takes to market technical products, and why writing publicly can become one of the biggest accelerators of your career.

If you’ve ever considered building your own product, becoming a solopreneur, or simply becoming a better engineer, this conversation is packed with practical advice from someone who’s actually done it.

I think this episode has broad appeal beyond data engineering. It’s really about engineering careers, entrepreneurship, and building products that solve real problems, which should make it one of your more accessible interviews.

• Building and selling a Kafka startup

• Life inside Confluent during its rapid growth

• Why Michael left big tech to become a solopreneur

• Building ShadowTraffic from scratch

• Finding customers before writing code

• Why marketing matters more than most engineers think

• Using AI without becoming dependent on it

• The future of software engineering

• Writing online and building an audience

• Advice for engineers who want to start their own business

I know calling Cloudflare the new kid on the block is a little tongue-in-cheek, but it seems fair when applied to Data Platform tooling. My guess is that when someone says "Cloudflare," the first thing that comes to mind isn't data. They’ve become a household name with the literal power to take down the internet with no notice.

Best known as a digital middleman between a website's server and its visitors, it seems they are sticking their fingers into other honeypots … namely Data.

In the past, I poked at Cloudflare’s R2 Iceberg setup, but that was at least a year ago, and it was more about Apache Iceberg than about Cloudflare as a proper Data Platform.

To be honest, it’s probably one of those “better late than never” situations, but they sure are late to the “one-stop shop for data” party. Everyone else has been on the Lake House bandwagon for a good, solid 5 years now. All the big players need to have a hand in the data space.

We live in the age of AI, and data is the gold behind the Agents/LLMs, so it makes sense that players like Cloudflare don’t want all the data to leave their systems to other players. It makes sense as a business to provide customers the entire Data Stack, so they have at the very least, the option to stay put.

This is what I want to cover today; it’s as much for me as for you. I want to know …

• What data products does Cloudflare offer?

• What it looks like to build on each of those products.

• Can I build a complete Data Stack with those tools?

• Pricing.

Who knows if we will make it through all of that before I run out of patience and start ranting. You know me; it’s a toss-up. You can trust me to call it like it is. I’ve built plenty of Data Stacks from scratch, so I plan to approach this like I would anything else.

We will see what works, what doesn’t, what’s missing. Or it will all be rainbows and butterflies. Never say never; it’s possible.

Customer-facing analytics that ship in days, not quarters

Customer-facing analytics are slow, the warehouse bill keeps climbing, and every new chart turns into an engineering project. Cube is the agentic analytics platform built for embedding: drop governed dashboards and an AI analyst into your product in days, with built-in caching that keeps queries fast and warehouse costs down. Pelago, Constant Contact, GoFundMe, and 400+ other SaaS companies ship customer-facing analytics on it.

Get started for free.

Check out Cube today!

What data products does Cloudflare offer?

Ok, so when I said Cloudflare is selling the entire “Data Platform,” I wasn’t just making it up. You can see it front and center on their webpage, which actually makes our job a little easier.

This is actually something many companies forget to do. Cloudflare took the extra step of telling us exactly how they envision a Data Platform working with their products. This is helpful because we don’t have to guess their intentions or the best way to build out a Data Platform on Cloudflare. Architecturally, it’s good to have a starting point.

Ok, right off the bat, from their docs we can see that they mostly focus on “Data Warehouse and Lake House.” This consists of a few core technologies they mention and directly takes center stage in the docs.

1. R2 for Storage (think s3)
2. Apache Iceberg for storage layer (classic)
3. R2 Data Catalog (for .... Data Catalog) <- this is new to me
4. R2 SQL (SQL Query Layer)
5. Pipelines ??? (sorta ambiguous)

Their docs are surprisingly straightforward and simple, and the concepts are very easy to understand. This is a breath of fresh air in the never-ending glut of marketing announcements about name changes and new AI-centric features that get poured onto our heads on a weekly basis.

I do have to mention on gaping hole that jumps out to me conceptually that isn’t forefront in the docs, yet is core to any Data Platform … and I’m not sure how to take it.

What about the actual Data Pipelines? How exactly should we write code to write, transform, and read data within the Cloudflare Data Platform? Their landing page mentions “Pipelines,” but their docs list R2 Storage, Iceberg, Data Catalog, and R2 SQL, making no mention of “Pipelines.”

Again, from my point of view as a newcomer to Cloudflare as a data platform, and their tantalizing mentions of Pipelines, and one would assume the actual creation of data pipelines being core to building a Data Platform … why is it a mystery or hidden compared to the other parts of the platform like …

• Storage with R2

• Iceberg as the lake house format

• R2 Data Catalog as the Data Catalog

• R2 SQL as query layer …

The Pipeline … the code … is the glue that brings all these things together. Conceptually, why no overview of the “glue” … the “Pipelines” as they called them on the landing page?

So, overall, what we have so far conceptually from Cloudflare is …

Again, everything here is straightforward; my biggest questions are around rounding out the Data Platform. Some of the things rattling around in my head …

• Can we use Apache Airflow to trigger stuff inside these tools?

• Otherwise, how do we schedule and orchestrate the Data Platform?

• Is R2 SQL just a glorified Notebook?

• What are Pipelines, and how can we write code that interacts with this Data Platform?

Instead of answering these questions off the bat, which would be smart, I just want to walk through each data product they offer and build out a fake Lake House. Eventually, we will be forced to answer my questions or find a way around them.

You can only read docs for so long, at some point it’s best just to try things out.

Subscribe now

R2 Storage + R2 Apache Iceberg.

Ok, so we will start at the bottom and work our way up. Also, I’m going to use the UI for this, so I can get used to Cloudflare as a platform and make sure I’m not accidentally missing anything when setting up the pieces of the Data Platform.

First, let’s create a bucket from which we can hopefully install or enable Apache Iceberg. Right away, I noticed something useful: a reference to the Wrangler CLI, which we can apparently use to interact with R2 and, I’m guessing, other parts of the Cloudflare system.

Easy enough to create, and I also saw a Settings page with some info about Iceberg and the Data Catalog.

Enabling the R2 Data Catalog popped up a box,

Next steps: R2 Data Catalog can automatically handle table maintenance,
 such as compaction, to keep your R2 Data Catalog's performance optimized. 
All we need from you is a Token to grant our table maintenance service 
access to your catalog resources. We can create one on your behalf,
 or you can enter your own.

Now we have a Apache Iceberg Rest API URI we can use for the typical Data Catalog actions we are all used to.

So at this point we have two simple fundamentals of classic Data Platform, Lake House style setup …

• R2 Storage Bucket

• Apache Iceberg Catalog

  • which apparently handles table maintenance like compaction

• Data Catalog URI

So, next, let's create some Apache Iceberg table(s), and then we can figure out how we would like to ingest data, which, by extension, may become our data pipeline setup.

Well, almost.

In my mind, I wanted to use Apache Airflow, combined with something like Polars or DuckDB, and connect to the Cloudflare Data Catalog for REST Iceberg, and do it that way, but I really, really wanted to be able to find those “Pipelines” that were mentioned.

Finally, when clicking around in the UI, I saw a line for them under Storage and Databases, why there I don’t know.

Anywho, this is going to take us down a tangent, but we might as well start the trip and get it over with. You can’t access Pipelines on the Free Tier I’m on; you have to upgrade your plan. Here is the list of benefits.

It’s a lot to digest.

Always a little scary buying stuff for y’all. Better turn off that auto-compaction on Iceberg tables, eh? Probably spin up a big @!#$! cluster to do and then send me the bill. I know how these SasS vendors work.

Anywho, I would go through all those features for you, but I ain’t got the time. I just want to get to building our Data Platform, and I would like to use Pipelines to do it.

Here is the Worker page, at least they give me a billing dashboard so I can keep my powder dry.

Apparently, this means we need to learn something about Cloudflare Workers.

• This does appear to be the missing piece of our puzzle on the Data Platform. I’m assuming we can use Pipelines alongside Workers to build our data flows.

Also, I noticed they mentioned cron jobs and durable Workflows, which sounds like what we need, but it’s also confusing they don’t mention Pipelines, what are the differences between the two? Not sure.

Anyways, it appears we need to read the documentation on Pipelines, and they fit generally what we are looking for.

Pipelines, as we can see, connect streams and sinks, data sources and data targets. “Streams” is a loaded term. I know Cloudflare bought Arroyo, a Rust-based streaming platform, sometime back. I’m curious if these pipelines can handle batch as well.

I get that people like Streams and streaming pipelines, but 80%+ of most data pipelines, if not more, are batch-based workloads. It appears “Pipelines” are now useless to us … since … “Pipelines connect streams and sinks via SQL transformations.”

• Even the Cloudflare docs for Pipelines mention nothing but streams, the tables they use for the examples are even called streams.

I’m not saying this is a totally bad thing; what I’m saying is that Cloudflare Data Platform seems to be built for streaming use cases almost exclusively. What do I know? Maybe this is the niche of the data world they want. Everyone under the sun, including Databricks with ZeroBus, has a streaming offering bundled into their Lake House offerings.

Not that Cloudflare cares what I think, but if they want to make real in-roads in the data community, batch pipeline support is key.

We gotta do something.

Well, we can’t let these hobbits stop us form getting a Cloudflare data platform running. Let’s just make our own pipelines, workers with Apache Airflow, and then use R2 buckets, Iceberg, and Catalog for our Lake House.

Let’s get cracking. Start with the end in mind; isn’t that what they say? Here is our Airflow DAG for our Cloudflare lake house. One failed, and finally, a good run.

I know it’s just a plain old Airflow DAG, but if we were building a real lake house, the ability to use a classic orchestration tool like Airflow would always be a good sign.

We can work our way from the top down this time. Let’s look at the Airflow DAG.

from __future__ import annotations

import os

import pendulum
import polars as pl
from airflow.sdk import dag, task

from include.iceberg_catalog import (
    ANALYTICS_SCHEMA,
    ANALYTICS_TABLE,
    NAMESPACE,
    RAW_SCHEMA,
    RAW_TABLE,
    create_table_if_not_exists,
    ensure_namespace,
    get_catalog,
)

_EARTH_RADIUS_KM = 6371.0088


def _haversine_km() -> pl.Expr:
    """Great-circle distance (km) between start/end coords; null when coords null."""
    lat1 = pl.col("start_lat").radians()
    lat2 = pl.col("end_lat").radians()
    dlat = (pl.col("end_lat") - pl.col("start_lat")).radians()
    dlng = (pl.col("end_lng") - pl.col("start_lng")).radians()
    a = (dlat / 2).sin() ** 2 + lat1.cos() * lat2.cos() * (dlng / 2).sin() ** 2
    return (2 * _EARTH_RADIUS_KM * a.sqrt().arcsin()).alias("distance_km")


@dag(
    schedule=None,
    start_date=pendulum.datetime(2020, 4, 1, tz="UTC"),
    catchup=False,
    tags=["cloudflare", "iceberg", "polars", "divvy"],
)
def cloudflare_iceberg_pipeline():
    @task
    def create_tables() -> None:
        catalog = get_catalog()
        ensure_namespace(catalog, NAMESPACE)
        create_table_if_not_exists(catalog, RAW_TABLE, RAW_SCHEMA)
        create_table_if_not_exists(catalog, ANALYTICS_TABLE, ANALYTICS_SCHEMA)

    @task
    def load_raw() -> int:
        csv_path = os.environ["DIVVY_CSV_PATH"]
        df = pl.read_csv(
            csv_path,
            schema_overrides={
                "ride_id": pl.Utf8,
                "start_station_id": pl.Utf8,
                "end_station_id": pl.Utf8,
            },
            try_parse_dates=True,
        ).with_columns(
            pl.col("started_at").cast(pl.Datetime("us")),
            pl.col("ended_at").cast(pl.Datetime("us")),
            pl.col("start_lat").cast(pl.Float64),
            pl.col("start_lng").cast(pl.Float64),
            pl.col("end_lat").cast(pl.Float64),
            pl.col("end_lng").cast(pl.Float64),
        )
        # Align column order to the Iceberg schema.
        df = df.select([f.name for f in RAW_SCHEMA])

        catalog = get_catalog()
        table = catalog.load_table(RAW_TABLE)
        df.write_iceberg(table, mode="overwrite")
        return df.height

    @task
    def build_analytics() -> int:
        catalog = get_catalog()
        raw = catalog.load_table(RAW_TABLE)
        raw_lf = pl.from_arrow(raw.scan().to_arrow()).lazy()

        summary = (
            raw_lf
            .with_columns(
                pl.col("started_at").dt.date().alias("ride_date"),
                ((pl.col("ended_at") - pl.col("started_at")).dt.total_seconds() / 60.0).alias(
                    "duration_min"
                ),
                _haversine_km(),
            )
            .group_by("ride_date", "rideable_type", "member_casual")
            .agg(
                pl.len().cast(pl.Int64).alias("ride_count"),
                pl.col("duration_min").mean().cast(pl.Float64).alias("avg_duration_min"),
                pl.col("duration_min").median().cast(pl.Float64).alias("median_duration_min"),
                pl.col("distance_km").mean().cast(pl.Float64).alias("avg_distance_km"),
            )
            .sort("ride_date", "rideable_type", "member_casual")
            .collect()
            .select([f.name for f in ANALYTICS_SCHEMA])
        )

        table = catalog.load_table(ANALYTICS_TABLE)
        summary.write_iceberg(table, mode="overwrite")
        return summary.height

    create_tables() >> load_raw() >> build_analytics()


cloudflare_iceberg_pipeline()

And some Airflow Python helper functions.

from __future__ import annotations

import os

import pyarrow as pa
from pyiceberg.catalog.rest import RestCatalog

NAMESPACE = "divvy"
RAW_TABLE = f"{NAMESPACE}.raw_trips"
ANALYTICS_TABLE = f"{NAMESPACE}.analytics_daily"


def get_catalog() -> RestCatalog:
    """Build a RestCatalog for the Cloudflare R2 Data Catalog from env vars."""
    return RestCatalog(
        "cf_r2",
        **{
            "uri": os.environ["CF_CATALOG_URI"],
            "warehouse": os.environ["CF_WAREHOUSE"],
            "token": os.environ["CF_CATALOG_TOKEN"],
            # Explicit R2 S3 FileIO creds for data-file access.
            "s3.endpoint": os.environ["R2_S3_ENDPOINT"],
            "s3.access-key-id": os.environ["R2_ACCESS_KEY_ID"],
            "s3.secret-access-key": os.environ["R2_SECRET_ACCESS_KEY"],
            # R2 wants "auto"; fall back to "us-east-1" if PyArrow FileIO rejects it.
            "s3.region": os.environ.get("R2_REGION", "auto"),
        },
    )


# --- Schemas (PyArrow -> PyIceberg accepts a pa.schema in create_table) -----

# Raw trips: mirrors the Divvy CSV columns, 1:1.
RAW_SCHEMA = pa.schema(
    [
        pa.field("ride_id", pa.string()),
        pa.field("rideable_type", pa.string()),
        pa.field("started_at", pa.timestamp("us")),
        pa.field("ended_at", pa.timestamp("us")),
        pa.field("start_station_name", pa.string()),
        pa.field("start_station_id", pa.string()),
        pa.field("end_station_name", pa.string()),
        pa.field("end_station_id", pa.string()),
        pa.field("start_lat", pa.float64()),
        pa.field("start_lng", pa.float64()),
        pa.field("end_lat", pa.float64()),
        pa.field("end_lng", pa.float64()),
        pa.field("member_casual", pa.string()),
    ]
)

# Daily analytics rollup.
ANALYTICS_SCHEMA = pa.schema(
    [
        pa.field("ride_date", pa.date32()),
        pa.field("rideable_type", pa.string()),
        pa.field("member_casual", pa.string()),
        pa.field("ride_count", pa.int64()),
        pa.field("avg_duration_min", pa.float64()),
        pa.field("median_duration_min", pa.float64()),
        pa.field("avg_distance_km", pa.float64()),
    ]
)


def ensure_namespace(catalog: RestCatalog, namespace: str = NAMESPACE) -> None:
    if (namespace,) not in catalog.list_namespaces():
        catalog.create_namespace(namespace)


def create_table_if_not_exists(catalog: RestCatalog, identifier: str, schema: pa.Schema):
    """Idempotently create a table; return the loaded Table."""
    if catalog.table_exists(identifier):
        return catalog.load_table(identifier)
    return catalog.create_table(identifier, schema=schema)

Mostly, we are just using the pyiceberg Python package to set up the connection to the R2 Data Catalog, along with the configs we need. After that, we can use Polars’ Iceberg write feature to get our data where we need it.

• Also, note when reading tables, if we want, we can use pyiceberg itself along with pyarrow to read; it’s just a way of showing you the interoperability of Arrow, how it shows up everywhere, and can be integrated into pretty much every tool under the sun.

Also, if we head over to the Cloudflare UI, we can see those tables appear, along with a nice overview. Not bad!

We can even examine the individual tables very closely and get all sorts of interesting data, including schema, partitioning, etc.

Don’t forget they have that new R2 SQL, which seems to be more or less a Notebook in the UI for querying our data. Cloudflare speaks highly of it, of course.

I tried finding the R2 SQL in the UI, but it was not available. Apparently, I was wrong about that, at least as far as I can tell. It seems you must submit queries through the Wrangler CLI we discussed at the beginning.

So, to do this, we need to install the Wrangler CLI. Of course, we/you would need an API token as well; they are easy enough to create or obtain in the UI.

I mean, personally, I’m not the biggest fan of this; The Notebook is the interactive standard of choice across the industry, making developers' lives easier. Not sure why Cloudflare hasn’t jumped on this train.

export WRANGLER_R2_SQL_AUTH_TOKEN=xxxxxxxxxx

Now we should be able to run an SQL query.

npx wrangler r2 sql query "3ee64e77beb1e2c68a3ae7c1cd4d232e_wonderful-lakehouse" "
SELECT
    *
FROM divvy.analytics_daily"

Well, it works; at least I can say that for all the Cloudflare Data Platform parts and pieces, they are incredibly simple, straightforward, and just work. That’s more than you can say for many other resources and tools.

• As always, the code is on GitHub for your use.

Share

Thoughts and musings.

I should return to Cloudflare with some sort of streaming use case; it appears that is their bread and butter, and admittedly, this has always been a hard nut to crack, that is, the combo of Lake House + Streaming.

Having recently played with Databricks Zerobus, it would be an interesting comparison.

• I have many great things to say about Cloudflare’s “Data Platform,” along with a few critical comments.

Cloudflare does a great job with their products; this Iceberg setup on R2 with the Data Catalog, even with R2 SQL via the CLI, is all just nice … and it just works. It’s simple.

You saw how simple the Airflow DAG code was, how easy the setup was, and how attractive this would be to ANY data team. One simply cannot underestimate the power of simple engineering. Also, the ease of integration with all Python tools and packages, including Arrow, makes it a flexible setup.

We start with Wes’ journey from building GoldenEye fan websites as a teenager to creating Pandas while working at a quantitative hedge fund, and eventually launching Apache Arrow, one of the foundational technologies behind today’s modern data ecosystem. Along the way, we discuss Cloudera, Parquet, DuckDB, DataFusion, Spark, and how the industry evolved from Hadoop to today’s lakehouse architectures.

The second half of the conversation dives deep into AI. Wes explains why large language models make experienced engineers more productive but won’t magically replace software engineering, why architecture and good taste are becoming more valuable than writing individual lines of code, and why projects like DuckDB and

• Apache Arrow remains incredibly difficult to recreate with AI alone. We also discuss open-source, local AI models, token costs, multimodal data platforms, and what new engineers should focus on to build long-term careers in software and data.

If you’re a data engineer, software engineer, architect, engineering leader, or simply interested in where AI is taking our industry, this is a conversation you won’t want to miss.

Topics We Cover

• How Pandas was created

• The story behind Apache Arrow

• Why Arrow became the standard for modern data systems

• DuckDB, DataFusion, and the next generation of data tools

• The evolution from Hadoop to lakehouses

• Why AI won’t replace great software engineers

• Architecture vs. coding in the AI era

• Building trustworthy open source software

• The future of data engineering

• Advice for new engineers entering the industry

If you enjoy conversations with the people building the future of data engineering, subscribe for more interviews with the creators of the tools we use every day.

This is a strange feeling for me. Something new and exciting from AWS besides another mass layoff? What’s wrong with those people? It’s almost like they care or something. Definitely sus. If you’re like me, you’ve spent decades under the big thumb of that not-so-benevolent AWS Cloud.

Our lives and work are consumed by that company; every twitch they make sends ripple effects down the long terminal halls of bits and bytes, eventually breaking some 5-year-old boto3 call in Python running on a decrepit EC2 instance via cron.

I’ve had to bear the brunt end of the sharp sticks they poke into the eyes of the software masses. IAM, Glue, Athena, dear sweet Jesus … EMR, Redshift, Fargate, Lambda, MWAA, the list simply never ends.

To be honest, I’d given up hope that anything useful or interesting would come out of AWS at this point. Then, lo, I heard the voice of one crying in the wilderness, “… prepare the way for AWS Lambda MicroVMs, make their path straight.” I said, here am I, send me.

Like a breath of fresh air on the face, a bubbling stream in the forest, a warm fire in the winter … this is what my poor heart felt when I first heard the whispers of AWS Lambda MicroVMs.

Thanks to Delta for sponsoring this newsletter! I use Delta Lake daily, and I believe it represents the future of Data Engineering. Content like this would not be possible without their support. Check out their website below.

A new era of compute?

Funny to think about the large swath of end-user products built upon those two behemoths of compute, EC2 and Lambda. I think it’s fair to say that if you locked yourself in an oven with your laptop and were given a day to design and write some SaaS thiny-ma-bob, and were only given EC2, Lambda, and pick-your-poison language, it could be done.

• Part of me wonders if the collective “meh” from the jowls of the software community at large at the announcement of AWS Lambda MicroVMs has to do with the satisfied, comfortable compute blanket we find ourselves wrapped in today.

Who needs something new when there's the reassurance of familiar things that work well enough?

These MicroVMs seem to sit inside the gap between AWS Lambda and EC2 instances. Does this broaden the market for use, meet pent-up demand, or simply fill a small niche?

I mean, the world as we know it is practically built on these two compute options from AWS: EC2 and Lambda, which are at opposite ends of the spectrum.

The EC2 Workhorse

EC2 instances were launched in ~2006, making them probably among the oldest and best-known AWS services. I think EC2 boasts somewhere in the neighborhood of 600 different types/sizes. Everything from EKS to EMR, from Fargate to Beanstalk, is underpinned by EC2.

Of course, EC2 might seem simple on the surface, but it actually can be complex when sifting through 600 different options. CPU, Memory, Storage, Networking, Compute Types, Pricing Options, Backups, etc, etc.

This sort of unlimited flexibility and customization is foundational to what makes EC2 compute so attractive; you can build your own puzzle and get exactly what you need.

The Lambda Workhorse

My personal favorite AWS compute option has always been Lambda. The simplicity, speed, and scalability of Lambdas unlocked new ways to architect systems of all kinds. From SFTP servers to data pipelines, from messaging systems to HTTP routing. Lambdas are just as flexible as EC2, but smaller, faster, and more lightweight.

There is just something so sweet and simple about a Lambda that triggers when you want, does a little work when you want, then disappears without you ever knowing.

Versatile, fast, flexible, and simple. Core tenets of good software engineering.

For *most people, Lambdas are incredibly cheap as well.

The empty space … the gap.

As much as I don’t like to admit it, AI- agentic software, some call it- appears to be a large reason why AWS MicroVMs came into being. At least, the marketing and documentation seem to indicate this.

I know for a fact that the inner sanctum of AWS has been thinking about this “gap” or “problem” in compute offerings for some time now. Mistakenly, in 2023, while bored on some lazy Sunday afternoon, I wrote about AWS Lambda, Python, and Rust.

It gained quite some traffic, and must have ended up on the desk of the power brokers inside AWS. A short time later, I found myself sitting digitally across from some folks on the AWS Lambda team, having my brain pulled apart about the best and worst pieces of Lambda.

On many fronts, AWS MicroVMs seem to answer these challenges of bridging the gap between Lambda and EC2. There are many use cases where you want the speed, simplicity, and scalability of a Lambda, but WITHOUT the time and memory limits, and where you need the flexibility and size of an EC2 instance.

Sure, you could just leave Kubernetes or an EC2 Cluster running in Fargate or somewhere else, but these are very heavy engineering solutions, complex, they take time and engergy away from solving the core issue, AND they force you into a certain architectural box when solving problems.

With all this as background, let’s explore AWS MicroVMs by putting our hands to the plow.

Not so gentle introduction to AWS MicroVMs

Ok, for you and me, this will be trial by fire; it is new for us and everyone else, so let’s start with concepts, then move on to more details, and finally to implementation. My goal is not to make anyone an expert, but rather to introduce this tool as a new pattern and architectural choice when designing software solutions.

We might not be putting MicroVMs directly into production, but we are learning and growing, expanding our horizons, and keeping abreast of the changes that are lofted upon us.

• concepts

• details

• implementation

AWS has introduced Lambda MicroVMs, a new serverless compute service designed for applications that need to securely execute code written by users or AI models. Think coding assistants, online IDEs, data science notebooks, SQL playgrounds, or SaaS applications that allow customers to run custom Python or JavaScript.

One of the big innovations here is that every user or job gets its own isolated virtual machine, but without requiring you to manage EC2 instances, containers, or Kubernetes. Again, this is Lambda-ish, obviously, and it's probably one of the biggest selling points.

Unlike traditional Lambda functions, which are stateless and short-lived, Lambda MicroVMs combine three capabilities that previously forced developers into trade-offs:

• VM-level isolation so one customer’s code can’t interfere with another’s.

• Near-instant startup using Firecracker MicroVM technology.

• Suspend and resume support, allowing a MicroVM to preserve its memory and state for up to 8 hours instead of starting from scratch every time.

AWS says this is built on the same Firecracker virtualization platform that already powers more than 15 trillion Lambda function invocations every month, but now developers can directly use that technology rather than having it hidden behind traditional Lambda.

• Deploying one is straightforward. You package your application as a Docker image, create a MicroVM image from it, and launch isolated MicroVM instances as needed. Each instance has its own dedicated HTTPS endpoint that supports HTTP/2, gRPC, and WebSockets, making it easy to build interactive applications.

This was a bit I found interesting, but it sorta makes sense in the light of the Agentic and AI use cases that AWS seems to be focusing on out of the gate. HTTP endpoints as an entry point for MicroVMs.

The service can be deployed using familiar AWS tools such as the Lambda Console, CloudFormation, CDK, or the Agent Toolkit for AWS, making it fit naturally into existing AWS workflows.

Pricing is also different from Lambda Functions. Instead of paying only per request and execution time, you pay for a baseline amount of compute while the MicroVM is running, with additional charges only if your workload temporarily exceeds those baseline resources. This reflects that a MicroVM is designed to remain alive and preserve state rather than disappear after each request.

Again, this seems to be a fundamental shift in the compute technology AWS offers. Time will tell if it can fight its way to anywhere near the top of the AWS compute stack; it’s hard to say- old habits (lambda + EC2) die hard. Either way, we have a new option when designing bespoke systems.

Previously, you generally had to choose between:

| Want...              | You used... | Problem                    |
| -------------------- | ----------- | -------------------------- |
| Fast startup         | Lambda      | Stateless, 15 minute limit |
| Long running process | ECS/Fargate | Infrastructure overhead    |
| Strong isolation     | EC2         | Slow and expensive         |
| Per-user sandboxes   | Kubernetes  | Operational nightmare      |

Why This Matters

This is less about replacing Lambda Functions and more about filling the gap between Lambda and containers.

Previously, developers had to choose between:

• Lambda offers speed and simplicity but sacrifices persistent state.

• ECS/Fargate for long-running workloads, but manage container infrastructure.

• EC2 for complete control and isolation, but at the expense of serverless benefits.

Lambda MicroVMs aim to combine the best of all three by offering isolated, persistent, serverless virtual machines with minimal operational overhead.

For AI applications in particular, it’s a compelling fit. Every AI agent, coding assistant, or customer session can have its own secure execution environment that starts almost instantly, stays warm while in use, and resumes exactly where it left off. That’s something traditional Lambda simply wasn’t designed to do.

A few MicroVM details to cover.

We should take a few moments to just to cover some technical details found in the MicroVMs AWS docs. Here are some of the most exciting stats: we are no longer bound by a 10 GB memory limit. I was surprised we can only get 32GB max, I mean, we will take it, hopefully some larger sizes come soon enough.

Also, to create a MicroVM Image, you provide a zip package that contains a Dockerfile and your application artifacts, which you upload to Amazon S3.

• Dockerfile

• code artifacts

• S3

Also, there is a concept of States; this will be an important part of MicroVMs and a change from Lambdas. We are now entering a world of Lambdas with State.

MicroVMS are HTTP machines.

This is probably the most telling and disappointing part of MicroVMs, at least from my point of view. When reading the docs, marketing, blogs, etc, it’s clear they are pushing the AI agentic narrative. Also, this could just be my disappointment with what AWS thinks is necessary.

• These MicroVMs are, in essence, long-running HTTP servers.

What I mean is that their functionality is wired to be HTTP-centric: communication happens over HTTP; there is no “trigger” logic like a normal Lambda.

1. Why is an HTTP server the entry point?

MicroVMs are not invoked like Lambda functions. No handler (event, context); no event payload delivered to your code. A MicroVM is a long-running VM exposed as an HTTPS endpoint; you talk to it over the network.

So the app must be an HTTP server listening on a port (8080). Two reasons:

- The trigger is an HTTP request (see part 2).

- The lifecycle hooks are themselves HTTP endpoints that Lambda calls during build — GET /aws/lambda-microvms/runtime/v1/ready, /validate. Even a pure batch job has to serve those paths, or the snapshot never completes.

An HTTP server is structural to the MicroVM model. That part is not a choice.

Of course, like any good engineer, we can twist and mold whatever we like to do whatever we want, even if it’s not made to do that.

Get 20% off for 1 year

MicroVMS in action.

The moment we’ve all been waiting for. You and I both, but I think all the time we spent getting to this point was just as important. Heck, I have no idea if I’m doing MicroVMs any justice with all my musings and hot takes about having a new compute tool to re-architect systems.

Also, if we want to build a tool workflow, an actual use case for MicroVMs that has some basis in the real world, we need to come up with that and build to those specs.

• Here is one.

## Automted Customer Data Dump Processing

A customer(s) periodicly dump large amounts of historical data in
CSV format. The business requires two things, 
1. process that incoming data
2. report immeditaly with meta data/stats about that data dump.
It's important to note that this is not only a data processing task,
but more importantly, it serves to notify the relevant business unit
all information pertaining to the data dump by said customer.

This is an interesting problem, is it not?

Yes, we have a classic data processing problem to solve, which is not novel in and of itself at all. The second part, classically, indicates that a second part, just as important if not more so, is that we need to treat this historical data dump from a customer as a logical unit of processing in totality.

Next, that leaves us, as Engineers, to decide the best way to solve this problem.

Let’s say we have about 81.GBs of data to process, for this example, we will use the Backblaze hard drive open-source dataset. We will say, for argument's sake, that this data gets dropped all at once, as part of a customer’s data dump. CSV files to the last.

So classically, how would we chomp on 80GBs of data at once, at the same time recognizing we need to somehow treat this entire process as, at the very least, a logical unit of work? Because at the end of the show, we need to report all the stats back to the business.

Of course, maybe we could get away with using Lambdas if we processed the CSVs one by one, but this is problematic because Lambdas work independently of one another, are short-lived, and we would have to use some complex logic and something like a database to track everything we are processing and maybe even process final stats.

• We want simplicity overall, first and foremost.

Also, it’s an option to spin up some EC2 on Fargate with enough spice to eat up this dataset with Spark, or maybe Polars or DuckDB in streaming mode. This also seems overkill, no?

Adding the overhead of some ECS cluster and figuring out the spin-up and spin-down of the large instance just seems unnecessary if we can find another way.

Let’s use a MicroVM to do this work in one fell swoop. The simplicity of a Lambdas in terms of architectual overhead and management, the size and capability of an EC2, no lambda small memory or time limits.

Subscribe now

Brass tacks.

Again, we need a few things to actually build a MicroVM. First, a Dockerfile, our codebase, zip it, send it to S3.

• Dockerfile

• code as zip

• send to S3

Of course, you can go into the AWS Console, under Lambdas, and find all the clickety-click buttons to create a MicroVM, as per below.

Below you will find all the code to create, update, trigger, and terminate a MicoVM. Here are the general steps that were coded to do all this. I wrapped everything, all these steps, into Bash scripts.

## Deploy — real AWS Lambda MicroVM

Prereqs: an S3 bucket for the code artifact, and an IAM **build role** (see `iam/`).

```bash
export ARTIFACT_BUCKET=my-artifact-bucket
export BUILD_ROLE_ARN=arn:aws:iam::123456789012:role/MicrovmBuildRole
export OUTPUT_BUCKET=my-output-bucket          # optional; defaults to ARTIFACT_BUCKET


aws iam create-role --role-name MicrovmBuildRole \
  --assume-role-policy-document file://iam/build-role-trust.json
aws iam put-role-policy --role-name MicrovmBuildRole --policy-name build \
  --policy-document "$(envsubst < iam/build-role-permissions.json)"

# 3. Build image → run MicroVM → trigger a dump run → tear down.
bash scripts/build_image.sh     # zips, uploads, create-microvm-image, polls to CREATED
bash scripts/run_microvm.sh     # run-microvm, polls to RUNNING, saves id/endpoint to .microvm
bash scripts/trigger.sh         # auth token + POST /process against the real dump prefix
bash scripts/teardown.sh        # terminate-microvm

bash scripts/update_image.sh    # (later) ship new app code as a new image version

These bash scripts, available on GitHub, just wrap the AWS CLI commands to work with AWS Lambda MicroVMs. I would say this is very similar to AWS Lambda, but differs in a few aspects.

• We still can use a Dockerfile in conjunction with our application code.

• We must build the image, deploy the code and Dockerfile into S3.

• This CREATEs a MicroVM.

• Now we must actually get the MicroVM into the RUNNING state.

• After, it is RUNNING we can TRIGGER it with a call to its HTTP endpoint.

This is a little more than I bargained for, to be honest. There are a lot of steps and a lot of code wrapped in those Bash files. Ah well, we will see what comes next, eh? I will crack just a few of those files for you here, so you can get an idea, and figure if you want more you will head to the GitHub repo and try it yourself.

Let’s do this with code first. Starting with a few things, we need some IAM roles, etc. This is all available on GitHub.

aws iam create-role --profile confessions --role-name MicrovmBuildRole \
  --assume-role-policy-document file://iam/build-role-trust.json

aws iam put-role-policy --profile confessions --role-name MicrovmBuildRole --policy-name build \
  --policy-document file://iam/build-role-permissions.json

Set some ENV Vars …

export ARTIFACT_BUCKET=confessions-of-a-data-guy
export BUILD_ROLE_ARN=arn:aws:iam::992921014520:role/MicrovmBuildRole
export OUTPUT_BUCKET=confessions-of-a-data-guy

The next big piece of work in the MicroVM Dockerfile, critical and the core of making a new MicroVM on AWS. Also, make sure if you are using the AWS CLI commands like me, that you update your CLI to the latest version so the lambda-microvm commands are available.

Here is the Dockerfile itself. Super simple. We have to use FastAPI as a web server to handle the HTTP requests; more on this later.

FROM public.ecr.aws/lambda/microvms:al2023-minimal

WORKDIR /app

RUN dnf install -y python3.11 python3.11-pip && dnf clean all
COPY requirements.txt .
RUN python3.11 -m pip install --no-cache-dir -r requirements.txt

COPY app/ ./app/
COPY sample_data/ ./sample_data/

EXPOSE 8080

CMD ["python3.11", "-m", "uvicorn", "app.server:app", "--host", "0.0.0.0", "--port", "8080"]

Then we have to build and push the image with our code wrapped up in a zip to S3.

#!/usr/bin/env bash
# Package the app, upload to S3, and create the MicroVM image (poll until CREATED).
set -euo pipefail
source "$(dirname "${BASH_SOURCE[0]}")/config.sh"

cd "$ROOT_DIR"

echo ">> Packaging app.zip"
rm -f app.zip
zip -r app.zip app Dockerfile requirements.txt sample_data >/dev/null

echo ">> Uploading to s3://$ARTIFACT_BUCKET/$ARTIFACT_KEY"
aws s3 cp app.zip "s3://$ARTIFACT_BUCKET/$ARTIFACT_KEY" --profile "$AWS_CLI_PROFILE" --region "$AWS_REGION"

echo ">> Creating MicroVM image: $IMAGE_NAME"
aws_mvm create-microvm-image \
  --name "$IMAGE_NAME" \
  --code-artifact "uri=s3://$ARTIFACT_BUCKET/$ARTIFACT_KEY" \
  --base-image-arn "$BASE_IMAGE_ARN" \
  --build-role-arn "$BUILD_ROLE_ARN" \
  --environment-variables "OUTPUT_BUCKET=$OUTPUT_BUCKET,OUTPUT_PREFIX=$OUTPUT_PREFIX,DATA_SOURCE=s3"

echo ">> Waiting for image to reach CREATED (Ctrl-C to stop polling)"
while true; do
  state=$(aws_mvm get-microvm-image --image-identifier "$IMAGE_ARN" \
    --query 'state' --output text)
  echo "   state=$state"
  case "$state" in
    CREATED) echo ">> Image ready."; break ;;
    CREATE_FAILED) echo "!! Build failed. Check CloudWatch: /aws/lambda-microvms/$IMAGE_NAME"; exit 1 ;;
  esac
  sleep 10
done

Then, if we do our job right, we have a Lambda MicroVM Image that shows up in the console or on the CLI. This means step one is done. We have “CREATED” our MicroVM, and it’s ready for next steps (RUNNING, then TRIGGER).

Get 20% off for 1 year

See the below image.

Next, we have a bash script to RUN the MicroVM ... aka get it into RUNNING state. Think of it like starting an EC2 instance. I guess we could, in theory, have something triggering this bash script. Maybe an Airflow DAG at the end of each day looking for a new data dump.

set -euo pipefail
source "$(dirname "${BASH_SOURCE[0]}")/config.sh"

echo ">> Running MicroVM from image: $IMAGE_NAME"
run_json=$(aws_mvm run-microvm \
  --image-identifier "$IMAGE_ARN" \
  --ingress-network-connectors "$INGRESS_CONNECTOR" \
  --egress-network-connectors "$EGRESS_CONNECTOR" \
  --idle-policy '{"autoResumeEnabled":true,"maxIdleDurationSeconds":900,"suspendedDurationSeconds":300}')

MICROVM_ID=$(echo "$run_json" | python3 -c 'import sys,json;print(json.load(sys.stdin)["microvmId"])')
ENDPOINT=$(echo "$run_json" | python3 -c 'import sys,json;print(json.load(sys.stdin)["endpoint"])')

echo ">> microvmId=$MICROVM_ID"
echo ">> endpoint=$ENDPOINT"

while true; do
  state=$(aws_mvm get-microvm --microvm-identifier "$MICROVM_ID" \
    --query 'state' --output text)
  echo "   state=$state"
  [ "$state" = "RUNNING" ] && break
  sleep 5
done

printf 'MICROVM_ID=%s\nENDPOINT=%s\n' "$MICROVM_ID" "$ENDPOINT" > "$ROOT_DIR/.microvm"
echo ">> Running. Saved id/endpoint to .microvm — now run scripts/trigger.sh"

And TRIGGER it. Just because the MicroVM is running doesn’t mean it is doing anything. We would have to have another process, in this case my bash script, or maybe a Lambda Function, or Airflow Call, to the MicroVM endpoint to TRIGGER our code to do a thing.

Whatever that thing is that our code does.

set -euo pipefail
source "$(dirname "${BASH_SOURCE[0]}")/config.sh"

[ -f "$ROOT_DIR/.microvm" ] || { echo "!! run scripts/run_microvm.sh first"; exit 1; }
source "$ROOT_DIR/.microvm"

PREFIX="${1:-hard_dive_dump/2026-07-02/}"

echo ">> Creating auth token for $MICROVM_ID"
token=$(aws_mvm create-microvm-auth-token \
  --microvm-identifier "$MICROVM_ID" \
  --expiration-in-minutes 30 \
  --allowed-ports '[{"allPorts":{}}]' \
  --query 'authToken' --output text)

echo ">> POST https://$ENDPOINT/process  (prefix=$PREFIX)"
curl -sS "https://$ENDPOINT/process" \
  -H "X-aws-proxy-auth: $token" \
  -H "Content-Type: application/json" \
  -d "{\"bucket\":\"confessions-of-a-data-guy\",\"prefix\":\"$PREFIX\"}" | python3 -m json.tool

Remember, your logs available in CloudWatch will only be as good as your app; log often and at important points so you know how your application is doing; otherwise, you will have to jump inside the MicroVM, just like the good ol’ EC2 days.

• Ok, this is where I got super annoyed. Yes, my MicroVM runs fine, code and all, but, I have something to say about it all.

I take back what I said.

Ok, no takebacks? Well, I take it all back. I’m no longer excited about MicroVMs; there is nothing Micro or lightweight about them in my opinion. The amount of code and triggers and states and this and that, it’s more flipping complicated than an EC2 box that I can spin up and down much more simply.

Classic.

I’ve been foiled.

So much flipping code. Go browse the below if you want.

I mean, why am I not showing you the end result? Because this is crazy is wh’s what it is. Look at the GitHub repo, all the flipping files and backflips.

Who the crap wants to run data pipelines inside a FastAPI app? All behind an HTTP port? Give me a break.

It seems to me that MicroVMs have some very cool features, but the use cases are a little narrow. This isn’t something that the masses will probably use, at least not in its current form. You really need to have some strange use case, a long-running, but not that long-running (8-hour max) task that needs the benefit of being suspended and restarted? Sure, maybe some AI workflows.

• I get they try to sell you on the VM isolation, big deal, whatever.

Yes, once you build a MicroVM image that states it, running it is fast. I found that out by playing around. Sure, it’s nice to have Lambda-style super-fast starts with bigger memory and size limits. But that’s just it. It’s cool but utterly useless.

The whole idea of doing EVERYTHING via HTTP is really dumb, especially since the darn thing runs for 8 hours max and then shuts off. So I need to build an app that handles HTTP routing and communication but is short-lived to do some “Data Science” stuff they were talking about.

I think someone at AWS is smoking something good.

MicroVMs combine the worst things about Lambdas and EC2 instances into a single product.

Yeah, maybe I will regret this hot take; maybe I don’t know what I’m doing, and I missed something completely. That’s totally possible. Leave me a comment.

Leave a comment

The worst part of AWS Lambda MicroVMs is that they took away ALL the best parts of Lambdas.

• low complexity, concepts, and code requirements

• various helpful trigger and firing mechanisms.

If EC2 and Lambda were in high school, made love behind the football stadium, and had a baby out of wedlock, AWS Lambda MicroVMs are that baby. Everyone wants to love it, but it’s a real pain in the !@#$@!.

Sure, maybe I’m just mad that I got it wrong from the start and was excited about something that turned out to be not what I thought. Yeah, that’s probably most of it. But honestly, I just fail to see where this weirdo fits into *most architectures.

Hey you! %20 off paid subscription.

Yeah, I’m looking at you, you little stinker. You made it to the end of the article, eh? Well, good for you. Hey, do you realize I put myself through hell every week, hacking on junk like this, all for you?

• Just so you have something to scroll on your phone when you’re bored at work?

Get 20% off for 1 year

Throw a guy a bone or two, you give Amazon your money to drop crap on your doorstep. I love you more than Amazon does.

What? You want to know what you get for your money. Well, you can sleep well at night knowing you’re not a free loader. Also, you get plenty of Member-Only content, access to my Discord channel, where you can listen to all my Podcasts right after recording, and early access to articles like this.

In this episode of the Data Engineering Central Podcast, I sit down with Dave Langer to explore nearly three decades of experience across software engineering, business intelligence, analytics, data science, and AI.

Dave’s career spans COBOL programming on IBM mainframes, enterprise architecture, Microsoft’s Xbox division, machine learning, startup leadership, authorship, and building one of the largest personal brands in the data space.

We discuss why many of the biggest problems in data haven’t changed, even as the tools continue to evolve. We dive into the reality behind self-service analytics, the importance of dimensional modeling, what organizations are getting wrong about AI adoption, and why developing strong analytical skills matters more than ever.

• Dave also shares practical advice for data professionals navigating the AI era, explaining why tools like Copilot should be viewed as partners rather than replacements.

If you’re a data analyst, BI developer, data scientist, or data engineer wondering what the future holds, this conversation offers both perspective and optimism.

What We Cover

• How Dave got started programming on a Commodore 64

• The transition from COBOL to modern analytics

• Why the core problems in data haven’t changed

• The evolution of business intelligence and dashboards

• How Dave discovered machine learning

• Why data science needs more than just Jupyter notebooks

• The limitations of self-service analytics

• Why semantic layers and governance matter for AI

• Advice for staying relevant as AI reshapes the industry

• Building a personal brand in data

• Writing a technical book and becoming an independent creator

Connect with Dave:

• LinkedIn: https://www.linkedin.com/in/davelanger/

• Substack: The DIY Data Scientist

The DIY Data Scientist

Learn real-world, do-it-yourself (DIY) analytics skills that make you stand out at work with weekly hands-on tutorials designed for ANY professional, including Excel workbooks, data and code.

By David Langer

• Book: Python and Excel Step-by-Step

Subscribe for more conversations on data engineering, analytics, AI, and building a career in modern data.

It's a race to the bottom, as per usual, inside the inner tech sanctum where legions of software grifters burn tokens in dark terminals, sending up a sweet-smelling aroma to the Sardaukar licking up the costly sacrifice of the faithful.

• Did I perchance capture your attention?

On a more serious note, this is not another doom-and-gloom AI slop crybaby post. Look, there goes a flock of wawa’s. Claudius Maximus reigns supreme either way, I'm not one to kick against the goad. I'm just here to report what I see on the front lines of the battle into which we, techies that is, were hurled without our consent. Again, and forever more.

I'm neither gloomy, or ecstatic about the current unraveling of the ground on which we've built out careers, lives, and the truths we once held dear.

I talk and interact with, on my Podcast and in real life …

• the Rust, bespoke, and unspoiled hand that has yet to touch an LLM.

• The Junior Dev who lost their way in this Mad Max agentic nightmare.

• The ever-confident C-suit executive who bounds along with endless optimism at the possibilities.

• The confused Product Manager is unsure of what the next move is.

• The once priceless backend contractor is scrambling to save itself.

• AI-enamored founders who will never look back.

• Trust voices who hate and are skeptical of AI “things”

• Trusted voices who love and embrace AI “things”

• And myself, tossed from utter terror and despair to determination in a matter of hours.

And, the best part is, here we all are, living in the same world, breathing the same air, working in the same data community.

Sheer Terror and Apathetic Acceptance.

It’s probably good that all of us overly ripened Developers who’ve been living off the fat of the land for these last few decades are shaken away from our luxurious slumber. Big paychecks, jobs galore, special knowledge, undaunted careers.

Things are changing. Must to the happiness of our ancestral pre-cursors who are looking down on us in disgust from the clouds, we’ve collecting happened onto harder times, more difficult times, challenging times.

While they had to make a bow out of wood, a point out of rock, kill something, make a fire, and cook it, just to survive … or write some network driver in Fortran or Perl … we found ourselves asking Claude to stop making mistakes.

No wonder they are disappointed in us, and now watch with glee while we squirm, waiting for that email to tell us our lover has knifed us in the back and now is doing our job, and we are “redundant.”

My LinkedIn feed is filled with both ends of the spectrum.

• Disillusioned masses were laid off and desperate, in total despair.

• Pure thrill and excitement at the possibilities of the future

We live in an upside-down world for sure. Watching the bell curve of despair and elation in my network for the last year, I’m ready to make some feeble predictions about what your next move should be, and what kind of world we are now living in.

Maybe I should just draw you some pictures. (without AI … *gasp)

I would agree that AI is a game-changer, but then again, so were a lot of other technical things that came about over the last few decades, maybe not on the same scale, but arguably easier to adopt. So, why, pray tell, are there still many a COBOL mainframe running around with aged programmers hacking away?

• Why is the internet run on PHP? Why are there still a massive number of SQL Server and Oracle data warehouses cranking away? Why are most business analysts still using Excel? Why C and C++ with Rust around? Why Python for data when it’s slow and dumb?

I would argue that because humans are humans, even with AI still around. They are cautious by nature; only a brave few will drive forward without question. The majority follow the crowd closely, peering around corners, more willing to talk about doing a thing than to do it.

Maybe it’s some hidden DNA sequence where our jumpy forebears tried not get eaten by a lion or something.

Subscribe now

Thanks to Cube for sponsoring this Newsletter! The best way to support me and this Newsletter is to check out our Sponsor below and click the link.

Cube on Databricks: The AI-Native BI Platform for Your Lakehouse - One Semantic Layer for BI, Embedded Analytics, and AI on Databricks

Cube is the AI-native BI platform built on a universal semantic layer. Semantic layer, query engine, visualizations, embedded analytics SDKs, and an AI Analyst, all built on a single set of metric definitions. Point it at any warehouse or lakehouse, and you get one consistent analytics experience across dashboards, customer-facing apps, and AI/LLM use cases.

Register for the Webinar!

Different cultures, different takes, different abilities.

For the most part, people operate in the bubbles and circles in which they have spun the web of their lives. Sure, there a few who’ve moved between the 12 person startup and 120,000 person corporate giant, or the family owned business in the Midwest.

• These are not the same people

• These don’t have the same talent

• The cultures are different

Some Engineering groups don’t handwrite or review a single line of code anymore; they proudly Gastown their way to whatever result they want. Design reviews, code generation, PRs and reviews, testing, release, bug fixes.

Others work in places where they have a locked-down version of CoPilot on a 2-year-old model. AND, they are in a business context where the people at the top think that is the cat's meow, cutting-edge stuff.

• And of course, we have everything in between, from all-in to all-out.

How many of you have been in tech long enough to be familiar with how long it actually takes to roll out some massive shift in a technology stack, and how successful those rollouts are?

Let’s be real.

There is some distance between Anthropic telling us no one will write code, and that reality becoming fact throughout the world. Umm … like … probably longer than your lifetime.

Share

That said, beat em’ at their own game.

Either way, no one knows the hour. You are probably wondering what’s next for you, what your next move should be. I think I have a few thoughts I would like to throw in the ring.

What the possible and very specific chinks in the LLM armor might be, and how one can, in good measure, differentiate oneself to survive the unknown cataclysm that appears to be bursting above our heads in biblical torrents.

Here it goes, take it for what it’s worth, just my take.

1. Use AI to streamline and increase your productivity.

   1. AI can help you do your job better, so use it. Duh.

2. Read the (AI) room, at work, match + 20% whatever they want.

3. Work hard on your soft (human) skills.

4. Simply work harder (I didn’t say longer) than most people.

   1. How to be Better Than Everyone Else

5. Continue to learn new things monthly.

   1. Becoming a “better” Data Engineer

6. Re-teach yourself the fundamentals periodically.

   1. Your Limiting Factor (in Software)

7. Learn to plan projects and lead people.

8. Spent time on systems design and architecture.

   1. Designing Data-Intensive Applications

9. Improve everything you touch.

10. Have a positive can-do attitude.

11. Learn to have maniacal focus, finish what you start.

    1. 

12. Read a lot of books and blogs.

    1. Staff Engineer by Will Larson

    2. The Pragmatic Programmer

13. Spend time outside, be healthy, and sleep well.

You will notice I didn’t write a lot about AI, in the age of AI, for becoming 10X better than the 100X Claude Engineer. The fallacy that Claude Engineer falls into is that somehow they are different than everyone else who has the exact same tool. This is highly unlikely.

AI is the Great Equalizer.

To make yourself employable, long-term and short-term, it’s important to be good at what you do, AI or no, creating technical solutions, but the best of the best have always been more than simple code pushers.

• People who care about their craft leave things better than they found them, with positive attitudes, team-player qualities, and a willingness to lead and teach others. Life-long learners who are not afraid of new tools and lead by example.

Trust me, that 100X Claude Engineer around the corner is obsessed with their prompts and the next billion-dollar SaaS idea they are going to code out this weekend.

They are not thinking about …

• communicating with non-Engineering groups

• being able to bring others together to finish a project

• focusing on finishing one or two tasks at a time

• thinking about the bigger picture

• delivering value to the business

• what code not to write

• how to reduce complexity

• … etc

The best engineers are thinking about those things AND using AI to their advantage to help them do those tasks better and faster, where it makes sense. To them, AI is just a tool to enable more productive work and better outcomes, pretty much like every other new technology they’ve come across.

I am both a believer and unbeliever of AI and software at the same time.

You can be more than one thing at the same time; you don’t have to join others in the camps they find themselves in, begging you to join them, na, trying to convince you against your will to join them. Just ride the thin red line.

It’s neither helpful to hate AI nor to throw everything out the window in favor of AI. Be smart, be nuanced in your approach. I still exercise my thoughts and brain, I use AI to 10X those 100X Claude Engineers by still spending most of my time with Claude to think through problems, steelmanning designs, examining all possibilities, planning projects and work, and learning new things.

Leave a comment

Subscribe now

We discuss why semantic layers have suddenly become one of the most important concepts in modern data platforms, how tools like Claude Code are transforming engineering workflows, and why the core problems in data haven’t really changed despite massive advances in technology.

David shares his perspective on where agentic workflows are headed, what AI means for junior engineers entering the field, and why experienced practitioners may be more valuable than ever before. We also dive into the evolution of data platforms, lessons learned from startups, the promise of tools like DuckDB and MotherDuck, and how organizations should think about adopting AI responsibly.

Whether you’re a data engineer, analytics engineer, engineering leader, or someone trying to understand where the industry is headed, this conversation offers a practical and honest look at what’s coming next.

What We Cover

• David’s journey from analyst to startup founder

• The rise of semantic layers and why they matter

• Why data modeling is still critical in the AI era

• How AI coding agents are changing engineering work

• What Claude Code is enabling today

• The future of agentic data pipelines

• Why DuckDB and MotherDuck are gaining traction

• The challenges facing junior engineers

• Career advice for data professionals at every stage

• Whether David is optimistic about the future of AI and data

Connect with David:

• LinkedIn: https://www.linkedin.com/in/david-jayatillake/

• Substack:

  davidj.substack

  A substack about my journey in data: building data teams/stacks for org impact

  By David Jayatillake ✝️

As I was winding my way through the various twists and turns of LinkedIn lately, strange it has become the least of the social media evils, by happenstance there popped a post about Comet, that Datafusion baby that’s been trying to take a corner out of Spark for a few years now.

I did poke a stick at Comet back in 2024, but it was a little painful. You can read about that experience blew if you’re interested.

One would have to assume with the increase in Comet chatter, that the project has gotten its sea legs under it and hopefully is a slightly better experience this time. I’m crossing my fingers at this point that I don’t have to build my own JAR this time.

I hope one can simply download the correct pre-built JAR for the Spark version we are interested in. That would be a step in the right direction.

Truth is I have do no “pre-work” when I sit down to write articles like this and explore or revisit tools. Endevoring to simply approach the problem space like the average engineer who interact with some new “thing” for the first time. I want to encounter that same experience and relay that to you. Save you some trouble.

I do not have particularly high hopes from what I might encounter on this foray into Mordor, based on past experience. Let me explain what I mean by this. There are two types of Engineers and tooling they build in the world …

1. Smart engineers building cool tools but lack critical 
developer-centric awareness.


2. Smart engineers building cools tools and understand 
developer-centric are the key to success.

This has been a problem as long as I’ve been (attempting) to write code for the last 20 years. To this day, it simply doesn’t change. Some tool and their ardent acolytes say “Look, we are faster, come unto us.” Yet, they never seem to crack the nut and become mainstream, and can’t figure out why.

All the while continuing to waste their resources on negativity, pulling down other people and tools in an effort to lift themselves into a place by pulling other things down. This never has worked, and never will, longterm, in any sphere in life.

I’ve been privy to see behind the curtains, meeting people at Databricks and MotherDuck (DuckDB), who are creating and running the creme-de-la-creme of data tooling. What makes them succesful?

• The have the fastest tool … NO

• The focus their energies on negativity and others .. NO

They simply have an almost irresistible set of qualities that is impossible to ignore or hid … or be pulled down in the mud by others.

• Positivity in all things

• Love what they do and other people

• Put Developer experience BEFORE everything else

• Serve and give back to the data community

So simple, but yet so hard for some folk to grasp. Why? I don’t know. It’s probably something inside them, unhappy people will always be bumbling along looking for people to suck down into their misery.

Let’s start what I’ve been avoiding

Ok, at some point we have to get to what we are after today, namely figuring out if Apache Datafusion Comet has improved itself over the few years since we last visited it. One thing is for sure. In the Year of Lord 2026 if someone is using Spark, they better be using Databricks.

I know there are still a few odd old-school curmudgeons banging around in the dark corners, BUT if you want to be taken seriously in the Spark world today, you must go where the users are. Databricks. Sorry. Not Sorry.

• Can we get Comet working on Databricks?

First, we need a JAR.

The GitHub page for Comet gives a semi-not-so-much clear instructions.

The installation guide gives us more info, including what Spark Versions we are allowed to use with Comet.

So, we will have to pick a DBR version that matches something in here. Strangely, they say “Published jar files are only available for released versions, ” which insinuates they have published, pre-built JARs for supported Spark versions, although there is no link or mention of where to find those JARs … one could assume Maven maybe?

• You would think linking and being obvious about where pre-built JARs are located for potential dev “customers” might be a fairly obvious thing to do. Or not.

Ok, let’s go find some JARs (hopefully), before falling back to building our own. Indeed, they have a Maven repo with the different Spark version Jars.

We should be able to line this up with a Databricks DBR version that fits our needs eh.

Let’s pick Spark 4.0.

So now we’ve got our comet-common-spark4.0_2.13-0.16.0.jar that we can align with DBR 17.3 LTS. Next, we just need to gather the required Spark configs needed to go along with this JAR, we can unwind what we need from the Comet examples given.

Based on what I can see, this is what we will need to add to our Databricks Cluster config.

--jars $COMET_JAR \
    --conf spark.driver.extraClassPath=$COMET_JAR \
    --conf spark.executor.extraClassPath=$COMET_JAR \
    --conf spark.plugins=org.apache.spark.CometPlugin \
    --conf spark.shuffle.manager=org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager \
    --conf spark.comet.explainFallback.enabled=true \

As well as possibly

--driver-class-path spark/target/comet-spark-spark4.1_2.13-0.17.0-SNAPSHOT.jar

Also, if you want your brain to go numb, you can read through the Comet Configs located here, it’s the next page that comes up in the Installation Instructions.

I’m assuming some of these are important, and greatly assist with how Comet performs, but God knows how one is simply supposed to scroll through all 1000 of them and know what to do. Your guess is as good as mine.

• Might be nice if they put some sort of overview of the the most important ones, things that should not be left to default, or the most commonly used ones and put some sort of conceptual overview together for how to tune them.

To top it all off, there is a very large and complex “Compatibility” section detailing what Comet can, and cannot do, where it will fall back to Spark etc. This one will give you a headache as well.

If you meet someone behind the old oak tree at midnight, throw salt over your left shoulder, and say these words … “Ooga Booga” 3 times over. At that point you will know if the production Spark pipeline you have will benefit from Comet and how to tune the configs for optimal performance.

Tear* … let’s do this thing.

Ok, let’s switch over to Databricks and get our cluster setup. First, let’s get this JAR available somewhere Databricks clusters can access it. Probably a Volume would be the easiest way to do this.

Here I have made a volume Volumes/confessions/default/jars and put our JAR up in there.

Next, let’s make a little init.sh script that can run on any cluster.

#!/bin/bash
COMET_JAR="/Volumes/confessions/default/jars/comet-common-spark4.0_2.13-0.16.0.jar"
LOCAL_JAR="/databricks/jars/comet-common-spark4.0_2.13-0.16.0.jar"

cp "$COMET_JAR" "$LOCAL_JAR"
chmod 644 "$LOCAL_JAR"

Next, we can put that bash file into the same Databricks volume.

Ok, so let’s just use one of those dreaded All-Purpose clusters in Databricks, configure it with the init script, as well as the rest of our configs.

As well, we can add in that init script we made.

And finally, our Spark configs for Comet.

I have zero idea if this is going to work, just a shot in the dark. We will find out soon enough, maybe. Ok, next we need some data in s3 to munge around with. Let’s use the open-source Backblaze harddrive dataset.

We can get all the 2025 data and use that. This is 365 files for about 43GB of CSV data in total. Not big data, but then again, most Spark pipelines running today picking up CSV files at not munching on Big Data.

This will do for now. Let’s write some average Spark code that would represent the average Spark pipeline in production.

Read more

Well, it’s that time of year again. My feed has been full of a little bit of this, little bit of that from the Databricks Data and AI Summit 2026. It’s kinda hard to miss, although this year, I must say, it seemed a little on the quiet side as compared to others.

Who knows, 2026 has been a busy year in the world in all ya know, wars, rumors of wars, layoffs, and generally it’s either the beginning of a new world or the end of our current one.

Like any good hunter of technology, I keep my veteran ear to the ground, ruffling through the fluff and litter, trying to find out what is actually worth your time, and what is just another layer on the AI cake we are all too tired to choke down.

So I’m just going to give you my take on the important announcements and products released or announced at this years Databricks Data and AI Summit 2026. I mean I wasn’t there, so take me with a grain of salt.

Here’s my list, and I will then follow it up with my take on each one.

• Zach Wilson went after blowing a raspberry at them last year.

• Lakehouse//RT aka “Reyden”

• LTAP - OLAP + OLTP on a single copy of data in the lake

Yeah, I’m giving a big ho-hum to the plethora of AI shiny rocks released, like Genie Ontology, Genie ZeroOps, Omnigent, Unity AI Gateway, etc. The world of Agents and AI is still in flux, no clear winners, everyone is throwing stuff at the wall to see what sticks.

For my part, I’m a big believer in Databricks, they have a propensity to fundamentally change technology and how we use it. Truly ground breaking technical solutions and products they bring to market. If you can’t admit that, then check yourself. That being said, I’m just reviewing what I consider products or announcements that meet that critera. Ground breaking and game changing.

If you think I’m full of it about something, or I missed something big, drop a comment.

Zach showed up.

Hey, I like a little spice in my life, the hot stuff keeps everyone on their toes. I regularly get on the wrong side of powerful people, much to my joy. Nothing like free head space to bring me more followers.

Anywho, last year Zach was throwing rocks at Databricks, as you can see below. Good Lord you gotta love seeing behind the curtain sometimes. We need our own data Netflix series on this sorta thing ya know?

But, Zach attended this years Summit, which I was glad to see. I think Zach is one of the smartest and hard working engineers I’ve ever scene, and I love Databricks because they make the best products. I’m glad they are getting along now.

Anywho, enough on that, next.

Subscribe now

Lakehouse//RT: Databricks Brings Real-Time Analytics to the Lakehouse

This one caught my poor little ears as soon as the words dripped out onto LinkedIn. Again, all this stuff is new so who knows what the future holds or if anything will come of it, but it solves some major pain points we’ve been dealing with in a new and novel way.

• I’m excited for this one.

One of the biggest challenges in modern data architecture has been serving real-time applications from a Data Lake or Lake House. When we adopted technologies like Delta Lake and Apache Iceberg, aka file storage with ACID, we gained some stuff and lost others. Well lost isn’t a good word. Delta Lake + Spark ain’t no Postgres + Python ya know??

Traditional LakeHouses excel at ETL, analytics, machine learning, and business intelligence, but when teams need dashboards that refresh in milliseconds or applications serving thousands of users simultaneously, they often introduce an entirely separate serving database such as ClickHouse, Pinot, Druid, or Redis.

That second system brings another copy of the data, another synchronization pipeline, another set of security policies, and another operational burden.

It brings complexity and overhead. These technologies just had a difficult time jiving.

Lakehouse//RT is Databricks’ attempt to eliminate that architecture altogether. Instead of exporting data into a specialized serving database, Lakehouse//RT delivers millisecond query performance directly against Delta Lake while keeping the data inside the governed Lakehouse.

What is Lakehouse//RT?

Lakehouse//RT is a new real-time compute designed specifically for workloads that require both very low latency and extremely high concurrency. It is powered by a new execution engine called Reyden, which Databricks says was built from the ground up for operational analytics, application serving, observability, dashboards, and AI agents.

I’m still unsure if we are dealing with two things, or one … read this closely …

• Powered by Reyden, a “new engine for realtime workloads”

• Lakehouse/RT, “real time data warehouse”

Are these two separate things you can use, or you use them both at the same time? I don’t know. Time will tell the details.

Unlike traditional analytical warehouses that optimize long-running reports, Reyden focuses on thousands of simultaneous interactive queries while maintaining consistent response times. According to Databricks, preview customers have seen:

• Up to 16× faster performance than existing real-time serving layers

• Query latencies as low as 10 milliseconds

• Sub-100 ms performance on much larger datasets

• Around 12,000 queries per second while maintaining low latency

Eliminating the Serving Layer

The biggest architectural shift isn’t simply that queries are faster, we’ve been hearing people say “My thing is faster,” for a decade or more. The difference is that Databricks wants to remove an entire layer of infrastructure.

Today’s architecture often looks like this:

Delta Lake
      │
   ETL / CDC
      │
ClickHouse / Pinot / Druid / Redis
      │
 Applications & Dashboards

With Lakehouse//RT, Databricks wants applications to query the Lakehouse directly:

Delta Lake
      │
Lakehouse//RT
      │
 Applications
 Dashboards
 AI Agents

That means:

• no duplicate storage

• no synchronization pipelines

• no additional serving clusters

• no proprietary storage formats

• no duplicated governance

Everything continues to use Delta Lake and Unity Catalog. Some will argue that this isn’t new technology, and that Databricks sees the growth of ClickHouse for example, and decided they needed to do something about that.

I agree, but I also think that Databricks providing this sort of extremely fast Compute option is ground breaking inside the Data Platform they provide. You simply CANNOT discount the reduction in complexity and code when you

Built for Operational Analytics

Databricks positions Lakehouse//RT for workloads that have traditionally been difficult to run directly from a data lake:

• customer-facing SaaS applications

• operational dashboards

• observability platforms

• security analytics

• embedded analytics

• AI agent retrieval

• interactive business intelligence

These are all scenarios where thousands of users—or AI agents—may issue queries simultaneously and expect responses in tens of milliseconds rather than seconds. It takes something tradtionally done outside Databricks, or at the minimum with third party tools, and brings it back inside the MotherShip.

All the chickens coming home to roost so to speak.

Simpler Operations

Lakehouse//RT also introduces a different compute model. Instead of selecting warehouse sizes manually, Databricks automatically determines the appropriate baseline compute. Rather than scaling by duplicating entire warehouse clusters, it incrementally adds or removes nodes as concurrency changes, aiming to improve utilization while reducing costs.

Governance Doesn’t Change

One of the more compelling aspects of Lakehouse//RT is that governance remains centralized. Since the data never leaves the Lakehouse:

• Unity Catalog permissions stay intact

• security policies are defined once

• business logic isn’t duplicated

• data lineage remains consistent

Organizations no longer need to recreate governance rules inside a separate serving database. Governance is indeed the next big topic in the world of data and AI. Security breaches every work, Claude 100X Engineers releasing buggy code left and right.

Can’t be too careful these days.

How It Fits Into Databricks’ Bigger Picture

Lakehouse//RT makes even more sense when viewed alongside Databricks’ other announcements this year.

• Lakebase provides a PostgreSQL-compatible operational database.

• LTAP unifies transactional and analytical storage on a single copy of data.

• Lakehouse//RT provides millisecond analytical serving directly from that same data.

Together, Databricks is attempting to collapse what has historically been three separate systems:

• OLTP databases

• analytical warehouses

• real-time serving databases

into a single platform built around Delta Lake, Unity Catalog, and specialized execution engines.

My Take on RT

This feels like one of the most strategically important announcements from this year’s Data + AI Summit. The performance numbers are certainly impressive, but the bigger story is architectural simplification.

For decades, data teams have accepted that customer-facing applications require a separate serving database. Lakehouse//RT challenges that assumption by making the lakehouse itself fast enough to serve those workloads. Will it catch? I don’t know. We will all find out in a year I guess.

The remaining question is whether those benchmark results translate to the wide variety of real-world production environments that rely on ClickHouse, Pinot, Druid, Elasticsearch, and similar systems today. If they do, Lakehouse//RT could remove an entire category of infrastructure from many modern data platforms.

Share

Lake Transactional/Analytical Processing Architecture

I’ve been waiting for this one since I was knee high to a grasshopper. Jezz, even Forbes is writing about this one, what the heck do they know about anything??

This is the one thing I wasn’t ready for, although it makes total sense. It was LTAP, short for Lake Transactional/Analytical Processing, and if you spend your days building data platforms instead of making keynote slides, this is the announcement that deserves your attention.

The elevator pitch sounds almost suspiciously simple. Databricks wants operational databases and analytical workloads to operate on the same copy of data, eliminating CDC pipelines, ETL jobs, replicas, synchronization processes, and the collection of brittle plumbing that has somehow become accepted as “modern data architecture.”

After reading both the press release and Ali Ghodsi’s interview explaining the thinking behind it, I don’t think this is really a story about ETL at all. It’s a story about removing a 40 year old architectural assumption that everyone stopped questioning years ago.

Today's architecture

Application
     │
 PostgreSQL
     │
 CDC / ETL
     │
 Data Lake
     │
 Analytics

For decades we’ve accepted that applications belong in one database while analytics belong somewhere else, usually connected together by a growing pile of Kafka topics, replication jobs, Airflow DAGs, managed CDC services, and a Slack channel dedicated entirely to asking why yesterday’s pipeline failed again.

The industry has spent years trying to make this architecture less painful instead of asking whether the architecture itself is the problem.

Databricks thinks it is.

The obvious comparison is HTAP, which promised to unify transactional and analytical workloads years ago. The problem was that HTAP largely tried to shove both workloads into the same engine, which meant eventually your dashboard and your checkout page were competing for the same resources.

Databricks is taking a different approach. Instead of building one engine that tries to do everything, they’re building one storage layer that multiple specialized engines can operate against.

LTAP

          Delta / Iceberg
                │
   ┌────────────┼────────────┐
   │            │            │
Lakebase   Lakehouse   Lakehouse//RT
  OLTP        OLAP        Real Time

That distinction is important because LTAP isn’t replacing PostgreSQL with Spark, and it isn’t asking Spark to become an operational database. It’s saying the engines can stay specialized while the storage becomes unified. That is a much bigger architectural shift than “we removed ETL.”

• The other piece that clicked for me came from Ghodsi’s explanation of why they’re doing this now. His argument is that the real pressure isn’t coming from human developers anymore. It’s coming from AI agents. Humans create applications relatively slowly. Agents don’t.

They create databases, clone environments, test ideas, throw them away, and repeat the process constantly. Databricks claims roughly 80 percent of the databases on Lakebase are already being created by agents rather than people.

Whether that number surprises you or makes you instinctively reach for a fact check, the direction is hard to argue with. Infrastructure built around dozens of data copies and endless synchronization simply doesn’t scale when your primary users become software instead of humans.

The real takeaway here isn’t that Databricks found a clever way to remove another Airflow DAG. They’re trying to remove entire categories of infrastructure. No operational database replica feeding analytics. No warehouse copy that’s always fifteen minutes behind production. No “Zero ETL” marketing that quietly hides another synchronization service behind the curtain.

• One copy of the data, multiple engines reading and writing it, and governance living in one place.

Will it work? That’s the billion dollar question, or perhaps the hundred billion dollar question given Databricks’ valuation. Lakebase still has to prove it can be a production operational database at massive scale, and LTAP has to demonstrate that this elegant architecture survives contact with messy enterprise workloads.

But if Databricks pulls it off, we may eventually look back at maintaining separate OLTP and OLAP systems the same way we now look at nightly FTP jobs and hand-written shell scripts. Necessary once, painful always, and eventually replaced by something that made us wonder why we tolerated the old way for so long.