If you’re job hunting right now, you need to understand the differences between startups, small businesses, and larger corporations. They all have their plus and their negatives, and picking the wrong one can waste months of your search.

Startups: Versatile, But Risky Right Now

Startups are very versatile work-wise, and that is good. Small teams mean every day is different, and you actually get a chance to move something, which you rarely get in a bigger company. Salary can be good, can be bad, depending on the startup, and stock options are the same story: can be good, can be bad.

The negative: startups are looking for specialists, especially now. They are usually not for beginners, because people need to be ready for the job from day one, there’s no time to slowly ramp you up. That’s not good for job security, and smaller teams aren’t great for career development either, simply because there aren’t that many jobs available inside the company, so there’s nowhere internally to grow into.

My honest opinion: with startups, most likely you will be in trouble right now.

Small Business: Startup-Like, But Steadier

For data jobs especially, small businesses feel very startup-like. You get higher job security than at a startup, which is a real plus, but salary-wise these are usually not so good, so you’re trading money for stability.

The resources to grow usually go to the main business, and the small business doesn’t get much of that, so don’t expect a big training budget or fancy tools. But you still get the chance to move something, and the work is versatile, just like at a startup, so you get some of the startup upside without quite as much of the risk. That’s the trade-off.

Larger Corporations: My Actual Tip

Everybody wants to work at FAANG, but there are plenty of other corporate jobs out there too.

The good: corporations usually have entry level jobs even now, because they can actually afford it. You get a lot of on-the-job training, because again, they can afford it, and there are lots of resources available for the job itself. Job security is better, although you know how it is right now. Even the larger companies are in trouble.

Career development is usually better too, with real on-the-job training, and you can move to another job pretty easily because the company is so big. There’s another opening somewhere else within a three hundred thousand people company, right? Salary is also very often very good, because they can afford it, and they want the best people, so they pay for it.

The negative here is really big, and you need to understand this: well-defined platforms with tools and pipelines, Facebook for instance. You can’t move a lot here. You are inside a platform that is already done, so your chance of moving around is limited. On top of that, corporate structure and processes, bureaucracy and politics, is super strong here.

So Where Should You Apply?

Keep all of that in mind, because it’s a difficult call. But my tip: corporations. That is the one thing I would apply for right now, and also medium businesses.

What’s your experience: startup, small business, or corporate? Leave a comment.

***

Ready to become a Data Engineer? Then join my Learn Data Engineering Academy today!

Looking for free hands-on projects and tutorials? Then check out my Free Labs.

More Data Engineering content on: YouTube | LinkedIn | Spotify

Most courses teach you one tool and leave it there. That’s not how real engineering work goes.

We put six free labs on Learn Data Engineering, covering Docker, Python, Kestra, Exasol, DuckDB, and Databricks, and no single one teaches you everything you need. But go through all six, and the skills start connecting in ways that actually matter on a real project.

Docker: The One You Can’t Avoid

Docker is one of those skills every engineer needs sooner or later, even if it’s just for a quick job that has nothing to do with a big architecture.

In the lab you learn to deploy tools using pre-built images, but also how to build your own custom images that run Python code so you can process data anywhere, not just on your laptop. You even publish your own image to a registry so other people can pull and use it.

That last part is the moment it clicks: you’re not just using Docker, you’re distributing your own work with it.

👉 Try the free Docker Lab

Python: The Language That Shows Up Everywhere Else

Python is the connective tissue between almost every other lab. You work with data files, transform data, call APIs, talk to databases, and on top of that you implement logging, validation, and tests, the stuff that separates a script from something you can actually trust in production.

And here’s the thing: everything you build in Python can be packaged straight into the Docker images from the lab before. Whether it’s an AWS Lambda function, a Spark job on Databricks, or a quick exploration script, it’s Python underneath, and Kestra can schedule all of it.

👉 Try the free Python Lab

Kestra: Making the Chaos Visible

Orchestration is needed everywhere, and without it engineers are basically lost. Kestra doesn’t process your data itself, it manages what happens and makes problems visible when something breaks, which is a distinction a lot of people miss.

In the lab you observe and maintain pipelines, connect to Google Cloud services like BigQuery to build a medallion architecture, and set up event-driven pipelines with message queues that react the moment something happens instead of running on a fixed schedule.

This lab is a good example of how orchestrator tools, no matter which one you pick, sit on top of the actual processing engines, and it shows you how to do that well.

👉 Try the free Kestra Lab

Exasol: What Scaling Actually Feels Like

Just focusing on platform-as-a-service tools is lazy. At some point you need to understand how a self-hosted database actually works, and Exasol forces that. You deploy a horizontally scalable analytics database on your own cloud, and you get hands-on with partition and distribution keys, the stuff that decides whether your cluster performs or falls over.

You also scale the cluster yourself and feel the difference, and you run Python processing directly inside the database, callable from plain SQL. Even ML and LLMs run inside it now, and the simplicity of that is what stands out: a database being more than just read and write operations is one of the more interesting things you can experience in this lab.

👉 Try the free Exasol Lab

DuckDB & MotherDuck: Killing the Pandas Habit

DuckDB gives you a super simple OLAP store, and it’s great to learn precisely because it’s so lightweight, you set it up with a single Python import. More and more people are dropping files into a quick DuckDB instance and transforming them with SQL instead of writing pages of pandas code, and once you try it, it’s obvious why.

MotherDuck takes it further as a lighter alternative to platforms like Snowflake, and combined with DuckDB you can run one query that spans local and cloud data at the same time, with the workload split between the two. Best of both worlds, and it’s easier than it sounds.

👉 Try the free DuckDB & MotherDuck Lab

Databricks: Where It All Comes Together

Databricks is THE data platform right now, still growing fast, and the free tier makes it accessible to everyone. The lab shows you how to build ETL and streaming pipelines entirely in SQL through declarative pipelines, and how to use the Lakeflow designer to build pipelines with drag and drop and AI assistance instead of writing everything by hand.

You also implement streaming with AWS Kinesis so data gets processed the moment it lands, and you use Genie to just ask questions about your data in plain English.

It’s a great addition to normal PySpark work, not a replacement, and that’s an important distinction. A lot of people think Databricks is only for building a star schema or a medallion architecture, but most of the real use cases in this lab don’t even touch that.

👉 Try the free Databricks Lab

It’s Never Really About One Tool

That’s the actual point of building it this way. In a real job you rarely touch just one of these tools, you combine them depending on what the project needs.

A typical setup: you write your transformation logic in Python, package it into a Docker image so it runs the same way locally and in the cloud, and let Kestra schedule and monitor the whole thing so you actually know when something breaks.

Where that pipeline lands changes with the use case. If you need serious horizontal scale on infrastructure you control, Exasol is where the Python-in-database features and clustering skills from the lab pay off.

If you just need to explore a file quickly or hand a colleague a lightweight analytics layer, DuckDB and MotherDuck cover that without spinning up anything heavy.

And if the whole thing is a proper streaming pipeline into a lakehouse with Kinesis feeding it in real time, that’s Databricks territory, running on the same Python skills but now inside PySpark and declarative pipelines instead of a plain script.

None of these labs exist in isolation, and neither does real engineering work. All six are available for free in our Free Labs section.

Which one would you start with first?

If you have any career questions let me know. I’m happy to help!

Andreas

***

Ready to become a Data Engineer? Then join my Learn Data Engineering Academy today!

Looking for free hands-on projects and tutorials? Then check out my Free Labs.

More Data Engineering content on: YouTube | LinkedIn | Spotify

I look at a lot of GitHub profiles through my coaching students, and most of them make the same mistake. A pile of repos, no descriptions, no context, and then that link gets attached to a CV and nobody’s impressed. Here’s how to actually fix it, step by step.

Step 1: Build your profile README

GitHub lets you create a special repo, the same name as your username, and whatever you put in its README.md shows up right at the top of your profile page.

Most people don’t even know this exists. Create it, add a README.md, and use it to introduce yourself, what you do, what you’re learning, and what you’re working on right now. You can use Claude or ChatGPT to help you write the first draft, but make it sound like you, not like a corporate bio.

A short intro, a few lines about your focus area, and maybe a small list of your current projects is enough. Don’t overthink this one, it just needs to exist and look intentional.

Step 2: Pick which projects actually deserve a spot

Before you touch a single README, look at your repo list and be honest. Not every project needs to be public, and not every project needs a portfolio treatment. Pick two or three that actually show real skills, a full pipeline, a project with some complexity, something that took real decisions to build. A tutorial you followed once doesn’t count. Quality beats quantity here every time.

Step 3: Write a real README for each project

This is the step almost everyone skips, and it’s the one that matters most. If someone opens your project and can’t tell what it does in ten seconds, you’ve already lost them. I’ve used the same structure with coaching students for years, and it works.

Start with a title and a short intro, what the project is and what your goal was building it. Add a screenshot of the architecture, even a simple one drawn in draw.io is fine. Then talk about the dataset, what it actually is, how much data you’re working with, and how much it could realistically be in production.

Mention any constraints you had, because you can’t just say you spent thousands of dollars on cloud infrastructure for a portfolio project, and being honest about your limits actually reads as more credible, not less.

List the tools you used, and then break the technical side into categories. I always use Connect for data integration and ingestion, Buffer for anything with message queues, Processing for the transformation logic, Storage, Visualization, and don’t forget Orchestration and Observability if you have them.

Show off your actual pipelines, stream processing, batch processing, whatever you built, and add screenshots or even a short demo GIF if you can.

Step 4: Add the part people never think to include

End with what you learned and what broke along the way. This is the part that actually builds trust, because it shows you understand your own project instead of just copying a tutorial. A short TLDR conclusion at the end helps too, some people will only read the top and the bottom.

Step 5: Use a template so you’re not starting from zero

I put a full portfolio README template in my free data engineering cookbook, the exact one I use with my coaching students. You don’t need to build this structure from scratch every time, just drop the template into your project, fill it in, and you’re done.

Doing this for even two or three projects makes a real difference for anyone who lands on your GitHub. They understand what you built, how you think, and what you actually learned, instead of just scrolling past a list of repos that mean nothing to them.

Is your GitHub actually telling your story right now, or is it just sitting there?

***

Ready to become a Data Engineer? Then join my Learn Data Engineering Academy today!

Looking for free hands-on projects and tutorials? Then check out my Free Labs.

More Data Engineering content on: YouTube | LinkedIn | Spotify

AI writes most of the code now. I still think you need to understand what’s happening under it, and that comes down to seven Python topics. Not exotic ones. The boring stuff that shows up in every real pipeline.

Exception handling and logging

Things break. APIs time out, files are missing, a column that used to be an integer suddenly isn’t. You need to catch that, and you need to log it somewhere, a file, a database, CloudWatch, whatever you’re using.

Without this, a pipeline just fails silently at 3am and you find out from an angry Slack message the next morning.

And it’s not just about catching the error, it’s about catching the right error.

Wrapping an entire script in one giant try/except that swallows everything is a common shortcut, and it’s a bad one, because now the pipeline looks “fine” while it’s quietly skipping half the data. That’s worse than no error handling at all, because now you trust a number that’s wrong.

Catch specific exceptions, log what actually happened, and let the things you didn’t expect actually blow up so you notice them.

Datetime

Timestamps are everywhere, and they cause more bugs than people expect. Timezones, daylight saving, string formats that look identical but parse completely differently.

I’ve seen a pipeline break because one system logged UTC and another logged local time, and nobody noticed until a report was off by an hour.

Get comfortable with datetime early, it saves you a debugging session later.

The annoying part is that these bugs almost never show up in testing. Everything works fine for months, and then daylight saving flips or a partner system switches formats, and suddenly a join that used to match perfectly is off by an hour and nobody can figure out why.

Store everything in UTC internally and only convert to local time at the very last step, right before it’s shown to a human. That one habit alone prevents most of these headaches.

JSON and JSON validation

Every API you touch speaks JSON. You send it, you receive it, and a lot of the data lying around in data lakes is JSON too.

The part people skip is validation. You get a payload, you assume the structure is what you expect, and then one day a field is missing and your whole job crashes downstream.

Tools like Pydantic will validate the shape of your JSON before it causes damage further down the pipeline.

Think about it this way, an upstream team changes an API without telling you, a field that used to always be there is suddenly optional, and your pipeline has no idea until it explodes three transformations later.

Validating at the point where data enters your system means you find out immediately, with a clear error, instead of hunting through five steps of transformations trying to figure out where a null crept in.

Unit testing

Whenever you write a transformation, you need to know if it actually does what you think it does. That’s unit testing.

The cool thing is, AI is genuinely good at writing these for you now, give it your function and it’ll spit out reasonable test cases in seconds. But you still need to understand what a good test actually checks, otherwise you’re just approving whatever the AI wrote without knowing if it’s testing the right thing.

I’ll be honest, this is the one that gets skipped the most, even by people who know better. It feels like it slows you down when you’re trying to ship something. But the one time a transformation silently drops rows on an edge case nobody thought of, a test would’ve caught it before it ever touched production.

It’s cheaper to write the test now than to explain to your manager why last week’s numbers were wrong.

Pandas

The big one. You need to work with dataframes, understand how to merge them, how to pivot them, how to group and aggregate.

I still see people trying to loop through rows manually instead of using vectorized operations, and it’s painfully slow on anything beyond a few thousand rows.

Pandas is one of those libraries where learning it properly once saves you from writing bad code for years.

Actually nowadays a lot of people also use DuckDB in python instead for quick transformations. Take the dataframe, write it into a duckdb database. DuckDB is basically just an imported library (If you don’t add a storage location then everything happens in memory.), do the transformation with SQL and then export the data again.

The requests library

Sending a request to an API sounds trivial until you’re doing it in production. Retries, timeouts, rate limits, handling a 429 response gracefully instead of just crashing.

The requests library is simple to start with, but there’s a real difference between a script that works once on your laptop and one that survives an API having a bad day.

That difference is basically invisible until the day it matters. Your script works perfectly in every test you run, and then it goes into production, the API has a rough afternoon, and your job either crashes completely or hammers the API with retries and gets you rate limited or blocked.

Adding exponential backoff and proper timeout handling takes maybe ten extra lines of code. It’s a small investment that separates a script from an actual production job.

Working with databases

Even a simple database setup, tables, SQL, and especially bulk loading. This is the one people underestimate the most.

Inserting rows one at a time works fine for a demo and falls apart completely once you’re loading real volumes of data. Bulk loading, whether that’s a COPY command in Postgres or a bulk insert method, is often the difference between a job that finishes in seconds and one that times out.

A pipeline that works great on a sample of a thousand rows can go live on the real dataset and just never finish, and by then it’s too late to fix it quietly. Nobody tells you this in school, but the way you load data matters just as much as the SQL you write to query it afterward.

Why this still matters in the AI era

I know the argument, AI writes the code, so why learn the fundamentals.

Here’s the thing: when the AI-generated code breaks, and it will, you’re the one who has to read it, understand it, and fix it. You can’t debug what you don’t understand.

These seven topics are what let you actually own the code AI hands you, instead of just hoping it works.

In my Python for Data Engineers course, we’re covering all seven of these hands-on. It is available in my Free Labs Bundle until end of August.

In the bundle you’ll also find a Databricks course, Kestra for orchestration, DuckDB, Docker fundamentals, and more! Get the bundle here.

Which of these seven do you think most people skip first?

***

Ready to become a Data Engineer? Then join my Learn Data Engineering Academy today!

Looking for free hands-on projects and tutorials? Then check out my Free Labs.

More Data Engineering content on: YouTube | LinkedIn | Spotify

I’ve been thinking about how much engineering work has actually changed in the last few years. Development, debugging, optimization, all of it. So I sat down and went through it piece by piece, asking what we still do ourselves and what has quietly moved over to AI.

And honestly, the list got uncomfortable to write.

Development Is Mostly Gone Now

AI writes code for all of us now. Not tab completion, not autocomplete suggestions here and there, that’s not even worth mentioning anymore. Actual code, whole features, entire files.

You might call this AI slop, but it is what it is. People even use AI for code reviews now..

Copilot is the wrong name for what’s actually happening. A copilot helps you fly, but you’re still the one flying. With coding, the AI is flying and we’re just telling it where to go.

So Github Copilot should just be called Pilot, and Claude Code should probably be called Claude Codes For You.

And it’s not one tool you can opt out of. Every platform builds this in now. Databricks has their AI doing development. I built a course for Kestra, and they have it too. There’s no way around it, and writing code just isn’t the big thing it used to be.

The Annoying Parts Went First

Documentation used to be the thing nobody had time for. We knew we should write it, we knew it mattered when someone new joined, but there was always something more urgent, so it never happened. Now it takes minutes and it’s actually decent.

Tests are the same story. AI writes them for us.

Code guidelines and style guides too, and this one is underrated. That used to be a real source of friction, people arguing about formatting and conventions, or a style guide sitting in a wiki that nobody followed. Now you just tell the AI to follow the guidelines, or create a skill and it follows them. All of that struggle is basically gone.

Data Modeling Too, and People Hate When I Say This

I’ve posted about this a bunch of times and people always read it wrong. So let me be very clear: I am not saying you don’t need data modeling. You still need data modeling. What I’m saying is that it’s not the big deal people make it out to be anymore.

AI is genuinely good at data modeling because it’s a structured problem and it’s seen a huge amount of it during training. Give it the context and it produces a solid model without much trouble.

What’s left for you is catching its mistakes. The AI is going to make mistakes, sometimes really silly ones. You need to understand what it creates well enough to catch when it makes a mistake. That’s a real skill and I’m not dismissing it. But it’s a very different thing from what people are still teaching across ten courses and a stack of books.

I mean, you can even have a conversation with the AI about the data model to see where there might be optimization potential.

Debugging Is Copy, Paste, Hope

How do you actually debug today? You hit an error, you copy it into Claude Code with a bit of context, and you see what comes back. Then you hope the solution is right. You try it. If it works, great. If not, you go around again, tell it that wasn’t it, here’s the new error, and try to figure it out together.

There isn’t a lot of deep thinking in that loop, and that’s not really a complaint. It’s just how it works now.

Root cause analysis holds up better on its own. That used to genuinely require experience, knowing how to follow the trail back through a system to where the problem actually started. Now you do it properly once, write those steps down in a markdown or skill file, and hand that to the AI as context.

Hell, you can even have the AI write the markdown file for you. After that, junior or senior, anyone can follow the same path. The experience gap on that task mostly closes.

Optimization Barely Needs Experience Anymore

This one surprised me the most. It used to be the clearest dividing line between people who’d been doing this a while and people who hadn’t.

Reading a SQL query plan took real knowledge, knowing what the operators meant and where the cost was hiding. Now you copy the query plan into Claude and it tells you what it sees and where the problem is. You don’t need that special knowledge or that special experience anymore.

Spark is the same story. You used to sit there analyzing the execution plan, trying to find the bottleneck and figure out how to implement a fix. Now you hand the plan over, maybe answer a couple of questions about your setup, and you get a useful answer back quickly. A huge amount of optimization work can be done this way.

So What’s Actually Left for Engineers?

I know I exaggerated a bit here and there in the article. Not everything works that perfectly... yet 😬

The important part now is knowing what needs to happen, not doing it yourself. Figuring out the context. Figuring out what people need, which is usually not what they think they do. Then giving the AI the important pieces and being able to understand and validate what it hands back.

The doing part is mostly AI’s job now. We keep telling it what to do, checking what comes back, and moving on to the next thing.

One More Thing About the AI Engineering Hype

Everywhere you look it’s LLMs, RAG, MCP servers, agents. It’s presented as if everyone needs this knowledge right now or they’ll be left behind, like becoming an AI engineer is the only path forward.

But think about your actual job, and the jobs of people you know. How much of that do you genuinely need today? We’re still at the beginning of this. Yes, tools are implementing MCP servers. But do you actually have an agent calling that MCP server? Most likely not. A lot of companies don’t have that skill yet, they don’t have that knowledge internally, and they’re nowhere near it.

It might be very interesting in the future. For the next year though, I’m not sure. This is still new for companies, and I’d be careful about treating it as urgent just because it’s loud.

***

Ready to become a Data Engineer? Then join my Learn Data Engineering Academy today!

Looking for free hands-on projects and tutorials? Then check out my Free Labs.

More Data Engineering content on: YouTube | LinkedIn | Spotify

Lately I keep noticing the same thing: every new feature is really about letting you do more while understanding less.

I’ve built courses on Spark, Databricks, on declarative pipelines, on Lake Flow Designer. I know these tools well, and I know plenty of people using them every day.

Now we have drag-and-drop pipelines. Declarative everything. A lot of it is stuff we used to configure by hand years ago, just wrapped in a nicer interface.

Why Everyone Does This

I don’t think it’s malicious. Databricks, Snowflake, all of them want as many users as possible, and the way you get there is by removing the parts that require thinking. Bring Spark to the masses. So let’s put 15 layers of abstraction on top so nobody has to understand what’s underneath.

And honestly, there’s a good side to it. You can bring in people who don’t have deep Spark skills yet and they can still get work done. That’s real. But there’s a cost to making everything easy that nobody puts on the slide.

Easy Makes People Sloppy

When everything is one click, you stop caring. You stop asking whether you should clean up this data or whether you even need this much compute. You just throw a couple more nodes at it because you don’t care.

A data scientist needs an all-purpose cluster to train a model, so they get a terabyte of RAM, and two weeks later they’re still using that same cluster to run select top 10 on a table. Notebooks running production code.

Thousands of dollars a day burning quietly in the background. How did we get here? One easy decision at a time.

The Bill Is an Engineering Culture Problem

People love to complain that Databricks and Snowflake are expensive, and they are. But a lot of that cost isn’t the vendor. It’s the culture that the abstraction encourages.

When you can’t see what’s happening underneath, you stop watching your compute, you stop asking if DuckDB would do the job, you stop treating spend like it’s your own money. Staying on top of that stuff is real engineering work, and the easier the platform makes it to ignore, the more it costs you to ignore it.

The Other Direction Still Exists and I love it

This is why I found a recent project interesting. We’re soon releasing a free course around Exasol, which is much more traditional OLAP store. You set up the infrastructure yourself, on AWS or on premises, you decide how many machines you want, and it’s fast because it uses in-memory with disk underneath and handles indexing for you.

It took me back to when not everything was a black box. You actually have to make decisions and understand why.

Why this distribution key. Why this layout.

On top of that, plenty of companies literally cannot use the big cloud platforms because of regulation, so they need to own their infrastructure and see their own ACLs, their inbound and outbound rules, the whole thing. That world isn’t going away, no matter how many people tell you it’s the old way.

Are you making it too easy for yourself?

Are you actually watching what your data platform costs and why, or are you trusting the abstraction to handle it for you?

And the bigger question behind that: are you starting to trust the drag-and-drop features so much that you’ve stopped thinking for yourself?

Because that’s what actually matters. Almost anyone can click things together.

Understanding why it’s built that way, what it costs, and what happens when it breaks, that’s the part you’re getting paid for.

***

Ready to become a Data Engineer? Then join my Learn Data Engineering Academy today!

Looking for free hands-on projects and tutorials? Then check out my Free Labs.

More Data Engineering content: YouTube | LinkedIn | Spotify

I found a real before-and-after data platform architecture on the dbt Labs website, and instead of just describing it, I want to walk you through how I actually figured out what happened. This is NOT sponsored ;) I just find this kind of stuff interesting.

The Case I Picked

dbt Labs publishes customer stories on their site, and one of them caught my eye. A company had databases scattered across five different teams, transformations happening in more than one place at once, and queries slow enough that people actually complained about it, and they rebuilt everything into a setup that cut costs by 70% and ran queries 30 times faster.

The problem is, these pages never show you the actual architecture. The decisions that actually mattered, why one tool got replaced and not another, how the pieces actually connect to each other, are left out on purpose. You only get that picture back if you already know the tools well enough to fill in the gaps yourself, which is exactly what I did here, going through it piece by piece like a detective looking at a case file.

If you prefer watching such architecture breakdowns, check out my YouTube video on this use case:

Reading the “Before” Picture

This company sells and installs solar panels and heat pumps, so their data platform pulls together sales data from a CRM, acquisition data from a second CRM, and operational numbers from energy and supply chain systems, all feeding into one shared reporting layer. That’s the context worth keeping in mind before looking at how messy the setup underneath it actually was.

The first thing I did was look at where all of that data actually landed once it left those source systems, and that’s where the mess started. Five separate databases, each one owned by a different team, each one doing its own transformations. On top of that, someone had added Azure Data Factory to pull data from the cloud side, which added another layer of ETL that duplicated a lot of the work already happening elsewhere.

This is a pattern I’ve seen a hundred times. Every team solves their own problem well, but nobody owns the whole picture.

Working Out What Actually Broke

The dbt Labs page listed four symptoms: scattered transformations, slow performance, low data velocity, and data quality issues. On their own, these are just buzzwords, so I went through each one and asked what would actually cause it.

Slow performance and low data velocity both come down to the same root cause. You can’t cleanly join five databases without a shared key, and here there wasn’t one. Every team built their own schema for their own purpose, so pulling one report that needed both sales and supply chain data meant somebody had to manually reconcile IDs across systems that were never designed to talk to each other.

The permissions issue makes sense too, once you think about who actually owns what. Five teams, five databases, and each team only has access to their own slice. That’s fine until someone outside that team needs the data for a report, and then they’re stuck waiting on a request instead of just querying it themselves. That’s not a technology problem, that’s an ownership problem that shows up three teams downstream as a slow dashboard.

The data quality issues follow from the same setup. When raw source data changes and five independent transformation layers are all reading from it separately, a single upstream change can quietly break more than one pipeline, and there’s no single place that catches it before it reaches a report.

Reconstructing the Fix

The after architecture was much easier to read, because it’s actually simple. Fivetran for ingestion, Airflow for scheduling, and everything landing in Snowflake as a single source of truth. From there, dbt runs the transformations, 1,400 models split into raw, cleaned, and core reporting layers. That’s a medallion architecture, even if they didn’t call it that on the page.

The interesting part is Stitch. It shows up as a legacy tool still running in parallel, and my best guess is Fivetran didn’t have a connector for one of their older systems, so Stitch is patching that gap. I could be wrong here, dbt Labs didn’t spell it out, but that’s the kind of detail you only catch if you know what these tools can and can’t do.

The other piece worth calling out is what happens once the pipeline is actually running. They layered a governance and data catalog tool on top of Snowflake for observability, which is only realistic because everything sits in one place now, on five separate databases you’d need five separate integrations just to see what’s going on. And because both Fivetran and dbt Cloud support CI/CD, changes to the pipeline get tested and deployed the same way you’d deploy any other piece of software, instead of someone manually running SQL scripts and hoping nothing broke downstream. Tableau ends up with a single clean connection instead of five, which on its own probably explains a good chunk of that 30x query speed.

The Numbers That Actually Matter

70% cost cut and 30x faster queries get all the attention, but the number I think is more important is 3 months. That’s how long the migration took, and the reason it was possible is that they built the new system in parallel instead of trying to migrate the old one piece by piece. You run both, you validate the new one, and then you switch off the old infrastructure in one move. It’s not a technical trick, it’s a project management decision, and it’s the difference between a migration that drags on for a year and one that actually ships.

Why I Do This

This isn’t a dbt article, and dbt Labs didn’t ask me to make it. I do this because reverse engineering real architecture decisions teaches you more than reading a generic best practices article ever will. The company left gaps on purpose, and filling those gaps with reasoning is the actual skill of being a data engineer.

If you’re sitting on a version of this five-database mess right now, I’d genuinely like to know what’s keeping you from consolidating it. Team politics, tooling, or just nobody owning the decision?

***

Ready to become a Data Engineer? Then join my Learn Data Engineering Academy today!

Looking for free hands-on projects and tutorials? Then check out my Free Labs.

More Data Engineering content: YouTube | LinkedIn | Spotify

I was talking to Dan Beach from Data Engineering Central recently who uses Cursor every single day. Years of experience, ships real stuff to production, fully bought into AI. And he told me that on a Saturday he sometimes sits on his couch feeling a little depressed, wishing he could just write some Rust by himself again.

That stuck with me, because I think a lot of us feel it and don’t say it out loud.

The Whiplash Nobody Admits To

There’s this constant back and forth right now. One day AI is incredible and you can’t imagine working without it. The next day you’re wondering if your skills are quietly fading away because the model wrote everything and you just clicked accept.

The honest version is that both things are true at once. You need to be technical to be good at your job, because you’re still the one responsible for what ships. But you also start to miss the part where you actually built the thing yourself, in the flow, music on, hours disappearing. That feeling doesn’t go away just because a tool got faster.

What Actually Gets Lost

For me, the love was never really about typing the code. It was about creating the solution, understanding the problem, making the right decisions. That part is still here, and AI hasn’t taken it.

But Dan put it so well. He said something has been lost inside him, like a piece of him. And I think what he’s pointing at is the creativity of doing something yourself, struggling with it, breaking it, and figuring it out. When the AI does all of that for you, you stay productive, but you stop growing in the way that used to make this fun.

The Cure Is Smaller Than You Think

Here’s what Dan did about it, and I love this. One Sunday he went and found a brand-new tool he’d never used, an Apache DataFusion based Spark accelerator, and he just ripped it apart by hand. Barely any blogs about it, AI couldn’t help much because it was too new, and he couldn’t even get it working. His wife asked why he was mad and he didn’t want to talk about it.

And he had the best time he’d had in ages. Because it took him right back to ten years earlier, breaking things, struggling, figuring it out alone. There is almost no better feeling than fighting a problem for an hour, walking away, and solving it in five minutes when you come back. For a moment you feel like a god.

Stay the Expert, Not the Prompt Machine

I’m not telling you to stop using AI. I use it every day too. The point is to use it as a tool and keep yourself as a person, as the engineer you are, in the loop, so you stay well-rounded instead of becoming someone who only knows how to talk to a prompt.

Remember where you came from. Keep the business context in your head. Understand the architecture. And every once in a while, go find something brand new and break it yourself, just because you love it.

When was the last time you sat down and struggled with something by hand, no AI, just because it was fun?

I discussed this and other topics in my latest podcast with guest Dan Beach from Data Engineering Central.

Watch it on YouTube. Or listen to it on your favorite platform.

***

Ready to become a Data Engineer? Then join my Learn Data Engineering Academy today!

Looking for free hands-on projects and tutorials? Then check out my Free Labs.

More Data Engineering content: YouTube | LinkedIn | Spotify

I posted on LinkedIn asking what the most overrated topic in data engineering is, and I said data modeling. A bunch of people came in to tell me this was a big L take, the worst thing I’d ever posted.

Honestly, those are my favorite posts. If everyone agrees with you, what’s the point of saying anything?

What I’m Not Saying

Let me be clear, because this is where people get mad. I’m not saying data modeling doesn’t matter. I’m not saying it can’t make things better. Lots of things make things better.

What I’m saying is that the way it’s usually taught is ignoring the reality of the last ten years. We’re not in the SQL Server world anymore, spending all day writing queries and building indexes.

Most people on a lakehouse today are not going into Databricks and creating indexes for anything. The ground shifted, and a lot of the old emphasis didn’t shift with it.

Medallion Kind of Proves the Point

Think about medallion architecture. Everybody uses it, everybody talks about it, and what is it really? It’s “drop your data in three different spots and clean it up on the way through”. That’s it.

That’s not data modeling in the old sense. If running Kimball-style dimensional modeling on top of your lakehouse feels right to you, that’s mostly preference, a way of thinking you’re comfortable with.

The fact that the whole industry adopted a simple bronze-silver-gold pattern instead tells you something about how much heavy modeling people actually need day to day.

AI Is Genuinely Good at This Part

Here’s the other reason it’s overrated. AI is very good at data modeling, as long as you know what you’re doing and give it good context.

It’s a structured, well-defined problem, and the models were trained on an enormous amount of Postgres schemas and SQL. You don’t need ten courses and a stack of books to spit out a reasonable schema anymore.

If you have the free time and you want to learn it deeply, go for it, I’ll never tell anyone not to learn something. But it’s not the make-or-break skill people pretend it is.

Where It Still Matters

I’m not throwing it out completely. A data model still shows up in real ways. You’ve got a streaming application, data is coming in, and it has to land somewhere.

People immediately jump to dimensional modeling, but plenty of the time that’s wrong. Sometimes it goes into a document store because a UI needs to read it, and that still needs a thoughtful model of how you store the data.

So it’s not dead. It’s just nowhere near the most important thing, and at scale in a lakehouse, beyond clustering and partitioning, it simply doesn’t carry the weight it used to.

Old habits die hard. So tell me honestly, is data modeling still as important as people say, or are we just teaching it out of habit?

I discussed this and other topics in my latest podcast with guest Dan Beach from Data Engineering Central.

Watch it on YouTube. Or listen to it on your favorite platform.

***

Ready to become a Data Engineer? Then join my Learn Data Engineering Academy today!

Looking for free hands-on projects and tutorials? Then check out my Free Labs.

More Data Engineering content: YouTube | LinkedIn | Spotify

Most people hit the same wall when they start learning Spark.

They write code, filter some data, join a table, select some columns, and nothing happens.

Then they run one more line and suddenly the whole thing executes at once.

It feels broken, but it’s not.

Two Types of Operations, That’s It

Everything you do in Spark is either a transformation or an action. That’s the whole model.

Transformations describe what should happen to the data, and actions tell Spark to actually do it.

Once you get this, Spark stops feeling random.

What Transformations Actually Do

Selecting columns, filtering rows, joining datasets, grouping data. All of that is a transformation. Every time you do one, Spark creates a new DataFrame.

And here’s the thing: DataFrames are immutable, so you never modify one. You always create a new one.

The super important part is that transformations are lazy. Spark doesn’t process the data when you write the code. It builds a plan. You can chain ten transformations in a row and Spark just waits, because that lets it optimize the whole thing at once: combining steps, pushing filters down, deciding how to split the work across the cluster.

This is why Spark scales. It’s planning before acting, and the data stays distributed the whole time. Nothing gets sent back to your driver machine.

What Actions Actually Do

Actions are the trigger. When you call an action, Spark executes everything it’s been quietly planning up to that point. Now the cluster actually starts working, right?

So there are two kinds of actions. Some don’t send data back to the driver at all. Things like write, saveAsTextFile, or foreach. They push results directly to storage or process data on the executors. These kind of actions are fine, whatever the data size.

But then you have the ones that do send data back to the driver, and this is where people get into trouble. collect, count, show, first, take, head. These all pull data back to one machine. The driver. And the driver is just one machine.

So if you have 16GB or 32GB of driver memory and you suddenly dump a massive result set back to it, you either create a serious bottleneck or your job just crashes.

That’s not a Spark problem. That’s an architecture misunderstanding. And it happens a lot.

The Mental Model That Makes It Click

The sequence is straightforward: you write transformations, Spark builds a logical plan called a DAG, an action appears, the driver sends tasks out to the executors, the executors process data in parallel, and results are either written out or returned to the driver.

Transformations are planning. Actions are execution. That’s the whole system.

Mistakes I See All the Time

Using collect() on large datasets. Counting massive tables in production. Overusing show() while debugging. Forgetting that nothing runs until an action appears.

Yes, these things work on a demo spark notebook, but not in production. These mistakes don’t happen because Spark is complicated, they happen because the execution model wasn’t understood first.

Once you understand it, you start designing better jobs almost automatically.

Spark Is Deliberate

People complain that Spark is unpredictable when really they’re just fighting the execution model instead of working with it. Spark builds a plan, optimizes it, and executes when told to. Transformations build the plan. Actions execute it.

If you keep that in your head, you’ll avoid a lot of frustration and a lot of production incidents.

This is honestly the moment where people stop “trying Spark” and start thinking like distributed systems engineers. The lazy evaluation isn’t a quirk. It’s the whole point.

What was your biggest Spark mistake before you understood this? Leave a comment.

***

Ready to become a Data Engineer? Then join my Learn Data Engineering Academy today!

If you want to build real platforms, master the full stack, and close your skill gaps, check out my Data Engineer Coaching program.

If you are interested, but still have a few burning questions on your mind: feel free to contact me via hello@learndataengineering.com.

For more information and content on Data Engineering, also check out my other blog posts, videos and more on Medium, YouTube and LinkedIn!

Recently I ran a test. I wanted to know how fast I could hit the Claude Pro usage limit if I just used Claude the way most developers use it — opening sessions, asking questions, pasting code, fixing bugs, with no real strategy behind any of it.

It took me 2.5 hours.

That was a wake-up call. Not because the limit is too low, but because I clearly wasn’t getting the most out of the tool. And I see this same problem with developers everywhere. They use Claude like a fancy version of Stack Overflow — type a question, get an answer, paste it in, repeat. It works, but it leaves most of the value on the table.

The developers who actually get 2–3x more done with Claude aren’t writing better prompts. They’re not paying for a more expensive model. They’re architecting the session.

These are the 5 habits that make the difference. None of them are complicated. But nobody tells you about them, and you only figure them out after burning through your usage limit a few times.

Real gains don’t come from better models — they come from architecting the session.

1. Plan Before You Code. Always.

The single biggest lever in AI coding isn’t a better prompt or a smarter model. It’s planning before you write a line of code.

The most expensive failure mode is Claude confidently writing 400 lines in the wrong direction.

Five minutes of planning prevents an hour of cleanup — and a lot of wasted tokens.

Plan inside Claude Code for small to medium tasks For anything non-trivial inside your coding session, press Shift+Tab to enter plan mode. Claude explores the codebase, proposes an approach, you review and approve it, and then implementation starts.

Use the opusplan model for this workflow. Opus reasons through the plan, Sonnet writes the code.

You get the expensive thinking only where it actually pays back.

Plan in the Claude app first for bigger work

This is the step most developers skip — and it’s a game-changer.

For new projects, major features, or significant refactors, don’t start in VS Code.

Open the Claude app in your browser and start a fresh chat.

Talk through what you’re building. Push back on Claude’s first answer. Change your mind. Throw out an entire approach and start over. In a chat, iteration is cheap. In a coding session, every pivot costs you context and tokens.

When the plan feels right, ask Claude to turn it into a project brief:

goals, constraints, file structure, key decisions, tech choices, and definition of done.

Bring the plan into your coding session Now you have two options:

• Paste the brief as the opening prompt in your Claude Code session

• Drop it into your CLAUDE.md file if it’s project-wide context that should persist across sessions

The principle underneath this: plan in the medium where iteration is cheap. Two-step planning gives Claude the why behind your code, not just the what. That’s what stops it from making bad decisions on your behalf later.

2. Write Specific Prompts With Verification Targets

“Improve this code” is the worst prompt you can write. It’s vague, it triggers broad scanning across files, and it almost guarantees Claude does something you didn’t actually want.

Compare these two prompts:

“Improve the auth module”

vs.

“Add input validation to the login function in auth.ts. Reject empty email, validate format with the existing isEmail util, return 400 with {error: ‘invalid_email’}. Test case: login(’‘) → 400.”

The second one takes 30 seconds longer to write and saves you 20 minutes of fix-up rounds.

Rule 1: Name the file, name the function

Don’t make Claude guess where the work happens. If you can point to the exact location, do it.

Every time you save Claude from having to search the codebase, you save tokens and time.

Rule 2: Give a verification target

Paste the failing test, the expected output, or a screenshot of the broken UI.

When Claude can verify its own work before handing it back to you, it catches issues before you have to ask for fixes.

Why this compounds

Every prompt either saves you a turn or costs you one. Across a session, that’s the difference between shipping a feature and still debugging it at 6pm. The 30 seconds you spend writing a clear prompt pay back ten times over by the end of the day.

3. Manage Context Like It’s a Budget — Because It Is

Your context window isn’t infinite. Every file you load, every command you run, every piece of tool output Claude sees — it all stacks up. When the window fills, performance degrades and Claude starts losing the thread of your work.

Three habits handle 90% of context problems.

Use /clear between unrelated tasks

Stale context taxes every subsequent message. If you spent the morning debugging auth and now you’re writing tests for the payments module, hit /clear. Before you do, run /rename so you can find the session later, then /resume to come back to it if you need to.

Use /compact at natural breakpoints

/compact summarizes your conversation history and replaces it with a compressed version.

Use it proactively — at the end of a subtask, before switching focus — not when you’re already at 95% full.

Tell it what to keep:

/compact keep the auth refactor decisions and the schema changes

That single line preserves what matters and lets the rest go.

Run /context to see what’s actually loaded

Most developers are shocked the first time they run this. Their CLAUDE.md, MCP servers, and skills are eating 20% of the window before they’ve typed a single character.

If that’s you, fix the base load first:

• Trim CLAUDE.md to under 200 lines

• Disable MCP servers you don’t actively use (/mcp to list them)

• Move detailed workflow instructions out of CLAUDE.md and into skills, which only load when invoked

Watch out for files that are too big for the context window

Here’s a real problem I ran into. I was developing a notebook that grew so large Claude couldn’t load it directly into context anymore. To read or edit it, Claude had to use a Python function — which meant every read and edit dumped huge output into context. On top of that, I lost visibility into what Claude was actually changing.

The fix: split big files into smaller, single-purpose ones before you start AI work. The same principle applies to huge log files, data dumps, and generated code. Preprocess or chunk them before Claude ever touches them. If a file is too big to load cleanly, it’s already costing you.

4. Match the Model and the Effort to the Task

Two independent dials. Most developers only know about one.

Model choice — how smart the engine is

• Sonnet is your default. It handles 80% of coding work at a fraction of Opus cost.

• Opus is for genuine architectural decisions and multi-file reasoning. Don’t sit on it for everything.

• Haiku is for cheap, fast, mechanical work.

Switch mid-session with /model. The right answer is almost never “use this one model for everything” — it’s “switch based on what you’re doing right now.”

Effort — how long Claude thinks before answering

This is the dial most developers don’t even know exists. Set it with /effort.

Extended thinking tokens are billed as output tokens, and the default budget can be tens of thousands of tokens per request. That adds up fast.

The levels, roughly:

• Low — renames, typo fixes, obvious feature work

• Medium — default for most coding

• High — gnarly debugging, architectural decisions, multi-file refactors

You can also set MAX_THINKING_TOKENS=8000 for an entire session focused on simpler work, then crank it back up when you hit something hard.

The key insight

Sonnet on high effort often beats Opus on low effort, at lower cost.

The thinking budget is doing the heavy lifting, not the model tier.

The common mistake

Cranking everything to max for the entire session. You’ll burn through tokens on trivial requests where Claude is “thinking” about whether to use let or const. Dial it down for the boring 80% of work, and dial it up for the 20% that actually needs reasoning.

5. Debugging: Paste Errors Properly and Let Claude Diagnose Before It Fixes

Debugging is the most common AI coding task — and most developers do it wrong.

They paraphrase errors, skip context, and let Claude jump straight to fixing without diagnosing.

Paste the full error verbatim

Stack trace, line numbers, error code, everything. Don’t paraphrase, don’t summarize.

Claude needs the raw signal — the exact error message often contains the clue that solves the bug.

Include what you already tried

A quick note like “I checked X, ruled out Y, the issue persists when Z” stops Claude from suggesting things you’ve already eliminated. Saves a full back-and-forth round.

Point to the file and function

Same rule as Tip 2 — name the location. Don’t make Claude guess where the bug lives.

For non-obvious bugs, use plan mode

Force Claude to diagnose before touching the code. This is especially important for intermittent bugs, race conditions, and issues that span multiple files. The worst thing Claude can do is “fix” the symptom without understanding the cause.

Match effort to bug complexity

• Missing import, typo, null check → low effort, Sonnet

• Distributed system failing under load, weird state issue, intermittent failure → high effort, possibly Opus

Cut losses early with /rewind

If Claude starts heading the wrong direction, double-tap Escape or use /rewind to restore your conversation and code to a previous checkpoint. Cutting losses early saves more tokens than any optimization technique. Don’t let a bad path eat your session.

The Bigger Picture

The thread running through all 5 habits is the same:

your job isn’t to type prompts. Your job is to architect the session.

• Plan first

• Scope tight

• Model-match

• Keep context lean

• Debug methodically

The gap between developers who do this and developers who don’t widens with every project.

On a small task it’s 10% faster. On a real codebase, it’s the difference between shipping and not shipping.

Remember the 2.5 hours I burned through Pro at the start of this article? That was me with zero of these habits.

Imagine what’s possible with all five.

***

Ready to become a Data Engineer? Then join my Learn Data Engineering Academy today!

If you want to build real platforms, master the full stack, and close your skill gaps, check out my Data Engineer Coaching program.

If you are interested, but still have a few burning questions on your mind: feel free to contact me via hello@learndataengineering.com.

For more information and content on Data Engineering, also check out my other blog posts, videos and more on Medium, YouTube and LinkedIn!

Hey folks,

I shared a few things this past week that I think are flying under the radar right now. A hands-on build, and two topics that I believe matter a lot more than the industry is admitting.

Here’s what you might have missed:

The biggest problem with AI that nobody talks about

I ran a quick poll this week: do you trust AI-generated reports? 10% said yes. 59% said only if they can verify it.

That’s exactly the problem I’ve been running into while building an AI-powered pipeline monitoring system. And it raised a question the AI hype cycle doesn’t want to answer: how do you actually trust the output?

Not ChatGPT answers. I’m talking about production BI reports and reliability dashboards that executives base decisions on. With SQL you can verify the query. With vector search and an LLM on top? The logic is invisible. That’s a real problem.

Read the full article

I Built an AI Agent You Can Ask Why Your Pipeline Is Failing - With Oracle AI Database 26ai

Most pipeline observability setups need 3 separate systems: a metadata store, a log store, and a vector database. That means 3 connections, 3 things to break, 3 places to debug.

In my latest video I show you how to consolidate all of that into one using Oracle’s AI Database 26ai, and build a LangGraph ReAct agent on top that answers natural language questions about your pipeline failures.

Ask why a DAG keeps failing on Mondays. It figures out whether to run SQL, do a semantic log search, or both. Runs completely locally via Docker, for free.

Watch the video

I learned about AI harnesses and you should too

When an AI system fails, the first reaction is always: we need a better model. But very often, especially in data work, that’s not the issue.

The real issue is that the system has no proper structure around it. That’s what a harness is. It controls which tools the model uses, what it can do, where guardrails are enforced.

I think it’s way more important than most people realize right now.

Read the full article

Now on YouTube and Udemy: Spark Declarative Pipelines on Databricks

My free 2-hour Databricks lab is now available as a full course on both platforms.

It covers Spark Declarative Pipelines and Lakeflow Designer: how to define Bronze → Silver → Gold pipelines with SQL, let Databricks handle dependencies and orchestration, and build visual pipelines almost no-code.

If you’re still managing pipeline logic in notebooks, this is worth two hours of your time.

Start course on Udemy

Start course on YouTube

​

That’s it for this time 🙌 If any of this hit home, especially the trust question around AI outputs, I’d love to hear how you’re thinking about it in the comments.

Talk soon,
Andreas

***

Ready to become a Data Engineer? Then join my Learn Data Engineering Academy today!

If you want to build real platforms, master the full stack, and close your skill gaps, check out my Data Engineer Coaching program.

If you are interested, but still have a few burning questions on your mind: feel free to contact me via hello@learndataengineering.com.

For more information and content on Data Engineering, also check out my other blog posts, videos and more on Medium, YouTube and LinkedIn!

I‘ve been building an AI-powered pipeline monitoring system. Ask it a question, get a report. Cool concept, right?

But here’s the thing nobody in the AI hype cycle wants to admit: how do you actually trust the output?

Prefer to watch instead of read? I covered this hands-on in my latest YouTube video:

This Is Not a ChatGPT Problem

I’m not talking about asking ChatGPT a question and wondering if the answer is correct. That’s a different thing.

I’m talking about production systems. You have a BI report, a sales report, a reliability dashboard, and an AI agent is generating it. Your business makes decisions based on that report. Can you trust it?

That is the real problem.

Here’s What I’m Actually Building

I’m working on a system that analyzes pipeline failures. I have DAGs, ingestion jobs, aggregation, partitioning, and so on. I create metadata around runs and failures, I store logs, and I use embeddings and vector search to make everything queryable.

The idea: instead of digging through logs manually, you just ask a question. “How many pipeline failures in the last 30 days?” And you get an answer.

When it’s a SQL query under the hood, I can verify it. I can look at the generated SELECT statement and check if the logic is right. That part I trust.

But when it’s a vector search against log embeddings? That’s where it gets uncomfortable.

The Trust Problem With Vector Search

The system found schema drift issues on specific dates. It found quality check failures with exit code one. It generated a full reliability report based on all of this.

But how do I know that’s the complete picture? Maybe my semantic search didn’t match everything relevant. Maybe the embedding didn’t surface the right logs. The system doesn’t tell me what it missed. It just tells me what it found.

That’s the problem. With SQL, you can read the query. With vector search and LLM reasoning on top of it, the logic is invisible.

You Can Test It in a Controlled Environment

If you set up a controlled environment where you know exactly which errors exist, you can write tests. Does the system return the right results for this specific scenario? Yes or no.

That works in development. But in production? With real, messy data? With errors you didn’t anticipate? You can’t write tests for every possible failure mode.

And if this is generating a monthly sales report that executives act on, “it worked in dev” is not good enough.

The Old Way Was Predictable

Before all this, you had SQL tables. You ran queries. You got exact results. Before that, Excel with formulas. You could trace every number back to its source. It was deterministic. Predictable. Auditable.

Now you ask a question in natural language and get an answer that might be different the next time you run it. The same question, a different day, slightly different results depending on what the vector search surfaces. That’s not a bug, that’s how it works. But it makes trust really hard.

One Tool I Know That’s Working on This

One company I’ve worked with that’s actually tackling this is arato.ai. They basically compare what you gave the LLM against what it returned and measure how well the output fits the input. They also flag things like hallucinations and off-topic responses at scale.

But honestly, I don’t know of a solid, widespread solution for this yet. And I find it suspicious that most people selling AI-powered analytics tools aren’t talking about it at all.

This Needs a Real Answer

If you’re building AI systems that people rely on for real decisions, this isn’t a nice-to-have. Trust and verifiability are core requirements. Not features. Requirements.

I don’t have a clean answer here. I don’t think many people do.

If you know of libraries, frameworks, or approaches for testing and validating AI-generated reports at scale, drop them in the comments. I’ll be looking into this more and I want to know what’s actually working out there.

***

Ready to become a Data Engineer? Then join my Learn Data Engineering Academy today!

If you want to build real platforms, master the full stack, and close your skill gaps, check out my Data Engineer Coaching program.

If you are interested, but still have a few burning questions on your mind: feel free to contact me via hello@learndataengineering.com.

For more information and content on Data Engineering, also check out my other blog posts, videos and more on Medium, YouTube and LinkedIn!

In a live stream with the guys from Altimate AI, we talked about something that was new to me:

AI harnesses.

And the more I think about it, the more I find it a really interesting topic.

Watch the live stream with Altimate here:

You might have actually seen this already without calling it that.

For example with OpenClaw. A lot of people looked at it and thought, okay, this is a cool setup around an LLM. But that is basically what a harness is. A structured way to control how the model operates, which tools it uses, and what it is allowed to do.

People are talking a lot about models, prompt engineering, context windows and all that. And yes, these things matter. But I think there is another topic that is much more important than most people realize right now, especially for data engineering.

That topic is AI harnesses.

And I think this is where a lot of people are still looking at the problem the wrong way.

It is not always the model

When an AI system fails, the first reaction is often: the model is not good enough. We need a better model. We need the newest model. We need more reasoning power.

Sometimes that is true.

But very often, especially in data work, that is not the real issue.

The real issue is that the system has no good way to operate inside the environment. It has no proper structure around the model. It does not know how to move through the task in a controlled way. And that is exactly where the harness comes in.

What a harness actually is

So what is a harness?

For me, the easiest way to think about it is this: the harness is the layer around the model that defines how the model actually works in practice.

• It decides which tools are available.

• It decides which workflows should be followed.

• It decides what the system is allowed to do and what it must never do.

• It decides where guardrails exist and how they are enforced.

• It decides what information gets pulled in and how that information is used.

That is a big deal.

This is not just coding

Engineering work is not just writing code.

A lot of the work is understanding what is going on, figuring out what is broken, and working through messy situations where the answer is not obvious.

That is exactly where AI systems struggle today.

They can generate something that looks correct. But they often miss the deeper context, hallucinate details that are not actually there, or take shortcuts that would never be acceptable in a real environment.

Without a proper harness, the agent is basically improvising in a complex system.

The real problem in data work

Because data work is not happening in one place.

The repo is only one part of the story. The real truth is spread all over the place.

• It is in the warehouse, the schemas, and the actual data values.

• It is in lineage, orchestration, and dashboards.

• It is in business definitions and naming conventions.

• It is in all the weird exceptions that companies have built up over time.

That means an agent can look very smart on the surface and still fail badly.

It can generate a report that looks polished, produce SQL that seems reasonable, and sound confident.

And then you look closer and notice it missed models, invented columns, misunderstood risk, or gave you something that would never pass in a real team.

That is why I think the conversation has to move beyond “which model is best?”

A better question is: what kind of harness is this model operating inside?

Because the harness is what turns raw model capability into usable engineering behavior.

Context alone is not enough

A good harness brings in the right context, but it does more than that. It also provides tools and skills.

For example, if the system needs lineage information, schema metadata or warehouse inspection, the harness should give it exact ways to get that information. If the system needs to create or review something, the harness should help it follow a known process instead of improvising.

That is also where guardrails become real.

And this part is super important.

A lot of people still think guardrails mean putting a sentence in the prompt like “do not expose PII” or “do not touch production.” But that is weak. That is a suggestion. It might work for a while, until the conversation gets longer, the context shifts, or the system goes off track.

Real guardrails should live outside the model. For example:

• a system should be technically unable to write in analysis mode

• a PII check should be enforced before data leaves a boundary

• unsafe actions should require a different mode or a review step

That is what gives teams trust.

Trust is the real bottleneck

And trust is the real bottleneck right now.

Most engineers are not rejecting AI because they hate automation. They reject it because they do not trust a system that can improvise freely inside a sensitive environment.

Honestly, I think that is fair.

The problem with vendor-shaped agents

The other thing that matters is independence.

A lot of vendor-built agents will naturally push you deeper into their own ecosystem. Again, that makes sense from their side. But from the user side, it can become a problem. Because now the harness is no longer only about helping you solve engineering tasks. It is also nudging you toward one stack, one platform, one way of doing things.

That may not be the best solution for your setup.

And since most companies have mixed environments anyway, this gets limiting very fast.

Why this will matter more

That is why I think this topic will become much more important.

Not because it sounds fancy, but because this is where the real control is.

Models will keep improving. That part is obvious.

But if you want AI to work like a serious engineering assistant, then the model alone is not enough.

You need the environment around it to be right.

That is the harness.

***

Ready to become a Data Engineer? Then join my Learn Data Engineering Academy today!

If you want to build real platforms, master the full stack, and close your skill gaps, check out my Data Engineer Coaching program.

If you are interested, but still have a few burning questions on your mind: feel free to contact me via hello@learndataengineering.com.

For more information and content on Data Engineering, also check out my other blog posts, videos and more on Medium, YouTube and LinkedIn!

Hey folks,

I shared a few things this past week that I think are pretty useful if you’re working with data pipelines, AI setups, or just trying to build better systems overall.

Here’s what you might have missed:

New FREE Databricks Lab + Walkthrough Recording

We just released a new free lab where you build full pipelines in Databricks using Spark Declarative Pipelines, SQL, and Lakeflow Designer.

Here, you learn how to actually build pipelines that are structured, reliable, and easier to manage.

You’ll work with Bronze → Silver → Gold pipelines, orchestration, Delta Lake, Unity Catalog, and also the visual side with Lakeflow Designer.

👉 Work through the Free Lab on YouTube:

I also did a full livestream where I walked through the lab and explained how everything fits together.

If you want a guided start, that recording is probably the easiest way to get into it.

👉 Watch the recording:

I Built My Own “ChatGPT” Locally (Open WebUI + Ollama)

In my latest YouTube video I show you how to run your own ChatGPT-like setup locally.

No API calls. No data leaving your machine.

It’s based on Open WebUI and Ollama, and it’s actually a really practical setup if you want more control or care about privacy.

This is an awesome project if you want something for your portfolio or if you’re thinking about internal AI systems.

👉 Watch the video here:

How to Stop Getting Surprised by Broken Pipelines

This is something I’ve run into a lot.

Pipelines “running fine”… but something is clearly off.

No alerts, no errors, but data is missing or delayed.

In my latest LinkedIn article, I break down why this happens and go into simple ways to fix this without needing a full observability stack.

👉 Read the full article here: https://www.linkedin.com/pulse/how-stop-getting-surprised-broken-pipelines-andreas-kretz-wkqjf

That’s it for this time 🙂

If you check any of these out, I’d be curious what you think. So let me know in the comment sections!

Especially if you try the lab or the local AI setup.

Talk soon,
Andreas

***

Ready to become a Data Engineer? Then join my Learn Data Engineering Academy today!

If you want to build real platforms, master the full stack, and close your skill gaps, check out my Data Engineer Coaching program.

If you are interested, but still have a few burning questions on your mind: feel free to contact me via hello@learndataengineering.com.

For more information and content on Data Engineering, also check out my other blog posts, videos and more on Medium, YouTube and LinkedIn!

One of the most frustrating situations I’ve run into as a data engineer is this weird moment where everything looks fine… but you just know it isn’t.

No warnings.

No alerts.

Everything is quiet.

But then someone pings you or you notice it yourself. Data is missing. A dashboard is off. Something is delayed and doesn’t make sense.

And now you’re sitting there, trying to figure out what actually happened, with zero clues. Going through logs for an hour.

In this article, I want to show you why this happens, what’s actually missing in many setups. And of course how you can avoid ending up in that situation in the first place.

Because most of the time, it’s not the pipeline that failed. It’s the lack of observability around it.

The system didn’t break. Your visibility did

When something goes wrong in a data pipeline, the first thing you ask is:

• What changed?

• Where did it happen?

• What exactly did we process before things went weird?

If you can’t answer that within a few minutes, you don’t have a reliability problem.

You have an observability gap.

And that’s not something you can fix in the middle of a fire.

Add metadata, even if it feels like overkill

In one of my coaching sessions, the student had mapped out a great pipeline design.

Streaming inputs, processing, two layers of storage.

But one thing was missing: Metadata about the data itself.

It sounds redundant. But it isn’t.

Simple additions like:

• When did the data arrive?

• What did the payload look like?

• How many records came through?

• Was the schema consistent?

These small things become huge when something fails silently.

Especially in high-frequency environments, where you can’t scroll through logs or manually replay messages.

You need a trace of what flowed through the system, even if the system “ran fine.”

Metadata gives you levers for debugging, monitoring, and trust

Here’s what good observability unlocks:

• ✅ You can spot volume anomalies before someone complains

• ✅ You can track the exact moment latency started creeping up

• ✅ You can alert on silence—because no data can be worse than bad data

• ✅ You can verify that transformation jobs aren’t skipping edge cases

• ✅ You can answer the question: “what exactly did we process?”

Without that, all you’ve got are charts of CPU usage and lag.

Which is great, until a payload goes missing, and you’re stuck guessing.

“We’ll add that later” almost always means “we won’t”

I’ve seen this one play out a dozen times.

The team is under pressure to ship. Metadata is a “nice-to-have.”

So they build the pipeline, it works… and they move on.

Then one day, someone asks: “Hey, did we process that update last Tuesday?”

“Did the job run? What offset was it on? Why did this record disappear?”

And no one knows. Because no one logged it. Because they were too busy to add the visibility that now seems essential.

If you’ve ever tried to debug a broken pipeline without metadata, you know the feeling.

You don’t need full-blown observability tools to start

You don’t need a full observability tool stack on day one.

Sometimes it’s enough to:

• Write simple audit records to a separate table

• Log timestamps and payload shapes per message batch

• Store summary stats: event count, schema version, volume

If you do have streaming infrastructure, you can even wire up metadata capture as part of your processing step. Just remember that every tool you integrate needs to feed your understanding, not just your dashboards.

Build It While It’s Still Quiet

A clean pipeline feels good.

A fast one feels even better.

But honestly, neither of those help you when something silently goes wrong at 2am and you have no idea where to start.

That’s where observability becomes real.

For me, this is not about tools or fancy dashboards. It’s more about how you think when you build something. You’re not just building a pipeline that works today. You’re building something that your future self needs to understand when things stop making sense.

And trust me, they will.

If you take one thing away from this, it’s this: Add the visibility while everything is still working.

Because once it breaks, it’s already too late to wish you had it.

Observability is not something you bolt on later. It’s something you build in from the start, even if it feels a bit unnecessary in the moment.

That “overkill” you’re thinking about right now is usually exactly what saves you later.

Hope this helps you at some point in the future :)

***

Ready to become a Data Engineer? Then join my Learn Data Engineering Academy today!

If you want to build real platforms, master the full stack, and close your skill gaps, check out my Data Engineer Coaching program.

If you are interested, but still have a few burning questions on your mind: feel free to contact me via hello@learndataengineering.com.

For more information and content on Data Engineering, also check out my other blog posts, videos and more on Medium, YouTube and LinkedIn!

Hey everyone,

You know, I’m fed up with having to create polished videos for my main channel all the time.

It’s a lot of pressure when videos have to perform because otherwise it looks bad.

That’s why I’m starting new again, sharing interesting topics with you. Without worrying how many people see it. Just like it was 10 years ago.

No script, no big setup. Just me talking, showing you interesting builds, ideas and nuggets that I find on the internet.

That’s why I created Engineering Unscripted.

👉 Check out the channel here: https://youtube.com/@engineeringunscripted?si=SFZkc4hfYB_Huv_7

(Do me a favor: only subscribe if you’re into more hands-on, unscripted stuff like this.)

First video is already out: I built an MCP server for AI agents

MCP servers actually sound like a super interesting topic and so I built one.

I’ll show you around what I have built, how it works and we are going to try it out.

I think you are going to like it.

👉 Watch video now:

What’s Next

On Friday we’ll dive into the AWS Well-Architected Framework, which is actually something completely different from what I thought.

And I’m also going to upload a video about OpenWebUI soon where you can basically build your own private ChatGPT.

Super fun topic.

The original 240k subscriber channel

I already have free courses lined up for the main channel Andreas Kretz (next will be Databricks).

And I’ll keep making interesting scripted videos there.

But now I am excited to fully start with the new one.

Leave me a comment how you like it so far and what I should cover next.

That would be awesome!

Andreas

***

Ready to become a Data Engineer? Then join my Learn Data Engineering Academy today!

If you want to build real platforms, master the full stack, and close your skill gaps, check out my Data Engineer Coaching program.

If you are interested, but still have a few burning questions on your mind: feel free to contact me via hello@learndataengineering.com.

For more information and content on Data Engineering, also check out my other blog posts, videos and more on Medium, YouTube and LinkedIn!

You can usually spot it pretty quickly.

A project kicks off with something like: “We’ll use Kafka, Spark, Snowflake, and Airflow.”

And only after that comes the actual question: “So… what are we building?”

If you’ve seen this before, you’re not the only one. This happens all the time, and it’s one of the easiest ways to make things more complicated than they need to be.

At the core, it’s this mistake: choosing tools before understanding the problem.

The stack-first trap

Look, I get it. Tools are fun.

Sometimes you just want to try something new, learn a specific technology, and build something around it. That’s completely fine when you’re experimenting or doing a side project.

But once this turns into something that people rely on, the mindset has to change.

It’s not about “what can I build with this tool?”

It’s more about:

• What are the actual use cases?

• Who is going to use this, and how?

• How fast does it need to be?

• How much can we spend?

• What happens if something breaks?

You won’t get those answers from a benchmark chart or from looking at GitHub stars.

Use case first, architecture second, tooling last

The projects that go well usually start in a very different way.

You define what you actually want to achieve. Then you look at the constraints. Then you think through how people are going to use the system.

Only after that does the architecture start to make sense.

And once you have that, a lot of decisions become almost obvious.

• Do you actually need streaming, or is batch enough?

• Do you need low latency, or is a delay acceptable?

• What kind of storage fits your access patterns?

• Do you really need strong consistency everywhere?

These are the questions that shape the system.

Tools come later. And when they do, you often realize that your first idea for the stack was not the best fit.

Real example: streaming doesn’t automatically mean real-time

I had a coaching session where the initial idea was to run everything in real-time with Spark.

Sounds powerful, right?

But when we broke it down, the actual needs were very different:

• Most data was coming in weekly from APIs

• Analysts were working mainly on historical data

• There was a small part with live data, but only for a few specific cases

So instead of building a complex real-time system, we simplified it.

A small streaming setup where it actually mattered, and batch processing for everything else.

Spark was not needed at all in the end.

And that’s the point. Once you understand the use case, a lot of complexity just disappears.

Tools are not your strategy

This is something I keep repeating because it’s easy to forget.

Tools should follow your strategy, not define it.

❌ Don’t pick Kafka because it’s “the standard”.

❌ Don’t pick Snowflake because everyone talks about it.

❌ Don’t pick Spark just because it sounds like the enterprise choice.

Pick something because it solves your problem in your situation.

That includes your team, your budget, your timelines, and how complex your system actually needs to be.

Very often, the better solution is the simpler one.

What I’d keep in mind

There’s no reward for having the most complex stack.

The engineers who do this well usually slow down at the beginning and ask the right questions.

✅ What really needs to happen here?

✅ Where does speed actually matter?

✅ What can we keep simple without breaking things?

Once you have clear answers to that, the rest becomes much easier. And then you can still use great tools. Just for the right reasons.

***

Ready to become a Data Engineer? Then join my Learn Data Engineering Academy today!

If you want to build real platforms, master the full stack, and close your skill gaps, check out my Data Engineer Coaching program.

If you are interested, but still have a few burning questions on your mind: feel free to contact me via hello@learndataengineering.com.

For more information and content on Data Engineering, also check out my other blog posts, videos and more on Medium, YouTube and LinkedIn!

AI is changing how we work as engineers.

But not in the way most people think.

Let’s break down what’s actually happening and what you need to focus on if you want to stay relevant going into 2026 and beyond.

🎥 If you prefer watching a video on this topic instead of reading, check out my latest YouTube video:

The Shift You Need to Understand

Think about this:

On one axis, you have work complexity.
On the other, efficiency gain from AI.

Now here’s the key insight:

• Junior work → high efficiency gains

• Mid-level work → moderate gains

• Senior work → lower gains (for now)

Why?

Because junior work is usually:

• well-defined

• limited in scope

• easier to automate

And that’s exactly where AI performs best.

What this means for you:

AI is not replacing engineers.

But it is replacing parts of the work that engineers used to do, especially at the junior level.

That’s why:

• it’s getting harder to break into the field

• expectations are increasing

• and “just coding” is no longer enough

The 3 AI Waves You Need to Be Aware Of

To understand where things are going, it helps to look at AI in waves.

Wave 1 — Coding with AI (Already here)

This one is done.

Everyone is using AI to code:

• Copilot

• Cursor

• Claude

Nobody works the same way anymore.

This is where we see the biggest efficiency gains, especially for junior-level tasks.

Wave 2 — Agents and Automation (We are here)

This is the current wave.

AI is no longer just helping you write code.
It’s helping you do the work around it:

• analyzing logs across systems

• debugging faster

• automating workflows

• pulling insights from multiple sources

This starts to impact mid-level engineers more.

Because now it’s not just about writing code.
It’s about operating and understanding systems.

Wave 3 — Autonomous Management and Optimization (Coming next)

This is where things are heading.

The idea:

• systems that optimize themselves

• systems that manage infrastructure

• systems that make decisions

We’re not there yet.

Even if it feels like it sometimes, we’re still very much in the agent phase.

Humans are still in the loop.
But when this wave matures, it will start affecting senior-level work.

What You Need to Focus On (Based on Your Level)

This is where it becomes practical.

If You’re a Junior

The bar is higher now.

You can’t rely on small, isolated tasks anymore.

1. Learn how to code with AI properly

Not vibe coding.

You need to:

• understand what the AI is doing

• guide it

• correct it

• move quickly with it

2. Build full pipelines

Show that you can:

• ingest data

• transform it

• store it

• orchestrate it

End-to-end.

3. Understand RAG basics

You don’t need to be an expert. But you should understand:

• how retrieval works

• how AI systems use external data

• how simple RAG pipelines are built

4. Build real projects

Not isolated tools.

Complete systems.

That’s what differentiates you now.

If You’re Mid-Level

Everything from junior still applies. But your focus shifts.

1. Performance and cost optimization

AI can help.
But you need to make the decisions.

2. Reliability and observability

Think beyond pipelines:

• monitoring

• logging

• debugging

• system behavior

3. Architecture thinking

You should be able to:

• understand requirements

• design systems from scratch

• evaluate trade-offs

4. Integrate AI into your workflow

Not just coding.

Use AI for:

• debugging

• monitoring

• analysis

This is where you become significantly more valuable.

If You’re a Senior

Your role becomes more strategic.

AI-assisted platform design

That means:

• understanding business requirements

• evaluating existing systems

• designing architectures

• using AI as a tool in that process

You’re not just building systems.

You’re deciding:

• what should be built

• how it should be built

• and why

AI helps you move faster.
But the responsibility stays with you.

What Actually Changes (And What Doesn’t)

AI is hitting:

• low-complexity work first

• then system-level tasks

• and eventually decision layers

But we’re not at full automation.

Right now, the best results still come from: humans working with AI

How You Stay Relevant

If you strip everything down, this is what matters:

Don’t optimize for tools. Optimize for ownership.

That means:

• building things end-to-end

• understanding how systems behave in production

• being able to explain why they’re built that way

Tools will keep changing.

We’ve already seen:

• new AI tools every few months

• new workflows constantly

• new abstractions on top of old ones

If you anchor yourself to tools, you’ll always be catching up.

If you anchor yourself to:

• systems

• decisions

• trade-offs

you stay relevant regardless of what changes.

And that’s really the shift.

AI is not removing engineers.
But it is raising the bar for what makes you valuable.

Ready to become a Data Engineer? Then join my Learn Data Engineering Academy today!

If you want to build real platforms, master the full stack, and close your skill gaps, check out my Data Engineer Coaching program.

If you are interested, but still have a few burning questions on your mind: feel free to contact me via hello@learndataengineering.com.

For more information and content on Data Engineering, also check out my other blog posts, videos and more on Medium, YouTube and LinkedIn!

Subnetting is one of those fundamental computer science topics, that I have seen so many people struggle with in my years of coaching.

At some point, you’re going to build something in the cloud.
AWS, Azure, GCP. It doesn’t matter. And then you run into things like:

• IP addresses

• Subnet masks

• CIDR blocks

• Ports

• Routing between networks

And if you don’t understand subnetting, things get confusing very quickly.

So let’s break it down in a simple way.

🎥 If you prefer watching a video on this topic instead of reading, check out my latest YouTube video:

IP Addresses Are Just Numbers Built from Bits

You’ve probably seen an IP address like this:

192.168.10.45

Each of these four numbers is built from 8 bits.

Each bit has a value:

128 64 32 16 8 4 2 1

So if we take the last part of the IP:

45

This is just a combination of those values.

For example:

32 + 8 + 4 + 1 = 45

That’s it. That’s how the number is built.

Same idea applies to every part of the IP address.

What Does /27 Actually Mean?

You’ll often see something like:

192.168.10.45/27

That /27 is the CIDR notation, and it tells you how many bits belong to the network.

A subnet mask for /27 looks like this:

255.255.255.224

Why?

Because:

• 255.255.255 = 3 octets (8+8+8) 24

• 224 = first 3 bits are 1 → 11100000

So in total:

8 + 8 + 8 + 3 = 27 bits

That’s where the /27 comes from.

Finding the Network Range

Now the important part: how do we know which network an IP belongs to? Because If two IP’s are not in the same network you need a Router to “connect” them. Otherwise these two hosts are not able to talk with each other.

With /27, the remaining bits define the host space.

The key insight:

👉 The block size is 32

Why?

Because:

256 - 224 = 32

So networks increase in steps of 32:

0 – 31
32 – 63
64 – 95
96 – 127
...

Example: Where Does 192.168.10.45 Belong?

Look at the last number:

45

Which range does it fall into?

32 – 63

So:

• Network address: 192.168.10.32

• Broadcast address: 192.168.10.63

• Usable hosts: 192.168.10.33 → 192.168.10.62

And yes—our .45 sits right inside that range.

What About the Other Ranges?

For example:

• 192.168.10.70 → next network (64–95)

• 192.168.10.20 → previous network (0–31)

So subnetting is really about grouping IPs into blocks.

Network vs Host vs Broadcast

Each subnet always has:

• Network address → first IP (not usable)

• Broadcast address → last IP (not usable)

• Host addresses → everything in between

Why This Matters (Especially in AWS)

This isn’t just theory.

You’ll need this when:

• Creating VPCs

• Designing subnets

• Setting up routing tables

• Connecting multiple networks

• Managing security groups and ports

A simple way to think about it:

• IP address = the house

• Port = the door

• Network = the neighborhood

If you don’t understand how the neighborhoods are structured, nothing connects properly.

Use a Subnet Calculator (Seriously)

You don’t have to calculate everything manually every time.

There are great subnet calculators where you can input:

• IP address

• CIDR block

And they’ll give you:

• Network address

• Broadcast address

• Host range

Super useful when you’re working in real projects.

👉 Here you can find the subnet calculator I used so you can practice on your own: https://jodies.de/ipcalc?host=192.168.10.45&mask1=27&mask2=

The point where it makes sense

Subnetting is one of those topics that feels confusing at first.

But once you understand

• how bits build numbers

• how CIDR defines networks

• how ranges are calculated

…it just starts to make sense.

At that point, you’re not guessing anymore. You actually see how everything fits together.

If you’ve struggled with this before, you’re definitely not alone.

I’m curious: was subnetting something that confused you early on? Let me know in the comments!

Ready to become a Data Engineer? Then join my Learn Data Engineering Academy today!

If you want to build real platforms, master the full stack, and close your skill gaps, check out my Data Engineer Coaching program.

If you are interested, but still have a few burning questions on your mind: feel free to contact me via hello@learndataengineering.com.

For more information and content on Data Engineering, also check out my other blog posts, videos and more on Medium, YouTube and LinkedIn!