On September 16th, Astronomer is bringing together industry experts to explore what’s next in data engineering at their Orchestrate Everything online data engineering conference.

You’ll hear REAL orchestration stories from data teams at Lyft, Wix, and Ramp, and on the patterns they’re using for AI inference, context engineering, MLOps, and AI-native data engineering.

Marc Lamberti, fellow data content creator and Astronomer expert, will help you prep for the official AI Orchestration Airflow certification exam, which you will take with a free certification code ($150 value).

This is a learning opportunity you don’t want to miss. You’ll walk away with knowledge of real-life orchestration experiences, learn how to use AI and Astronomer to orchestrate a data pipeline, and have a chance to add a new certification to your resume!

Sign up for this awesome opportunity to learn from the best data engineers here.

…and back to the Data Pipeline Summer challenge!

If you haven’t been following along, you can find the introduction to the challenge here and last week’s newsletter on data modeling here.

Now that we’ve built our data models to get the business the metrics it needs to see customers’ AOV and most popular recipes, how do we ensure our stakeholders get the latest data every morning? The only way to do this is to ensure our models run automatically each morning. This can be done through scheduled jobs, aka orchestration.

Orchestration is the piece of the data pipeline where the magic really happens. Data ingestion and data modeling all happen without you having to worry about running anything.

While you would typically use a tool like dbt Cloud, Prefect, Dagster, or Astronomer to orchestrate your data pipeline, we are going to write a local Python script with the help of Claude Code that allows us to run what we’ve been working on locally, with no extra tools needed.

How to prompt Claude Code to write you a Python orchestration script

❯ Hey Claude, I now have these dbt data models. I want them to refresh and run daily at 6am EST. How can I schedule/orchestrate them to run locally using the dbt build command?

You can expect Claude to call out a few different things based on your setup. For me, I asked for my pipeline to be scheduled on EST, even though my computer’s timezone is set to Arizona time. This changes how daylight savings is handled, which I’m honestly impressed that it recognized!

It also calls out that the ClickHouse server is spun off locally as needed, and may not always persist. This is a great call out when building any type of local pipeline. If using Cloud-based tools, this wouldn’t be the case, but here it matters a lot! For this reason, I suggest a way to retry the job if my computer is shut down at the normal run time.

❯ Can you give me a plan with options for this? Don’t do anything yet but I want to understand my options, the tradeoffs, and how each one would be executed.

I always recommend asking for the plan before you start executing anything. If you jump right in, not only do you not fully understand the architecture, by Claude is likely to make mistakes that you could have otherwise corrected.

Always ask for the plan and iterate on it until you are happy with what it wants to build for you.

❯ I don’t want two jobs. Let’s just do one at 6am EST. I also want this to work on any computer, not just a Mac. Is there a way to ensure the job runs if the Mac is off? Maybe it runs when the Mac comes back on if it misses that day?

Here I wanted to understand why it was saying certain things and if it was possible to find a universal job that wasn’t specific to a Mac. I offered some suggestions of an ideal state to see if it was possible to execute.

Now it’s time to give it the go to build! I recommend checking each step and validating as you go to ensure you don’t end up with something that doesn’t work and is completely out of left field.

If you give it the go to start building and the plan steps don’t match your expectations, don’t execute them! Go back to the plan mode and correct it on what you expect to see and not see here.

Even better when it suggests testing along the way for you…

Once it started creating the .plist file, I became a bit concerned. I didn’t know what this was or what it was doing, so I asked for some clarification. After the clarification, I wanted to see if I could omit it and still get what I wanted.

However, Claude clarified that it was only changing the system settings for this specific script, making me feel comfortable to move forward with this.

Again, ask and iterate as Claude executes your plan! You don’t have to send everything blindly all at once. A huge part of using AI in your work is ensuring you UNDERSTAND everything it is doing along the way. If you don’t understand it, there is no point in building it. And that is my hot take 😉

Once it’s finished building, be sure to test everything out so you aren’t surprised. We want to see the models rebuild when the script run, and ensures it starts up the ClickHouse server if it’s down at the time it is run.

And we got a failure…

Again, reiterate and let Claude Code determine the failure and ask it for a plan on how to fix it. Make sure you retest!

This is how a successful session with Claude Code should end! The code has been written, tested, and summarized. Now I can feel confidant that my local data models will run at 6am EST, or whenever I open my laptop the next day.

🏆 Challenge: Schedule your data models to run locally

Now use this same process to work with Claude Code on setting up a daily orchestration job for the data models you wrote in the last newsletter of the challenge.

Follow the same steps I illustrated above, keeping in mind specifics like your timezone, computer, and local setup. If you want, you can even try using one of the orchestration tools I mentioned, which will take you one step closer to the process of setting up a cloud-based pipeline.

Don’t forget to sign up for the FREE data engineering conference held by Astronomer, which will give you even more hands-on experience with data pipelines.

Have an awesome week!

Madison

In my current role, the data team is a full-stack analytics engineering team. Each of us works across the entire pipeline of our project, from configuring tasks in Snowflake for ingesting source data to building dashboards in ThoughtSpot.

It works well because there is no “passing of the baton”, which tends to slow things down.

Before, analytics engineers would need to communicate with data engineers to ensure the right data was available. This typically took a few weeks due to prioritization and communication issues. Then, after the analytics engineer would work on their data model, they’d have to talk with the data analyst to explain how it works. That would be a back-and-forth process to understand how the data is modeled and how it could be used to build dashboards. Because things often get lost in translation, the data analyst would then need to have the same business conversations with stakeholders that the analytics engineer already had.

In the past, I had projects that dragged out for months due to lag in communication, poor prioritization, and misunderstandings when they should have only taken a few weeks.

All of this could have been prevented if each of us worked on our own projects end to end.

AI now makes it easier than ever to bridge these skill gaps. Data analysts can build data pipelines with the help of AI, build scalable dbt projects, and write Python scripts to help automate processes. Previously, they may not have had these skills, or they may not have been as advanced as needed. Now, a basic understanding can take you quite far.

With AI making technical work less intimidating, the business knowledge of data analysts and analytics engineers become even more prized. You can’t have well-thought out, scalable solutions without understanding the in’s and out’s of the business and being able to understand your stakeholder’s problems. This is where data engineers need to lean in.

If data engineers and data analysts are both expected to know the technical skills and business communication skills because of AI, that then means they are essentially morphing into analytics engineers.

To keep up with the times and transition into a full-stack analytics engineer, data engineers need to:

• simplify complex technical solutions and concepts so that stakeholders understand what they are proposing

• understand common business metrics like AOV, CPC, LTV, GMV, etc.

• prioritize projects based on business needs and revenue potential

• communicate with stakeholders to get to the root of their problems

For data analysts to grow into a more technical full-stack analytics engineering role (and not get left in the dust by AI), they need to:

• master the basics of version control, specifically Git

• evaluate raw data for data quality issues and clean it for downstream usage

• model, document, and test data in a dbt project

• build data pipelines end-to-end including data ingestion, data modeling, and orchestration

Luckily, I’m currently teaching you how to build a data pipeline using tools like ClickHouse, dbt, and Claude Code. We are 3 weeks in and it’s not too late to join the challenge!

Start the challenge here

Data roles aren’t going anywhere, but they are evolving.

I don’t want to fear monger, but I’m not sure the data analyst role will exist in a few years. It’s not that the skillset they have isn’t important, it absolutely is. In fact, I think the skillset of talking with the business and translating their needs is actually the hardest to learn! Sorry data engineers, have fun learning that 😜

It’s just that data analysts now need to expand their skillset to also focus on the technical skills, which I think is the easier thing to learn. Start now and you’ll have time to grow into these as they become more and more important as AI evolves.

I’m curious to hear your thoughts on this.

Leave a comment

Have an awesome week!

Madison

Looking to upskill your analytics engineering skills?

• Hire me to train your data team in analytics engineering principles (reply to this email)

• Buy my ebook on the ABCs of analytics engineering

• Invest in a paid newsletter subscription to participate in the Data Pipeline Summer challenge

• Join the waitlist for my data analyst to analytics engineer course

Like most people, I started by learning modern data stack tools, looking for the best tool to solve a certain problem. While that worked at the time, I quickly realized the importance of understanding the principles outside of any tool.

Now, with AI, data modeling has become even more important as it applies business context to data, allowing analytics engineers to validate expectations BEFORE AI can use it to answer business questions.

Without data modeling, there is nuance that an AI model can’t figure out on its own, leading to data quality issues and incorrect assumptions.

Unfortunately, free datasets on the internet don’t mimic actual business problems, making data modeling hard to master. Luckily, our Data Pipeline Summer Aloha Fresh project does. The problem we’ll solve in this part’s challenge looks similar to problems I’ve faced in my full-time analytics engineering roles.

In the last part of Data Pipeline Summer, we learned how to create an AI-ready data warehouse, focusing on protecting raw data and PII while also optimizing for performance.

We loaded data for Aloha Fresh into ClickHouse and explored the raw data tables so we could model them this week.

If you missed that, be sure to check it out here so you can get your project up to speed.

What is Dimensional Data Modeling?

Dimension modeling delivers on two things- fast query performance and data that’s understandable by the business. Simplicity, speed, and usability.

With data, it’s common for projects to grow and grow in complexity until you realize you have a huge bowl of spaghetti that you’re somehow supposed to use to make quick insights. Dimensional data modeling exists to reduce this possible complexity.

To do this, the technique focuses on building tables as facts or dimensions. A row in a fact table represents a measurable event at a certain grain. A row in a dimension describes something. Together, these tables form a star schema.

Separating your data into facts puts business processes at the forefront of your data modeling strategy. With the business processes separated into their own tables, it reduces the complexity while also allowing for querying and “drilling down” across dimensions. You can think of drilling down as the aggregation of events across a dimension.

The Dimensional Data Modeling Process

Read more

Data warehouses are the source of truth of any modern data stack. It is where the data is stored and queried, making it one of the most important components.

Every tool in your data pipeline will touch your data warehouse.

With AI, everyone is using MCPs and various tooling to connect directly to your data warehouse, making security and guardrails more important than ever.

My data team has been working for months to make sure our PII is safe before folks on our team can begin to connect these tools to Snowflake.

In today’s rendition of Data Pipeline Summer, we will first learn what makes a data warehouse safe and ready for AI. We will cover:

• how to protect PII data and raw data

• how to manage users and roles

• how to connect AI tooling to your warehouse

We will end with this week’s challenge- loading the raw data for our challenge into ClickHouse.

If you missed last week’s introduction to data pipelines, be sure to give that a read before diving deeper into data warehouses.

Let’s get started!

Why do we care about AI touching our data?

Customers trust us to keep their data safe. AI is a tool that threatens that safety, especially when the data is email addresses, social security numbers, and credit card numbers. Some data is more dangerous when leaked than other, but that doesn’t mean all of it doesn’t matter.

Whenever AI touches your data, a black box is created. Sure, in some cases it tells you exactly what it’s doing with it, but others you don’t know what’s happening under the hood.

Because of this, it’s our responsibility as analytics engineers to protect this data and ensure AI only has access to the data we feel comfortable it having.

What type of warehouse is best for AI?

Analytics cloud data warehouses work just fine with AI. They allow AI to easily query structured data. In combination with a context layer (which we will talk about at the end of the series), you can query structured data as well as take advantage of unstructured data like your Notion documents and Slack messages.

Some teams may want to take advantage of storing large amounts of unstructured data outside platforms like Notion and Slack. For this type of work, it’s best to use something like a data lakehouse which stores structured and unstructured data. However, if you aren’t already using a lakehouse, you most likely don’t need one to take advantage of AI.

Protecting PII and raw data

Read more

By the end of the challenge, you will have hands-on experience building dimensional data models in dbt, working with Claude Code as a coding as…

Read more

Stakeholders start by asking us a question, and we put all the puzzle pieces together to give them the right answer. How rewarding is that!?

However, with the highs also come the lows. And by lows, I mean the validation phase.

After you’ve spent all this time building a data model with the correct grain, baking in the right dimensions, and solving data quality issues in the source data, you need to validate that what you did works as expected.

I always find data validation to be the hardest step because we don’t know what we don’t know. How do you know what to test? If I knew what to look out for from the start, I would’ve built my code around exactly that!

Luckily, AI can see things we may have missed. It can call out our blind spots and reassert our expectations.

This is why using Claude Code for validating your data models can be so powerful. You have an automated, robotic way of checking your assumptions and ensuring your code works as expected.

In this article, I’ll walk you through exactly how I work with Claude Code to validate my data models, making the process seamless and, dare I even say, enjoyable.

The problem

With building a data model from scratch comes lots of exploration of the source data. It also involves asking stakeholders a lot of questions about the business process you are modeling.

Even with this extensive research, there are things that slip through the cracks. Inevitably, there will be something you failed to explore or consider. This is why validating the source data and our data modeling logic as we work on it is so important.

You don’t want to lose hours progressing on a project that isn’t going in the right direction.

However, we can only test the pitfalls that we are aware of. For example, I’m currently building out a model that attributes purchases to certain locations on the application. Stakeholders want to know what locations are responsible for the highest conversion rates.

I know I want to attribute the most recent click on a given product before the purchase was made to the location that click occurred on. However, there’s a lot of different expectations of how this logic needs to work, and who wants to be the one to write all of the queries testing this?

Claude Code prompts

Read more

Working for myself full-time felt like a lot of pressure. I always enjoyed the side income from my data hustles and didn’t want to necessarily depend o…

Read more

Because of this, query speed will matter more than ever. You are no longer optimizing your code for your own work, but for the work of those who may not even understand data that well.

This leaves more room for error, performance issues, and long-running queries.

As analytics engineers, we need to understand query optimization so we can write code and design systems that are stable and performant.

Once again, foundational knowledge is the thing that carries us through this era of AI to ensure we can enable AI in other areas of the business.

Everyone has that one table that is so large and painful to query, that you try to avoid using it at all costs. It can sometimes be hours just to get one simple SELECT query to run.

This is because the system is reading every single row in your table to locate the piece of data that you requested. If there is no data pruning, it has no idea where to start.

It’s like paying someone to search every corner, of every room in your house to find your car keys, with no hints as to where they were last seen. This is time-consuming and expensive.

You can think of data warehouse optimization techniques like indexing, partitioning, and pruning as pointers to tell the person where to start searching. If you know you lost your keys in the kitchen, that will be quicker (and therefore cheaper) to search than the entire home.

What causes full scans?

When a full scan occurs, the system is reading every single row in your table. This is both timely and expensive because of the massive amounts of compute that this takes, especially on large tables. Not to mention, most data warehouses charge you based on the amount of data scanned (or compute).

This means, if you want to save money, your goal should always be to scan as little data as possible.

Full scans are caused by:

• no index on fields used in WHERE or JOIN clauses

• filtering on date parts or columns with functions applied

• comparing values of different data types

• overusing SELECT *

• inefficient joins

• lack of proper data modeling

I see a lot of these mistakes in AI-generated SQL, which is why avoiding these patterns is a foundational skill every analytics engineer needs to know.

Techniques to avoid full scans

To optimize your SQL queries, save money, and increase performance, you can use the following techniques:

Read more

In my pursuit of data science, Capital One recruited me to join a 6-month coding bootcamp, which led…

Read more

We can now do so much more in such little time. Data models can be built in minutes, pipelines can be debugged in seconds, and the overall pace of work is accelerating.

With all the doom and gloom out there, it can seem like AI is coming for our jobs. While AI is really good at the technical skills that once too…

Read more

But now there is a new way of working (hello AI), and semantic layers are necessary. They were previousl…

Read more

While I sadly didn’t attend Snowflake Summit this year, I still kept tabs on all the important updates coming out of the summit.

Quite frankly, there usually isn’t much that excites me. The best part about the summit is meeting the community IRL…

Read more

Stakeholders will soon turn towards AI tools for answers instead of data analysts. Because, let’s face it, it’s quicker. There’s less of a bottleneck.

They will still need a data person to help them validate me…

Read more

A data engineer at the time, I used it as an engine to apply SQL to large CSV files. This isn’t how dbt is meant to be used, but it worked for our use case.

Little did I know that th…

Read more

For better or worse, AI rapidly changes how we all work as analytics engineers.

The better:

• Small changes are quicker than ever.

• Boring tasks no longer make up 70% of our work.

• Complex …

Read more

It seems quite naive now, looking back, but you don’t know what you don’t know. Unfortunately, these lessons are typically learned the hard way.

A few weeks in, I started a S…

Read more

I’ve been dedicating most of my days for the last few months to successfully working Claude Code into my analytics engineering workflow. For most of the process, it was fun, new, and exciting.

I felt like I was flying through my work at record speed. Claude was my coding companion who helped me solve a…

Read more

This is what happened last week at work. My company had a hack week where we could solve any issue that’s been pressing us, with the help of AI, of course. The goal was to improve our AI skills and hopefully make something cool in the process.

Lately, my team has been talking about how difficult it is to validate AI code changes in the data world. Not only do you need to make sure your code performs as expected, but you also need to evaluate for backwards compatibility.

I’ve been leveraging Claude Code to do some painful refactoring in our dbt project, and while the changes it’s made seem great in theory, they end up changing the underlying data in unexpected ways. This then makes it difficult to evaluate why the underlying data changed, and if it’s ok to push the changes despite the changes.

In other words, we needed a way to validate the backwards compatibility of data changes using Claude Code.

Another analytics engineer on my team and I set out to solve this problem by creating an AI skill that automatically opens a GitHub PR and links to a validation notebook that it opens in Snowflake with context and queries needed to validate backwards compatibility.

In this post, I’m going to talk about this skill I built with a fellow analytics engineer to automatically open a Snowflake validation notebook upon AI code changes and my key findings along the way.

I’ll share learnings on:

• Principle of Least Privilege

• data governance for MCPs (users, roles, Snowflake security)

• using a CLI tool vs MCP

• iterating on and adjusting instructions in AI skills

• the final project and outcome and why it’s purpose is so important for analytics engineers in the age of AI

Principle of Least Privilege with AI and data

Read more

The truth is that it looks different every day.

We talk with stakeholders to get to the root of their needs, debug failing data pipelines, analyze produc…

Read more

Your company hasn’t dabbled much in AI yet, which means it hasn’t set up any guardrails for how you can use it. This is great for you because you can quickly set up a connection to the data warehouse to get the answers you need on the campaign!

You start by finding all raw Meta data and copying it into Claude. You then tell it that you want to see how successful the latest campaign you ran was based on click-through rate and cost. You also ask it to give you the results in a pretty dashboard you can quickly copy and paste into a presentation for your team tomorrow.

Claude spits you back some nice-looking metrics and a pretty graphic, and you are instantly relieved that you now have some data to present! This means you can act fast on creating a similar campaign next week.

When your team asks about the numbers, you thank your good friend Claude for her hard work and diligence!

This is exactly what NOT to do when it comes to using data with AI.

In this article, I’ll break down the five major pillars of safely using data and AI, including rules to follow.

♻️ If you’re enjoying this newsletter so far and think someone on your team (or even a stakeholder) could benefit from it, please share it with them!

Share

AI and PII don't mix.

If we go back to our example, we broke a major pillar in the very first step. We copied sensitive data directly into Claude.

When using data with AI, you always need to have governance top of mind. This means you cannot give AI access to sensitive data. Anything that identifies a customer can be considered PII, and therefore sensitive. Think names, phone numbers, emails, social security numbers, IP addresses, and even id’s. If you are dealing with any type of medical data, you need to be even more careful or you will be breaking HIPPA.

This is why I don’t give AI agents access to my data warehouse, import CSVs, or even copy and paste data into the chat. It is always better to be safe than sorry.

Here are some rules to follow to keep your data safe:

• only provide agents with highly aggregated metrics that are at a grain you feel 100% confident in

• ask agents for validation queries that you can then run within your data warehouse, providing them with the results of that query (whether it was successful, if it returned records, etc.)

• tag PII data so it is clear that a piece of data is sensitive

• don’t give stakeholders access to raw or untransformed data in your data warehouse

• always check with your legal and tech teams about the guardrails in place and what AI tools you can/cannot use

Garbage in, garbage out.

Just because data is available to use with AI, doesn’t mean it is accurate. In this example, we assumed the raw data we pasted was clean and ready to be consumed. If you know anything about raw data, that is almost never the case. There is a reason why staging models and data modeling are a best practices!

This is also a reason why we have jobs as data people… data is inherently messy.

Analytics engineers spend so much time speaking with the business and engineering to ensure data meets expectations. When you skip the well-thought-out data models, you are opening a lot of room for error.

AI agents should only have access to do that is properly cleaned and modeled. If you stakeholders have access to any raw or unclean data, that is a major red flag. You need to create more secure data governance practices with strict roles on what types of users have access to what.

Here are some rules I like to follow:

• if raw data hasn’t been properly cleaned, nobody should be able to use it!

• business stakeholders should only ever have access to core data models

• business stakeholders should NEVER have access to raw data (they shouldn’t even be able to see these tables!)

• business stakeholders and AI users need to have specific roles assigned to them that are highly governed, allowing the data team to control what they do and don’t have access to

AI is only as smart as the context you give it.

It’s important to remember that AI only has the context that you provide it. It knows nothing about your business, its particular challenges, how you work as a team, etc. It only knows the prompt and any resources that you give it access to.

In this example, we asked for click-through rate and cost. This could mean so many different things depending on your type of business. AI agents have no idea how we define these metrics unless we give it context or provide it with something like a semantic layer!

Claude in this example infers a metric definition and acts as if it is the truth.

When using data with AI, you need to provide accurate and extensive context in order to get the results you want. Even then, it’s imperative that you validate that those results match the conversations you’ve had with the business.

For example, I recently worked with Claude to help define some store-level metrics I was given. I didn’t have much context, but these were metrics that were already defined at different grains within my codebase. Claude was able to infer the correct definitions because of code comments and documented metrics already in my codebase. If we hadn’t had this information, Claude would have chosen generic definitions, referenced data models that seemed right, and most likely abandoned a lot of important filters that help get us the correct numbers.

If you can’t explain it, don’t ship it.

AI creates a black box. This means that you often can’t explain why it did something. If you can’t explain why a code change was made, you shouldn’t be using AI at all. You need to have a strong level of understanding of what AI agents are doing in order to use them!

If you can’t do what AI did yourself, the data team or someone else should be owning the task.

In this example, we presented metrics on the campaign without being able to explain how those metrics were calculated and what data it was using.

Explainability operates at two levels: what did the AI agent use (data resources) and why did it decide that (interpretability).

Again, if we can’t explain these two things, we shouldn’t be using AI to do it. This is what creates massive future tech debt and bugs that no human can solve.

You own the output, not AI.

AI isn’t responsible for AI outputs. YOU are responsible for what AI outputs, and in order to claim responsibility you need to be able to explain what it does. Explainability and accountability go hand in hand.

For everything you use AI to produce, you need to ask yourself if you feel comfortable taking full accountability of it. Because, if you don’t, it shouldn’t be produced.

With our above example, we didn’t take ownership of the work and instead gave ownership to Claude. Claude doesn’t own anything. The person who USED Claude owns the thing. Again, if you don’t understand the data sources, business logic, and full context behind what you are providing to your agent, you shouldn’t use AI at all.

When nobody takes ownership over their AI products, you see:

• bugs that nobody knows how to fix or wants to fix

• unanswered business questions

• sloppy code and lack of documentation

As data people we’ve worked too hard at building well-documented, clear systems to let it all go to shit over AI!

➡️ If you’re struggling to safely use AI with data in your company or individual work, reach out to me by replying to this email.

I’ve helped clients build AI-ready data warehouses, streamline their governance, and even train teams in safe data practices.

And, if you’re looking to improve your AI readiness, consider upgrading to a paid subscription where I share how I’m using AI on a daily basis in the subscriber chat and sharing monthly deep-dive data and AI tutorials.

Have a great week!

Madison