My first encounter with dbt — data build tool — left me with very weird feelings. Back then I was a pretty successful data analyst, master of SQL and wizard of Excel. When for a hundredth time I heard “dbt is amazing,” I decided to take a closer look. And dbt was weird. I mean, why would I need to install Python to run SQL? Wait, my SQL has to be compiled first? Why do I have to use terminal, instead of my IDE? All those rituals just to submit my SQL to the database? No, thanks, I don’t have time for that.

And look at me now. Using dbt daily and telling everyone that “dbt is amazing.” The tables have turned. Life has weird ways of teaching us its lessons.

And by looking at my journey I ask myself one question: what’s next? Is there a next thing that I need to develop into? Are there new ways of doing my best work and getting satisfaction from it? And I think I found it. At least for myself.

I call it Full Stack Analytics Engineering.

Full Stack What?

Before I dive into the “Full Stack” part, I wanted to stop for a moment and recognize what a great name “analytics engineering” is. It combines two worlds that previously existed in parallel.

The “Analytics” part is responsible for insights, analysis, and reporting side of data. For a long time, data analysts were doing this job. But good analytics is only possible when it stands on solid ground: functional data infrastructure and defined processes of working with data.

The “Engineering” part is responsible for the technical side of data. Great data infrastructure is only possible with proper planning and thinking about use cases and future scalability and maintainability. Engineering is a world of rules and well-defined processes.

When combined, a new role emerges. The idea of a data analyst who is not limited by data infrastructure limitations is mind opening. Same as the idea of a data engineer who can go further and be closer to the business. That’s how you engineer analytics.

It all began with a simple premise: what if we take software engineering best practices and apply them to data? Version control, testing, documentation, CI/CD. And it all worked great! We finally broke out of the vicious cycle of monstrous SQLs and broken dashboards.

The movement was so empowering that we started to explore adjacent ideas.

• We added data ingestion to our toolbelts. Thanks to the modern data stack, adding a new data source is as easy as a few clicks. No more waiting for engineering resources and broken connectors. We can have almost any data we want.

• We re-discovered data modeling practices. Well, they were never lost, it’s just that it was very hard to do them without proper tooling and foundational knowledge in the era of data volume and variety explosion. Data analysts barely learn how to model data for analytics, and data engineers are often too busy fighting failing pipelines and building new integrations.

• We added data observability tools. Finally we can see data dependencies, statistics of failing tests, and track the run time of data pipelines. We better understand how our data stack is performing and where the bottlenecks are.

• Python, bash, clouds, the list can go on and on…

Every new responsibility made our analytics more stable, predictable, and trusted. And at the same time, thanks to engineering rigor, we became more confident in our work. No more ad-hoc queries and constant metrics reconciliation.

It felt like we were settling somewhere.

But that was until AI happened.

The Emergence of AI

AI changed the rules of the game dramatically. What was once slow became fast. What was a craft is a commodity now. It feels like the whole class of “junior” titles went away.

Are we doomed? No, we are not. With a catch (as always).

History is repeating itself. The invention of the printing press made manuscript writers obsolete. Movies with sound made thousands of musician jobs disappear. ATMs eliminated bank tellers. And there are many more similar examples.

In each concrete case, the fear is not completely wrong — many jobs did indeed disappear. But every prediction of mass unemployment has consistently failed as well. And the reason is simple — people adapted and found new ways to be useful again.

With the advent of AI, will analytics engineering disappear?

No, it won’t. But it will evolve. My take — it will transform into Full Stack Analytics Engineering.

Let’s unbundle this.

AE + DE + AI = Full Stack Analytics Engineer

First of all, the core responsibilities of analytics engineers are still valuable. That is: knowing SQL, data modeling, data transformation, data quality and documentation, business intelligence.

But now we can go one level up. A lot of AEs are familiar with data engineering best practices. And again, thanks to modern data products, we can build a full data platform single-handedly: from ingestion, to storage, to transformation, to reverse ETL and BI layer. Analytics engineers now serve across both ends — backend and frontend of data, if you will.

Next, the rise of AI allowed us to be more capable. For example, I don’t remember how to properly ALTER TABLE to add a column, or how to assign a permission role in Snowflake. And I stopped googling these things: a quick prompt and a fast answer let me get on track faster and deliver work faster and more confidently.

It means a full stack analytics engineer can work across the stack and deliver a full suite of analytics to the business.

But what about other roles?

Other Roles

Does it mean that other roles, such as data analysts and data engineers, are becoming obsolete? The answer is again “no.”

This is similar to full stack software developers. This role didn’t eliminate front-end engineers and backend engineers. It rather supplemented them by supplying professionals who could support multiple ends, if additional resources are required.

Same with full stack analytics engineers. They are not replacing other jobs, but rather enhance them with additional expertise.

In most cases, analytics engineers are ideal first hires when establishing an analytics function in the team. They are capable of building analytics from the ground up and building a great foundation for future roles joining the team.

Wait a Minute, But I’m Not That

I hear this fear a lot, especially from junior specialists. Some people prefer to do one job of their liking and live a happy life. But it was the same with bank tellers and manuscript writers. If you don’t want to learn and adapt, you are out of the job market. Harsh, but true.

So, how can you ensure that your role is future-proof? For me it’s a couple of things.

First, learn the technical fundamentals. SQL and data modeling are still important, even in the era of AI. Explore adjacent fields, because every additional bit of knowledge is always valuable — though a mentor or a community beats another random YouTube tutorial.

Second, become proficient with AI. Learn agentic development, create Claude skills, play around with MCPs, set up a proper context layer. Even if we end up delegating analytics engineering to clankers, those skills will still stay with you regardless.

And lastly, level up skills that are still not available to AI. Managing projects, gathering requirements, talking to stakeholders, managing people. Learn what is valuable to businesses and try to deliver that. Personal communication and persuasive speaking are exclusive to humans (for now), so take advantage and don’t think that your technical skills outweigh that.

All in All

The change is here, no matter if you like it or not. Almost every creative field is endangered by AI and trying to find a solution.

My solution is simple — become universal and indispensable while I still can.

Interesting times.

Cheers.

Subscribe here or on LinkedIn to learn more about analytics engineering and AI!

Subscribe now

A follower sent me these questions in a DM a few weeks ago. All about moving from data analyst to analytics engineer, and everything that comes with it.

I don’t usually get the chance to answer questions like this in full, so I wrote it all out here instead of a short DM reply. If you’re an analyst thinking about the same move, or already doing this job and curious how someone else sees it, hopefully something in here is useful.

“You started as an Analyst and transitioned to Analytics Engineering. What motivated that transition?”

⚡️ I worked as a data analyst for about 8 years and in every job I had difficulties that followed me everywhere: excel reports needs to be updated manually, every data export was an exploration journey, two different dashboards had different numbers, and most annoying question was when stakeholders “are you sure this number is correct?”. For a long time I blamed myself and the quality of my work, e.g. I had to be more careful, I had to check the query, I had to perform more thorough analysis and so on. But the problem wasn’t with my skill. The problem was the inefficient process of building the analytical foundation.

Around 2020 I started looking into data engineering. And it was a field that followed a strong engineering rigor: code best practices, automations, curated datasets, proper data modeling. It felt all correct, but was overwhelming at time. The shift from data analysis to data engineering felt massive. And this is where I found out about dbt and analytics engineering.

This new title had best of both worlds: practical analytics and engineering best practices. And the more I learned about the role and the dbt framework, the more I was convinced that AE is something that I want to do: build scalable analytics and be useful to the business.

“Was there a particular project or experience that helped you get your first Analytics Engineer role?”

⚡️ Yes, there was a particular project that helped me transition to the AE role which could greatly demonstrate my desire to switch to AE full time.

At the time I was working as BI engineer working with Looker. If you are not familiar, Looker is a BI tool with the built-in semantic layer. And I’ve got a big project to work on: implement a Metrics overview dashboard for Monthly Board meetings. It was a great opportunity to structure the business metrics on a single canvas and showcase my ability to deliver quality data product.

At first I was excited about the project and quickly build a first prototype with a couple of first main KPIs. However, the more I worked on the dashboard, the more I realized that our data was a mess. Some datasets missed needed dimensions. Some metrics definitions were scattered across multiple dashboards. Some data transformations were done in the data warehouse, but some in the looker’s transformation tool. It was really hard and time consuming to find proper ways of delivering the project.

At the same time, our data platform engineers introduced dbt to our stack. So knowing about the best practices of using the correct tool for job, I’ve decided to implement all data marts in dbt, and use Looker only for visualization and complex metrics definitions.

And it worked very well! Using SQL I was able to create datasets that included all necessary dimensions and measures, add data tests and document every single column and source I used for it. Using those clean and analytics-ready datasets I was able to quickly rework the dashboard and deliver a clean and tested metrics to the Board.

“Looking back, what were the biggest skill gaps you had to bridge?”

⚡️ My two main skill gaps were data modeling and requirement gathering.

Working as data analyst I rarely thought about scalability and reusability of my datasets. Often my queries were just one-off scripts that took a lot of time, but gave almost no practical value for the future reuse. I lacked proper data modeling practice.

Another gap was a proper requirement gathering from stakeholders. When all my work looked like an endless ad-hoc cycle, there was no time for me to proper think about all edge-cases or further application of my solution. That led to re-work and back-and-forth with the business people whose real desire wasn’t satisfied with the numbers that I provided.

“What does a typical week look like for you?”

⚡️ My typical week is a combination of multiple types of work.

First type of work is individual contribution. It is a meaningful time for working on my tasks and writing code. This usually includes writing new data models for new product features and creating dashboard and semantic layer for those new models.

Second type of work is data maintenance. This includes fixing broken tests, implementing changes to the models according to changes in the business, helping with incoming data questions from stakeholders, fixing outdated dashboards and efforts to keep our data code clean and pristine.

Lastly, I spend time on meetings and mentoring. Meeting could be syncs with the team, cross-functional syncs and aligning on the roadmap. Additionally we spend some time on mentoring and self-education. For example, in my team we run BI office hours, where everyone can come and ask question about building a dashboard, metric definition or any blockers they experience with the dashboards. Sometimes we also spend time with the team going for workshops, doing internal trainings or participating in hackathons.

“Roughly, what's the tool or thing that takes up most of your time?”

⚡️ My answer to that question would be very different a year ago, but now it’s definitely an AI. With the advent of strong models and new ways of collaborating with AI, my new everyday tool is Claude Code and Codex. They now automate a big chunk of my work, leaving me with the work that cannot be automated (yet).

Also, I’m trying to learn and experiment with the latest developments in this field. Because it’s a fast growing topic that can really elevate my career, so skipping it may cost me a lot.

“Must have skills for beginners/data analyst trying to transition or looking for analytical engineering role?”

⚡️ Here are skills that I believe are important today:

• SQL — daily tool to manipulate the data. Still valuable, even in the AI era

• dbt — the most popular framework for data transformation lifecycle

• Data Modeling — learn how to build scalable and maintainable datasets

• Claude Code or Codex — hot new skills that are becoming must have

• Collaboration with stakeholders — learn to ask question, clarify requirements, follow up on projects and be a great data partner

“How deep does someone really need to know Python?”

⚡️ Although the classic AE role doesn’t require Python, I’d suggest to know at least basics of Python to be more valuable as a specialist.

A lot of companies today have blurred lines in responsibilities between data engineers, analytics engineers and data analysts. It means that sometimes you may need to read and write Python to be on the same page with the team. For example, you may need to understand how the data is gathered and transformed in Apache Airflow. Or you may need to read Pandas data transformations created by analysts in order to translate them to SQL.

Obviously, with the AI you can ask it to do the job, but you still need a way to verify the output. So it would be good to not only rely on the AI, but also be able to apply your own judgment.

“Which dbt concepts you use often?”

⚡️ Well, dbt has a lot of concepts and they all applicable in different situations. So there is no list of my favourite or most used concepts. But I like to think about this from a different angle.

dbt provides you a suit of tools that you should know and use:

• Transformations — the core feature. Learn about refs and source macros, materializations, model configs.

• Testing — almost on the same level of importance as transformations. Learn about data tests, custom SQL tests, unit testing.

• Documentation — becoming especially important in the AI age, because agents can get additional context from documented dbt models and column

• Advanced features — stuff that pushes your project to maximum efficiency: variables, macros, packages, observability

“If you interviewed me, what portfolio project would immediately make you think, “This person understands Analytics Engineering”?”

⚡️ As an interviewer, I rarely look at portfolios. But it doesn’t mean they are not useful. Because they are needed for you, not for the interviewer.

Portfolios can be good for practicing project architecture, data modeling, and data storytelling. At the interview I’m usually less interested about technical advancements you worked on, but more interested on how you tell me the story of the project. What insights you found, what data supports that, what are your recommendation to the business.

A person who understands Analytics engineering can tell a great data story without showing a single line of code.

“What are the top and most important interview questions that candidates struggle with? And what makes a candidate stand out?”

⚡️ When I ask “tell me about your past experience” many candidates start to explain the technical stacks and frameworks they worked with. But I want to hear what they achieved: were there clear business requirements, how this person clarified them, how the outcome helped the business.

Candidates who start with the business outcome are usually stand out more than those who start with the code and technical talks.

“If I have to learn skills important for this role, what would you recommend should be at the top?”

⚡️ I’d say two: data modeling and AI. Both are now essential to the role, and both can differentiate good analytics engineer from a great one.

“Lastly, what are some best practices you follow for documenting data models?”

⚡️ I have a couple of rules regarding data documentation:

1. Document everything for both humans and AI. You most definitely won’t remember why you applied a specific logic in your code unless you explicitly point that in the docs. Also, as I mentioned earlier, AI agents can extract useful context from data docs, so don’t skip that.

2. Another form of documenting models is adding useful comments. Don’t explain what the code is doing, explain the why behind it. It could be again a specific business logic, or gaps in the data.

3. Finally, always document the staging layer (input), marts layer (output) and every new business logic that gets generated in the intermediate layer.

Summary

If you're an analyst thinking about this move, most of what's holding you back probably isn't the tools. It's deciding to actually start. So just start, and probably one day I’ll learn something useful from you :)

If you liked the post, give it a heart and share with a colleagues.

Subscribe here or on LinkedIn to learn more about analytics engineering and AI!

Subscribe now

Recently, I asked my followers on LinkedIn:

“Which part of the interview process feels the hardest?”

Here are the top three answers:

• Data modeling (65%)

• Technical round (49%)

• I’m not sure what companies expect (32%)

The first two are fairly obvious. Technical expertise is the foundation of any analytics engineering interview, and data modeling is increasingly becoming a core skill (something Joe Reis has been actively advocating for).

What surprised me most, though, is that nearly 33% of candidates don’t know what companies are actually evaluating. On top of that, almost 54% said they’ve failed at least one interview.

So let’s break down how companies hire analytics engineers and how you can prepare for each step!

Update (May 2026). I released a book about preparing for AE interview!

Check it on Gumroad.

The interview process

Before diving into specific rounds, it’s important to understand one thing:

There is no single, objective measure of how well you performed in an interview.

Interview outcomes are inherently subjective and depend on the interviewer, the company, and the role. However, most interviewers assess candidates across a similar set of dimensions:

• Analytical skills — how you approach problems, talk about trade-offs, and structure your solutions

• Technical knowledge — your foundation in SQL, data modeling, and transformation tools

• Relevant experience — how closely your past work maps to their problems

• Communication skills — clarity, structure, and ability to explain ideas

• Culture fit — how well you align with the team and company values

You don’t need to be exceptional in all of these, but the stronger you are across the board, the better your chances.

To evaluate these dimensions, most companies use a multi-step interview process. Each company may have their own way of structuring the interview, but it usually includes some combination of the following four rounds:

1. Recruiter screen

2. Technical round

3. Take-home assignment

4. Stakeholder / final round

Not every company may use all four, but I’ve seen those patterns quite a few times.

The funnel reality

The funnel can be brutal. A typical analytics engineering role may receive 100+ applications:

• 20–40 candidates pass the recruiter screen

• 10–20 reach the technical and take-home stages

• 2–5 make it to the final round

• 1 receives an offer

That’s the reality of the current market.

Let’s walk through each stage and what interviewers look for.

Recruiter/HR screening

The recruiter (or HR) screen is usually a 20–30 minute call designed to answer two questions:

1. Should we keep investing time in this candidate?

2. Are we aligned on logistics (salary, conditions, etc)?

Typical questions include:

“Tell me about your background”

Focus on your current role, scope, tech stack, and why you work as an analytics engineer.

“Why do you want to work here?”

Talk about the product, domain, team, or growth opportunities.
Avoid speaking negatively about your current company — that’s a common red flag.

“Have you used X?”

Questions about SQL, dbt, orchestration tools, or analytics platforms.

Logistics

Salary expectations, start date, work authorization, location, etc.

“Do you have any questions for me?”

This matters more than people think. Good questions signal seniority and reduce the risk of mismatch.

Preparation tips:

• Prepare concise answers (1–2 minutes each)

• Avoid heavy technical jargon — recruiters are usually non-technical

• Prepare thoughtful questions about responsibilities, impact, and success criteria

Finally, make sure the role is actually a good fit. Job descriptions can be vague or overloaded with non-AE responsibilities. To align the job description with the role, check out my article about the role of analytics engineer where I talk about responsibilities and skills.

Technical round

This round evaluates your technical foundation. The better you perform here, the higher your chances of moving forward and getting the offer.

SQL skills

SQL is a hard requirement. If you fail here, you won’t pass the round.

You’ll usually see two formats:

1. Conceptual questions

• Difference between UNION and UNION ALL

• Types of joins and when to use them

• Window functions and common use cases

• How to debug a slow query

2. Live coding / whiteboarding

• You’re given a schema or dataset

• You answer business questions using SQL

• You’re expected to ask clarifying questions

You can practice with curated SQL problem sets (e.g. LeetCode SQL tracks) or even better — mock interviews with another person.

Data modeling

Data modeling questions are core to analytics engineering interviews. Interviewers want to see whether you can:

• Define business entities (users, orders, sessions, subscriptions)

• Choose the correct grain and primary keys

• Separate facts and dimensions

• Handle slowly changing attributes

• Design models that work well for BI and analytics

Typical prompts include:

• “Design an e-commerce analytics data mart”

• “Model subscriptions with upgrades, downgrades, and churn”

• “How would you build reporting tables from events, orders, and marketing spend?”

• “How would you model MRR or retention?”

dbt / transformation workflows

Even if the company doesn’t use dbt explicitly, they test the same concepts:

• Layered modeling (staging → intermediate → marts)

• Testing (uniqueness, not null, relationships, accepted values)

• Incremental models

• Documentation and its importance

• Debugging broken models or failing tests

The technical round usually lasts 1–2 hours and combines SQL, data modeling, and transformation questions.

Take-home assignment

The take-home assignment tests how well you apply your skills in practice.

You’re typically given:

• A mock dataset

• A realistic but scoped business problem

A well-designed assignment should take ~3 hours.

For example, in a gaming company, you might be asked to model telemetry data (players, sessions, matches, transactions) and help the Product team understand engagement, retention, and monetization.

Some assignments are very explicit about outputs. Others are intentionally vague — part of the test is deciding what to measure.

You’ll usually be evaluated on:

• Correctness — are your metrics and conclusions sound?

• Data modeling — structure, naming, conventions

• Data quality — tests, validation, deduplication

• Communication — README, explanations, storytelling

• Engineering judgment — maintainability and scalability

Clean, well-explained solutions almost always beat over-engineered ones.

Stakeholders/final round

This round evaluates whether you can turn messy business context into a clear analytics plan. It usually feels more like a working session with a Product Manager or Head of Data than a test.

Common themes:

• “Tell me about a project …” — it could be a real or fictional case, where you’d have to talk about aligning with business stakeholders, clarifying requirements, reasoning trade-offs and driving the adoption of the project

• Metric definition and investigation

  • “How would you investigate a metric drop?”

  • “Define a metric of success for a feature X”

• Stakeholder communication and pushback

  • “How do you handle a stakeholder who wants a metric ASAP?”

  • “How do you push back when a request doesn’t make sense?”

• Data trust, governance, and consistency

  • “How do you ensure metrics are consistent across teams?”

  • “How do you ensure data quality?”

While listening to your answers, interviewers implicitly assessing:

• Do you communicate clearly?

• Do you show clarity and ownership?

• Are you a pragmatic or perfectionist?

• How you deal with uncertainty and pressure?

• Whether you’ll escalate issues early or quietly struggle?

To sum up, in the stakeholder round, companies are checking whether you can clarify the decision, align on metric definitions, communicate tradeoffs, and end with a concrete recommendation. Red flags are jumping into SQL, being vague about definitions, and failing to tie your work to adoption and impact.

Final words

Preparing for an analytics engineering interview is a lot of work. And I hope my article helped you to understand what companies are actually looking for — and how to prepare intentionally for each stage.

Check bootcamp

Subscribe here and follow on LinkedIn to learn more about analytics engineering!

Subscribe now

A few weeks ago, I stumbled on a job posting for an Analytics Engineer position that left me frustrated. The company wanted everything at once: machine learning, statistics, data structures and algoritms — basically an entire data team packed into one person.

You’ve probably seen job posts like that too. Maybe you even wondered if you’re the one missing something. That moment pushed me to write this article.

My goal is to clearly explain what analytics engineers actually do, and what they shouldn’t be responsible for. Because when everyone, companies and candidates, understands the role, both benefit.

So let’s figure this out together!

👷 The Role of an Analytics Engineer

One way to understand what analytics engineers do is to compare them with the other roles on a modern data team.

Analytics engineering sits right in the middle, between data engineering and data analysis, and connects the technical side of data with the business side.

Let’s quickly look at each role:

As you can see, analytics engineers bridge the gap between both sides. Data engineers are often deep in the technical stack, and data analysts are often deep in the business context. Analytics engineers understand enough of both to enable fast, reliable, self-serve analytics.

Another key trait of analytics engineers is their ability to bring software engineering best practices into the analytics world:

• version control

• automated data testing

• isolated development environments

• continuous integration and delivery (CI/CD)

and so on.

This combination of engineering mindset + business collaboration gives analytics engineers a uniquely strong understanding of what the business actually needs and how data should be structured to support that.

Of course, in real life the lines between roles aren’t perfectly clean. Some analytics engineers work closer to the tech side (ingestion layer and pipelines). Others spend more time on analysis and BI tooling.

But regardless of the specifics, the core mission stays the same:

Turn raw, messy data into clean, reliable, useful datasets — so the business can focus on insights, not on hunting for data.

🛠️ Responsibilities and Skills

The responsibilities of analytics engineers are a common source of debate — mostly because not everyone fully understands where the role begins and ends. Some people confuse analytics engineers with data engineers. Others expect them to act like advanced data analysts. Many hiring managers simply don’t have a clear mental model of the role.

This misunderstanding hurts both sides.

Companies end up filtering through tons of irrelevant applications.

Candidates (especially experienced) skip roles with vague or unrealistic requirements.

So let’s fix that. Below is a clear, practical breakdown of what analytics engineers actually do and what skills they need to succeed.

#1: Building and Maintaining Data Pipelines and Transformations

This is the area where analytics engineering overlaps with traditional data engineering. But the focus is different.

Data engineers work deeper in the infrastructure: ingestion systems, pipelines, scalability, and platform reliability.

Analytics engineers take the raw data produced by those pipelines and turn it into structured, analytics-ready datasets.

To do that well, they rely on a set of core technical skills.

✅ Core Skills

• Strong SQL proficiency

• Mastery of dbt or similar transformation frameworks

• Git fundamentals (branches, pull requests, conflict resolution)

• (Nice to have) Python skills (basic understanding would suffice)

• (Nice to have) Familiarity with Airflow, Dagster, or dbt Cloud

👉 Key Responsibilities

• Build and maintain reliable transformation pipelines

• Write clean, reusable SQL code

• Ensure pipelines run on schedule and scale with data volume

• Review code from teammates and enforce best practices

#2: Designing and Modeling Data for Analytics

This is where analytics engineers start to differentiate from other roles.

Analytics engineers design datasets that power dashboards, decision-making, and self-serve analysis. If the data model is messy, unclear, or inconsistent, everything downstream becomes harder.

On a similar note, not only those datasets should be helpful and easy to use, but also they need to be performant and cost efficient. Business stakeholders won’t wait for the dashboard for 10 minutes and spend fortune on that.

✅ Core Skills

• Understanding dimensional modeling

• Knowing when to normalize or denormalize

• Familiarity with Slowly Changing Dimensions

• Modeling for BI tools and semantic layers

• Optimizing complex SQL queries

• Implementing incremental and scalable models

👉 Key Responsibilities

• Design clean, scalable, analytics-ready datasets

• Create unified metric definitions and ensure consistency

• Anticipate downstream BI needs and user behavior

• Balance performance, maintainability, and cost

#3: Ensuring Data Quality and Observability

Even the best data models are useless if they’re not trustworthy.

To build confidence, analytics engineers put strong data quality and observability practices in place.

This ensures that the datasets the business relies on are accurate, fresh, and dependable.

✅ Core Skills

• Writing automated data tests

• Monitoring data freshness and detecting anomalies

• Implementing pipeline monitoring and alerting

• Understanding lineage and dependencies

• Documenting logic, assumptions, and model behavior

👉 Key Responsibilities

• Ensure data accuracy, completeness, and freshness

• Implement tests to catch failures early

• Monitor pipelines and resolve incidents

• Investigate unusual spikes, drops, or inconsistencies

• Maintain documentation to support transparency and trust

#4: Collaborating with Business Stakeholders

This is where analytics engineering truly comes to life.

You can write beautiful SQL and build elegant models, but if they don’t solve real business problems — they don’t create value.

That’s why great analytics engineers are strong communicators who understand the “why” behind the data and build for that purpose.

✅ Core Skills

• Explaining complex data concepts in simple language

• Translating business logic into metrics and models

• Understanding domains like product, marketing, or finance

• Gathering requirements and asking smart clarifying questions

• Setting expectations, negotiating trade-offs, and prioritizing work

👉 Key Responsibilities

• Identify key decisions and define how data supports them

• Translate vague business questions into clear data solutions

• Run requirement-gathering sessions with stakeholders

• Communicate changes, limitations, and trade-offs

• Bridge technical data teams with non-technical business teams

🚫 Out of Scope

Now that we’ve defined the core responsibilities of analytics engineers, it’s equally important to clarify what isn’t part of the role.

This matters because analytics engineering naturally overlaps with both data engineering and data analysis. Some companies misunderstand this overlap and end up assigning work that belongs to a different discipline entirely. Others simply create unrealistic job descriptions because they’re not sure where the boundaries should be.

Let’s set those boundaries clearly.

Below is a list — in no particular order — of skills and responsibilities that are outside the scope ❌ of analytics engineering and should not be expected from this role :

• Ad-hoc analysis for business teams → Data Analysts

• Algorithms and data structures → Data Engineers / Software Engineers

• Advanced statistics and probability → Data Scientists

• Building machine learning models → Data Scientists

• Setting up Kafka or ingestion infrastructure → Data Engineers

• Managing Spark clusters → Data Engineers

• Owning data lake architectures → Data Engineers / Data Architects

• Writing backend microservices → Software Engineers / Data Engineers

• Developing custom Python ETL frameworks → Data Engineers

• Managing DevOps (Terraform, Kubernetes) → DataOps / Platform Engineers

These tasks might seem related because they all touch the broader data ecosystem — but they do not fall under analytics engineering. Mixing these responsibilities into an AE role leads to confusion, misalignment, and frustration on both sides.

🏁 Final Words

Analytics engineering is a modern, dynamic, and genuinely exciting role. It blends technical skill with business understanding, and when done well, it can help a company scale its analytics capabilities to an entirely new level 🚀.

But that doesn’t mean analytics engineers should carry the entire data function on their shoulders. Clear boundaries and realistic expectations are what allow this role to deliver its full impact.

I hope this article becomes a useful guide for both companies and candidates — helping everyone stay aligned on what analytics engineering is and what it isn’t.

And if you think I missed something or pushed too hard on the definition of the role… let me know. Let’s shape the boundaries together 😄

P.S. Subscribe here and on LinkedIn to learn more about analytics engineering!

Subscribe now

In the previous post, I discussed the first two important principles of robust data modeling:

• 📈 scalability, and

• ♻️ reusability.

In this post, I'll explore the other two critical principles:

• ⚡️ performance, and

• ✅ data quality.

Let's dive in!

⚡️ Performance

In analytical data modeling, performance can be measured by:

• how quickly the model can be built

• how rapidly business users can retrieve results from the data model

• the costs associated with building and querying the model

In cloud data warehouses speed and cost are deeply interconnected. Faster model building leads to lower costs, while longer query processing times result in higher expenses.

Simply put: optimize speed and you'll optimize costs💲.

But performance matters not only because it saves money. Business stakeholders need dashboards that load quickly, no one will wait hours for the dashboard to finish. So the key to optimizing performance is finding the right balance between speed and costs.

Let's explore some approaches to achieve this balance.

🛠️ Choose suitable materialization

The easiest way to optimize performance is to choose the correct data materialization.

Usually data models are materialized as database views or tables. Tables, in turn, can be materialized from scratch each run (full refresh) or aggregated incrementally.

Database views are useful when data models consist of light transformations and don't require joins or aggregations. Examples of light transformations are:

• CASE WHEN statements

• formulas on columns

• data type casting

• columns renaming

Most database engines can apply such transformations on the fly, so using table materialization could be a waste of time and compute. For example, dbt recommends making models in the staging layer as views because they don't contain joins or aggregations (GROUP BY).

Data models should be materialized as tables in two cases:

1. The model is a user-facing table that requires the best performance.

2. The model is heavy to calculate and is being used in multiple downstream models. All models in Marts or Gold layer should be materialized as tables to give the quickest response time for end-users.

Some intermediate models might contain complex business logic that requires multiple joins and are slow to query, so it's better to materialize such models as tables as well. This is especially beneficial if multiple downstream models use them in their calculations.

My general rule of thumb 👍 is materializing data marts as tables, the rest as views. And if views are too slow, switch them to tables as well.

As I mentioned earlier, tables can be created in two ways: by re-creating from scratch each run or by loading incrementally, processing only the new data.

Both approaches have their pros and cons.

Re-creating a table from scratch each run (i.e., full refresh) is definitely easier because you don't have to deal with schema migration or any additional DDL commands. But this method might be expensive, especially for big data sources.

Loading tables incrementally, on the other hand, allows you to process only the new data and save on processing time and money. On the flip side, you'll have to deal with occasional schema migrations (add/remove columns) and backfills. Here, my general tip is to start with the full-refresh approach and switch to incremental if the data size is big enough that you actually start having benefits from the more complex setup.

Action points:

❓ Check your data models to see if you can improve performance by applying the correct materialization strategy

❌ If all your data models are materialized as either only tables or only views, your model performance is likely suboptimal

✅ Ensure that models with light transformations are views, user-facing models are tables, and large tables are built incrementally. dbt Labs recommends this approach in their famous “How we structure our dbt projects” guide

🕵️‍♂️ Add Data Observability

The second important aspect is implementing robust data observability.

Data observability is the practice of monitoring and understanding your data pipelines. It helps with:

• tracking model performance, for example run times

• data lineage, to identify dependencies between models

• monitoring data quality

When a model moves to production, its runtime can silently increase due to data growth or project complexity. Having reports that show build times for each data model helps prevent unexpected cost increase.

Another problem data observability can help with is so-called "modelnecks" — bottlenecks where one long-running model slows down the entire pipeline because dependent models must wait for it to finish. By analyzing model build times alongside data lineage helps you spot and proactively resolve these modelnecks.

You can improve a slow-running model by selecting the right materialization strategy, implementing incremental loading, or refactoring it into smaller models to increase the throughput.

The final component of effective data observability is the ability to logically break down the DAG into smaller, more manageable parts. Many data lineage tools allow you to add metadata to models, such as tags, helping you identify critical pipeline components to prioritize during optimization. For example, dbt supports adding tags to models and selecting models based on these tags when visualizing or building models. This feature makes pipeline management significantly more efficient.

Action points:

❓ Check which metrics you track regarding model performance: run times, number of failures, and time in queue

❌ Inability to see these metrics over time may lead to silent cost increases and performance degradation

✅ Choose from different options to implement data observability. For dbt projects, you can start with dbt Cloud (paid) or install open-source tools like Elementary

🏎️ Utilize database optimizations

Last but not least is utilizing all possible technical optimizations available in your database. This tip alone could fill an entire book, so I'll provide some general guidance instead.

First, understand that different database types require different optimization techniques. Postgres, BigQuery, and Apache Spark have fundamentally different architectures, so you need to learn the specific advantages and limitations of your particular database or processing engine.

At the same time, some tips are applicable across a range of technologies. For example, all columnar data warehouses (BigQuery, Snowflake, Redshift) share common optimization techniques, such as:

• utilizing partitioning and clustering

• applying data pruning as early as possible

• avoiding ORDER BY when unnecessary

• preferring window functions over self-joins

• and so on

Therefore, the best approach is to begin by researching optimization techniques specific to your technology stack.

If you use Snowflake data platform, I recommend checking out the SELECT blog. This company specializes in optimizing Snowflake costs. They share valuable advice on how to tune data warehouse performance using not only the techniques mentioned above, but also vendor-specific settings like thread count or warehouse size.

Finally, an important tip is to learn how to read execution plans using the EXPLAIN keyword. This SQL command shows how your query will be executed in the database, revealing the actual operations performed.

Execution plans are very valuable for finding bottlenecks in SQL queries. I particularly look for:

• full table scans instead of filtered views (typically caused by missing or misplaced WHERE clauses)

• cardinality explosions (usually resulting from incorrect or poorly placed JOINs)

Some databases, like Snowflake, provide a user-friendly visualization of execution plans, making it easier to debug and identify performance bottlenecks.

Action points:

❓ Learn more about the underlying technology of your data stack

❌ Using only generic or random optimization tips cannot guarantee optimal performance for your specific technology

✅ Look for specialists or companies that focus on data stack optimizations. Excellent examples are this blog post about Snowflake or this post about Databricks optimization

✅ Data quality

The last crucial principle of building robust data models is maintaining the highest standards of data quality.

Data quality measures how well data meets consumer expectations. For example, consumers of your data models expect the data to be accurate, consistent, reliable, and much more. Higher quality data builds more trust and leads to greater use in business decisions.

Let's explore how to achieve a great data quality of your models.

🔁 Apply end-to-end data testing

A naive approach to data quality is to measure the characteristics mentioned above (like accuracy and consistency) only for business-facing data models. This means applying periodic testing just to tables that end users interact with. However, data quality measures can be applied to a much wider range of use cases throughout the data pipeline.

If we carefully examine how data flows into the data warehouse and is subsequently transformed into data models, we can distinguish several stages where data quality can be checked:

1. 🌱 External data used to build analytical models should be monitored for unexpected values, incompleteness, and timeliness (freshness)

2. 👷 During development of new models or modification of existing ones, data tests can be run on the final models to signal if the new or modified code breaks any assumptions about the data

3. 🚢 Before deploying new changes to production, automated tests can assure us that they won't break any existing reports

4. ⏱️ Finally, periodic tests on production tables can tell us if newly added data still meets our quality standards

This approach resembles the "Shift Left" approach I explained in the previous article. The earlier you catch a problem, the sooner you can fix it and avoid building production data models with errors.

Different types of tests should be applied to different stages.

For example, dbt freshness tests can be applied to source tables to ensure they don't have stale data. Also, data contracts can be applied to make sure that our assumptions about the incoming data still hold. Data contracts can check that the number of columns hasn't changed, or that data types remain the same as before.

During development, data tests and unit tests can greatly help in delivering high quality datasets. Data tests are used to check the assumptions about the data, for example that a column is not empty, or it contains only expected values. Unit tests are useful to check model transformation logic — by using mock inputs and expected outputs they can signal when changes break the logic of data models.

Periodic tests in Production typically include data tests and freshness tests. Fresh data sources provide confirmation that new data has arrived and models can be rebuilt. Data tests ensure that the new data remains correct and contains expected values.

Action points:

❓ Inspect different stages of the data flow and examine what kinds of tests are currently applied

❌ If you don't have any tests, implement them throughout your entire data pipeline, from ingestion to final data marts

✅ If you're using dbt, you can immediately start using data tests, unit tests, and freshness tests. For other stacks, try Soda tests or Great Expectations

☯️ Balance coverage and redundancy

While it may seem logical to test every possible model in your data pipeline, this approach is often counterproductive.

First, covering all data models with tests can significantly increase testing time and incur additional costs.

Second, many data tests will duplicate each other because some columns are simply carried from model to model without changes.

⚖️ You need to balance tests coverage with redundancy — on one hand you want good coverage, but not at the expense of longer waiting times and higher costs.

To streamline this process, you should implement testing conventions. These conventions should clearly define what to test, when to test it, and how frequently tests should be run.

Here are the testing conventions I use in my work:

• Always add a primary key test to your models. If the model doesn't have a primary key, generate a surrogate key and test it.

• Test columns as early in the pipeline as possible (applying the Shift Left principle). In dbt projects, this is typically in the staging layer.

• Define tests based on column category:

  • Primary keys should have not_null and uniqueness tests

  • Categorical columns should have accepted values tests

  • Booleans should be checked for accepted values (ensuring they contain BOOLEAN type, not STRING like "true")

  • Numbers should be checked according to business logic (e.g., order_value should always be positive)

  • And so on

• In downstream models, only test columns that were modified or newly introduced. Testing columns carried over unchanged is redundant.

• Redundancy is acceptable in the final layer with business-related models. You can add primary key tests and additional checks to all important columns, like testing for missing values, expected values, and business logic (for example, shipping date should always be later than the order date).

Action points:

❓ Check if your team has any conventions regarding data testing

❌ If you don't have any, it can mean that your strategy is not optimal, you are missing some tests cases or vice versa apply too many tests

✅ Start with the list of conventions I provided above and then extend and adapt to your needs. Try dbt-project-evaluator package that can measure tests coverage and some other useful metrics about the projects

🧠 Build with testing in mind

In software engineering, a popular approach to developing software is called test-driven development (TDD) involves writing tests before writing the code. Once tests are established, developers must create software that meets all quality standards to pass these tests.

This approach isn't always suitable for data models. We often design a model, begin implementation, adjust the initial design based on new information, and iterate again. In this context, writing tests first can be challenging.

That's why I suggest a more flexible alternative which I call test-informed development.

It's based on several principles:

1. When building models, consider which tests can be applied to them

2. Add necessary tests shortly after completing development, including those that might be valuable in the future

3. Follow a build-test-adjust-retest cycle to ensure your assumptions about the data remain valid after changing the code of the model

I'm advocating for intentional testing, where models are designed with testability in mind. This approach doesn't strictly follow TDD, but it prioritizes quality and correctness during development.

Classic TDD is technically possible when utilizing unit testing. For example, dbt 1.8 introduced unit tests as first-class citizens, allowing analytics engineers to test SQL logic without using any underlying data.

However, technical possibility doesn't guarantee universal adoption. Rather than requiring that everyone should write unit tests, it's better to embrace a more flexible approach where model developers naturally incorporate testing considerations into their process.

After building up a solid collection of tests in your project, you'll gain greater confidence when modifying logic, as comprehensive test coverage can identify bugs and potential issues before code reaches production. If you later decide to implement unit testing, refactoring large models becomes significantly easier because you can test not only the model's output but also its underlying logic directly.

Action points:

❓ Evaluate how your team develops and merges data models to Production

❌ Lack of testing guidelines and deploying to Production without proper tests indicates poor engineering practices that can lead to serious failures

✅ Educate your team about data testing importance, add "I have run data tests" to your PR checklist, and incorporate tests into your CI pipeline

Summary

This is the final post of “The Core Principles of Robust Data Modeling” a mini-series where I covered the last two pronciples. Check the first post here.

If you liked it, give it a heart and share with a colleagues.

Subscribe here or on LinkedIn to learn more about data modeling and analytics engineering!

Subscribe now

One of the main values analytics engineers provide is building a robust system of data models that supports the decision making process of the business.

This system holds on four key principles:

1. 📈 scalability

2. ♻️ reusability

3. ⚡️ performance

4. ✅ data quality

By focusing on all four, analytics engineers can build a truly remarkable analytical system that supports business stakeholders and data analysts, leading to faster insights and better decisions.

These four principles are interconnected. Progress on one will inevitably improve the others, as the boundaries between them are often blurry.

In this post series, I'll share practical tips on how to achieve each aspect of robust data modeling.

📈 Scalability

In analytics engineering, scalability is about handling changes with greater speed and confidence. This means data models should be designed for easy modification and extension.

In data world, requirements constantly evolve — new metrics, new dimensions, constantly changing business logic. The ability to quickly implement changes defines how scalable is your project.

Scalability also involves handling increased data volume, but we'll address that when discussing the performance aspects of analytical systems.

Several best practices can help achieve scalability.

🍔 Modeling layers abstraction

The first step in achieving scalability is organizing data pipeline’s assets into a system.

When transformation logic is just a pile of SQL and Python scripts hooked together, it's easy to get lost and make incorrect decisions about how to grow the system. There should be a clear pattern.

A classic example of such a pattern is the Medallion Architecture.

It suggests organizing all the data in your data warehouse or data lake into three layers (Bronze, Silver, and Gold) to progressively improve data models:

• Bronze layer represents raw data ingested from external sources

• Silver layer contains cleaned and validated data transformed from the Bronze layer, refined and cleaned of inconsistencies and errors

• Gold layer contains curated and enriched data ready to be used by analysts, data scientists, and Business Intelligence tools; usually denormalized and structured for efficient querying

If you use dbt framework, another popular option is the modeling layers approach from dbt Labs.

It suggests organizing data models into three basic layers:

• Staging — to perform basic clean-up and alignment of raw data

• Intermediate — to build reusable building blocks that contain business logic

• Marts — for final and polished data marts ready for analysis

Moreover, your models can be easily extended beyond these basic layers. For example, you can have reverse ETL layer, semantic models, and reporting layers. For inspiration, check how Gitlab structures their data warehouse layers.

Both approaches (medallion and modeling layers) share the same core idea: models should transform data from its raw state into a usable analytical model through progressive enhancements.

By logically organizing data models, we bring order to the project and ensure that changes can be implemented faster.

Action points:

❓ Look at your analytics project and try to figure out if you are utilizing layers to gradually transform raw data into final analytcal tables

❌ If you don't see any notion of layering, think about how you can logically split all your data assets into layers to promote standardization

✅ If you're already using medallion architecture or modeling layers - you're on the right track!

⭐ Data modeling methodology

Although the term "data modeling" means different things depending on context, in this post we're specifically discussing data modeling for analytics purposes.

In the data world, data modeling methodology defines how data models are designed logically and subsequently implemented physically.

Several popular methodologies exist, including Kimball's modeling, Inmon's modeling, and Data Vault, and maybe some others.

In real projects, teams rarely use a pure implementation of any single approach. More commonly, they adopt a hybrid that borrows effective patterns from multiple methodologies.

For example, a popular approach is applying Kimball's dimensional modeling to business entities, like orders, users, or bookings. This helps create specific data models that serve business needs. Patterns such as “star schema” or “snowflake schema” help enforcing consistent physical design.

As a practical guidance on applying dimensional modeling in your analytics, I highly recommend the BEAM framework (Business Event Analysis & Modeling).

This framework outlines how analytics engineers should iteratively build data models by:

• collaborating with business stakeholders

• capturing essential details of future models

• building models that are both useful and well-structured

Here is an awesome video that explains implementation of BEAM in more details.

Integrating data modeling techniques with the modeling layers creates a more scalable architecture with exceptional clarity.

When you need to change logic or add a new data model, you'll know exactly where to make changes. New models typically go into the exposed layer (Gold or Marts) and then work backward through intermediate layers to raw data. Existing models are easily adjustable because dimensional modeling promotes modularity and clear relationships between entities.

Action points:

❓ Review your data models, especially the final layer that contains business-facing tables

❌ If you see a set of disconnected tables that don't have clear relationships or contain partially duplicated logic — rethink your approach, try Model-storming exercise from the BEAM framework to see if you can unify your models

✅ If you see well-organized marts that are easy to join and work with — nicely done!

🧱 Naming and coding conventions

This simple technique brings significant order to all analytics projects.

It's about consistently naming things and using the same coding conventions throughout your codebase.

This reduces cognitive load, allowing you to focus more on business logic more than worrying about file naming.

The same principle applies to files organization. Files that properly arranged into layers naturally indicate where to create new models or locate existing ones.

Organized systems tend to stay organized.

Additionally, properly organized files are easier to maintain and simplify the onboarding process for new team members.

Take inspiration from the same guide from dbt Labs about modeling layers:

• Base layers sit at the top of your project (e.g., /staging, /intermediates, /marts)

• Within each layer, files are organized either by source for staging models (e.g., facebook, postgres, salesforce) or by business domain in intermediate or marts layers (e.g., product, finance, marketing, core)

• Files follow specific naming conventions, such as prefixing with stg_ in the staging layer, int_ for intermediates, or fct_/dim_ for marts

Action points:

❓ Review existing coding and naming conventions for your projects

❌ If you don't have any — it's time to design them! Sit with the team and discuss how you can standardize the naming and coding conventions

✅ Already having strong conventions? Try utilizing tools like SQLFluff, Ruff or other specialized packages to check your existing codebase and enforce the standards within the team!

♻️ Reusability

The second vital pillar of a robust data modeling system is reusability.

Reusability enables you to build new models on top of existing ones, eliminating the need to grow the number of models. You leverage code that has already been built and tested. This approach allows you to write less new code overall.

Here are a few approaches to achieve reusability in your project.

📦 Build Once, Use everywhere

The first principle is very straightforward — when creating a data model, think ahead about potential usage of that model beyond your current scope.

Quite often you might find yourself needing the same functionality over and over again. So instead of copy-pasting the code, you could implement a model that can be reused many times.

The main principle of reusable models is modularity and single-purpose responsibility.

General rule: one model = one responsibility.

Don't build overly complicated models that encapsulate a lot of logic. Such models are over-engineered and may be hard to reuse due to complexity or performance issues.

A particular red flag is having over-loaded intermediate models. Intermediate models need to be small building blocks, not final tables with all possible business logic.

Not only models can be modularized, but pieces of code as well. For example, in dbt you can use Jinja macros to create snippets with repeated logic. Things like currency or timestamp conversions, complex CASE statements, etc. can be encapsulated and reused across the project.

Action points:

❓ Search for data models in your project that are different yet have a lot of overlapping logic

❌ It’s probably hard to refactor all repeated models at once, so first good step is just creating a backlog of such models

✅ Start tackling the backlog as a second step, gradually cleaning up your DAG and making it more straightforward

😄 Fun fact: we used a metric called "number of screenshots required to capture a DAG" to measure how well we improved our models. We went from 8 to 3 screenshots after six months of work!

⬅️ Shift Left

The second popular approach to increase reusability is called "shift left."

This means implementing changes as early as possible in the data pipeline so they can be easily reused by all downstream models.

For example, if your “users” table contains first_name and last_name fields, and you frequently concatenate them to get the full_name in your data marts, this transformation is a good candidate to "shift left" to the staging model, making it available for everyone.

While this may sound trivial, I often find myself reinventing the same formula or dimension repeatedly in data marts when the better solution would be to push this calculation as far left as possible in the DAG.

Great examples of the “shifted left” pieces:

• Mathematical formulas (e.g., calculating tax rates)

• Timestamp conversions and unification

• Implementation of CASE statements logic

• Logic that requires JOINs (usually in intermediate models)

Another advantage of this approach is traceability. When all downstream processes use the same formula from upstream source, it becomes much easier to troubleshoot and backtrack changes to their source.

Action points:

❓ When working with data marts, start noticing similar (derived) columns that span across multiple models and check

❌ Are they coming from a single calculation upstream? If not, shift them left!

✅ In many cases, reusable logic can be shifted up to the staging layer, significantly reducing the number of copy-pasted pieces in downstream models

📝 Document your models

Including documentation in the Reusability section might seem unexpected, but stay with me on this one.

Engineers often create redundant models simply because they're unaware of existing models that could meet their needs. And this is where a comprehensive documentation can prevent creating redundancy by making existing models more discoverable and more understandable.

Effective documentation clearly states the purpose of the model, applied filters, limitations, and edge cases. Also, by reusing existing models you benefit from the experience of the model’s developer, who probably handled edge cases and unusual data behaviors.

Of course, if an existing model only meets 10-20% of your requirements, creating a new one is preferable. After creating a new model, document it and maybe explain why you chose creating a new model instead of extending an existing model.

Well-documented models, combined with proper organization (using the layers and naming conventions discussed earlier), can significantly reduce redundant code and enhance reusability across your analytics ecosystem.

Action points:

❓ Review the documentation coverage and its quality: do all models have descriptions that clearly explain their purpose?

❌ If documentation is sparse or unclear, prioritize documenting the most frequently used models first

✅ Consider implementing a documentation conventions, like including model's purpose, limitations, and usage examples

To be continued

In the next post I'll talk about the last two principles of robust data modeling: performance and data quality.

Make sure to subscribe here or on LinkedIn to not miss the second part!

Subscribe now

Have you ever encountered a mysterious error while working on a dbt project and couldn't figure out its source? I've been there too.

Some errors are easy to fix — usually caused by that one tiny change you just made. But others? Sometimes you may face puzzling issues that seem to originate from somewhere outside of your modifications (though I suspect they rarely do 😉).

Let’s break down how to solve them before they drive you crazy.

Types of errors in dbt (and how to outsmart them)

First of all, there are basically only four four types of errors in dbt. Let's explore them.

🚀 Runtime Errors – The "Something is Off" Error

When does it happen? When your project setup is incorrect.

🔧 Fix it by:

• Running dbt debug to check for general issues with the project, like missing profiles or broken database connection.

• Ensuring you're inside the correct dbt project folder (this is where your dbt_project.yml file is).

• Checking your profiles.yml file for the correct profile name.

• Verifying database permissions.

Usually you encounter these when you're setting up the project for the first time. But once everything is connected, you likely won't see these anymore.

🔄 Database Errors – When SQL Says ‘Nope’

When does it happen? When dbt tries to execute SQL that the database doesn’t like.

🔧 How to fix it:

• Try to compile the model with dbt compile command and execute the result in database directly. The command will output the compiled code directly to your terminal!

• Alternatively, look inside the /target folder for compiled SQL and test it in your database.

• Run your commands with the --debug flag for extra details (more on that later in the post).

In my opinion, these errors happen most often. Luckily, they are also the easiest to fix, since they are originating from SQL code you write.

⚠️ Compilation Errors – Jinja and YAML Gotchas

When does it happen? When dbt struggles to parse your Jinja or YAML code

🔧 Fix it by:

• Checking for invalid Jinja syntax. Maybe you forgot to close a curly bracket or made a type in macro name.

• Ensuring referenced models actually exist.

• Validating that your YAML files are correctly formatted. YAML syntax is also sensitive to margins.

Happens from time to time, but usually dbt can tell you exactly where the issue is.

🔗 Dependency Errors – The ‘Circular Nightmare’

When does it happen? When your models reference each other in a loop or have broken dependencies.

🔧 How to fix it:

• Hunt for circular dependencies and break the loop. If the loop seems unbreakable, maybe you could introduce an intermediate model to get out.

• If a model needs to reference itself, use the {{ this }} macro, not the ref()

Easiest problem to understand, but sometimes the hardest to resolve.

Debugging like a pro

1️⃣ Use the --debug Flag

dbt hides a lot of details in normal runs, but adding --debug unlocks more insights:

dbt --debug run

Now you can see exactly what dbt is doing behind the scenes. This also works for test and build commands.

2️⃣ Debugging Macros

Macros can feel like a black box, but here’s how to crack them open:

• Compile the macro separately to see what SQL it generates:

dbt compile --inline "{{ macro_name(params) }}"

• Use log() statements to print out variables and debug step by step:

{{ log("Reached here", info=true) }}
{{ log(variable_name, info=true) }}

• Use the built-in Jinja debug() macro like a breakpoint. This will open an interactive iPython debugger where you can inspect variables.

{% my_macro() %} 

... your macro code
{{ debug() }}

{% endmacro %}

The Future of Debugging in dbt

With dbt’s recent acquisition of SDF Labs, a new SQL parser is on the way. This means syntax errors could be caught before even running queries — exciting times ahead! But until that feature arrives, these debugging techniques should keep you sane.

What’s the weirdest dbt error you’ve encountered? Let me know about your horror stories in the comments! 👻

Subscribe now

Apache Airflow is a very popular orchestration tool in the data engineering world. One question that often arises in many teams is: "Is it possible to run dbt pipelines with Airflow?" Of course it is, and I'm going to show you how to implement this in practice.

Since Airflow is an open-source tool, there's no single "correct" way of integrating the two tools. Different teams may integrate them differently and even create their own integrations to serve specific requirements.

In this post, I'll show you two ways (+ one bonus way) of integrating them together. One way is simple and straightforward. The other is more custom and requires an additional package to be installed in your Airflow environment. Let's get started!

All code for his post you can find in this repository!

Option 1: Naive approach

The naive approach is straightforward and doesn't require external packages to be installed. However, since dbt itself is a Python package, you'll need to add it to your Airflow dependencies.

Once that's done, you can simply create a DAG that uses BashOperator to trigger dbt commands. Here's a sample code that runs three dbt commands: seed, run, and test:

There are a few important points to pay attention to.

First, ensure you specify the correct paths to your dbt project folder and the folder containing dbt profiles. Since there's no standard location for a dbt project, it could be in any subfolder of your Airflow repository. By using the flags --project-dir and --profiles-dir, we ensure that the dbt CLI uses the correct path to the project.

Second, as you've likely figured out from the previous point, you'll need to commit profiles.yml to the repository. Don't worry — there's no need to store your credentials there. You can simply read environment variables to make it work.

With such simple setup you get a simple way to run dbt models from Airflow:

However, there are several fundamental flaws with this option.

The main drawback is the monolithic architecture of the pipeline. You can't see the same DAG of models in Airflow as you do in dbt documentation. This means that if one model or test fails, you have to re-run the entire pipeline, as there's no way to pick up from the failure point.

Additionally, installing dbt alongside Airflow's main dependencies may lead to conflicts.

The need to commit profiles.yml to the repository also adds complexity. Wouldn't it be great to simply reuse connections from Airflow to run the dbt pipeline?

That's why the folks at Astronomer implemented a package called astronomer-cosmos that can tackle these limitations.

Option 2: Astronomer-cosmos package

The Astronomer-cosmos package for Apache Airflow enables you to run dbt Core projects as Airflow DAGs and Task Groups.

According to its developers, this approach offers several benefits:

• Run dbt projects against Airflow connections instead of dbt profiles

• Ability to install dbt in a separate virtual environment to avoid dependency conflicts with Airflow

• Simulate dbt build command by running tests immediately after a model is done to catch issues early

• Use Airflow’s sensors to immediately run models after upstream ingestion

• Turn each dbt model into a task/task group and configure retries, alerting, etc.

Here is a very simple code that can get you started:

As you can see, you'll need to provide a few configuration options.

Similar to the previous example, configure a full path to the dbt project. Additionally, in the execution_config, you should add a path to the dbt executable (in case it's in a separate virtual environment).

The profile_config variable contains settings for dbt profiles. Here, you can either use a path to profiles.yml (similar to the previous example) or use a connection from Airflow, in this case to Snowflake.

The DbtDag is a wrapper class that splits each model into a separate task, like so:

As you can see in the screenshot, each model task is followed by another task with tests for that model. This approach provides behavior similar to the dbt build command.

You can configure the DAG in many ways.

Using the DbtTaskGroup class, you can integrate dbt as a step within a more complex DAG. The render_config option allows you to make node selections for models, seeds, tags, and more. You can find all these examples in the demo repository: https://github.com/astronomer/cosmos-demo/tree/main

Bonus: using Airflow with dbt Cloud

If you're using dbt Cloud and want to leverage Airflow's scheduling capabilities, you can do so with the help of an official package. This package enables you to submit dbt jobs, check job results, and download artifacts. By combining Airflow with dbt Cloud, you can create a highly customized data pipeline tailored to your specific use case.

The package can be installed as:

pip install apache-airflow-providers-dbt-cloud[http]

To connect to dbt Cloud, you need to generate a service token. This token, along with your account ID and URL (found in Settings → Account), can be stored in Airflow’s connections. Alternatively, provide credentials as an environment variable:

export AIRFLOW_CONN_DBT_CLOUD_DEFAULT='dbt-cloud://account_id:secret_token@my-urlgetdbt.com'

Here is a simple DAG that can trigger dbt job from dbt Cloud:

This DAG simply triggers a dbt job configured in dbt Cloud environment. It also is possible to make this task DbtCloudRunJobOperator asyncronous by providing wait_for_termination=False argument.

If you liked this issue please subscribe and share it with your colleagues, this greatly helps me developing this newsletter!

See you next time 👋

A common question for data professionals is, "When was the data/dashboard last updated?" The answer often involves checking the created_at or updated_at columns in your data models for the latest timestamp. However, this might not be entirely accurate, as your models are only as fresh as your data source is.

Another critical aspect of data modeling is ensuring the quality of your source data. The saying "garbage in, garbage out" is relevant here. Even flawless SQL models won't compensate for poor incoming data quality.

Therefore, it's crucial to invest time in setting up proper data quality testing for your data sources. This will help reduce errors in your final dashboards. Let's explore how to accomplish this with dbt.

Freshness tests

A freshness test is a simple test you can perform on your data sources. As the name implies, it checks the freshness of the data sources, alerting you about stale or outdated data.

The method dbt uses to check freshness is simple. The first step is to check the timestamp of the last row loaded into the source table. Then, calculate the difference between last available timestamp and the current time. With this difference, dbt can tell you whether your data is considered stale or not. As different tables may have varying refresh intervals, you can set individual thresholds for each one.

Basic configuration for setting up a freshness test looks like this:

sources:
  - name: stripe
    database: raw_data
    schema: stripe
    tables:
      - name: charges
        freshness:
          warn_after: { count: 1, period: day }
          error_after: { count: 2, period: day }
	  loaded_at_field: loaded_at

This config means the following:

• we are checking charges table for freshness

• dbt will raise a warning if the data is 1 day old

• it will alert an error if the data is 2 days stale

• lastly, we specify that loaded_at is a timestamp column that has to be checked for freshness

You can tweak this config in several ways. For example, you may add freshness block to the source level (instead of each individual tables). In that case same data freshness rules will be applied to all tables.

Another improvement is to add filter parameter and limit the amount of scanned rows to only last week or so (use syntax of your database here):

freshness:
  error_after: { count: 1, period: day }
  filter: datediff('day', loaded_at, current_timestamp) <= 7

To trigger freshness tests you need to run command:

dbt source freshness

If you need to run freshness tests only for specific sources/tables, you could use node selection syntax:

# check all tables in the source named "stripe"
dbt source freshness --select source:stripe

# check "charges" table only
dbt source freshness --select source:stripe.charges

How often should you perform freshness tests? Aim to conduct them before each model run to confirm you're not using stale data. However, you might want to do it more frequently because freshness tests are separate from dbt run/build commands. Running freshness tests more often can help prevent stale data during production runs. Being aware that your sources are outdated earlier will provide you more time to correct the issue before materializing production models.

Assertion tests

Similarly to SQL models, you could add assertion data tests to the sources.

⚠️ Pay attention, since dbt 1.8 tests parameter was updated to data_tests. Old syntax will be deprecated with time. See explanations here.

Start by adding generic tests to columns:

sources:
  - name: stripe
    tables:
      - name: charges
	
        columns:
	  - name: id
	    data_tests:
	      - not_null
	      - unique

Or you could add singular tests by putting .sql file to /tests folder:

select *
from  {{ source('stripe', 'charges') }}
where amount < 0

To run tests on sources use the following syntax:

dbt test --select "source:*"

# or
dbt test --select "source:stripe"

# or
dbt test --select "source:stripe.charges"

If you are utilizing modeling layers, particularly the staging layer for your sources, you can move all those tests from sources to staging models. This is only feasible if your staging models are one-to-one representation of your sources. Another advantage of testing staging models is that if you use the dbt build command instead of dbt run, failed tests on staging models will prevent issues from propagating to downstream models.

What tests should I perform on the sources?

While this is a very open-ended question, I typically pick something from a set of “golden tests” that should ensure the correctness of my data sources.

Here you can find a few recommendations:

• Check for expected columns (e.g. lighter version of "data contracts"). For instance, try using the expect_table_columns_to_contain_set test from the dbt_expectations package.

• Check for duplicates. If the source has a primary key, this task is straightforward. If not, create a surrogate key and test it for uniqueness and non-null values.

• Check for completeness. This somewhat complex test should alert you about missing dates or particular column values (e.g., missing partition). I typically create custom SQL tests for such cases, but there may be a package available for this purpose.

• Check for relationships. A standard relationships data test should be useful for this purpose. I use it to check sources with the same identifiers, e.g., all user_id's from the orders table must match those in users table.

• Check for volume anomalies. This is particularly useful when the data is recent but contains a significantly lower or higher number of rows. Consider using the volume_anomalies test from the elementary package.

If you liked this issue please subscribe and share it with your colleagues, this greatly helps me developing this newsletter!

See you next time 👋

The release of dbt v1.8 will introduce native unit-testing capabilities. This exciting new feature, long-awaited by many, is significant for several reasons. In this post I’ll explore why this feature is important, how it differs from existing dbt tests, and how you can apply it to your existing project.

What is unit testing?

In software engineering, unit testing is a process in which you test the smallest parts of your code. Typically, these are functional units such as functions or methods within a class. This involves providing some input values and observing the output to figuring out if the code works as expected. For any given input, there should be an expected output - this is the essence of unit testing.

Applying this concept to dbt, the smallest functional unit would be a model. Similar to functions, a model takes one or more inputs (other models) and produces an output (the resulting dataset).

In accordance with unit testing principles, you could simulate an input and expect an output for any given dbt model.

This approach differs from the classic dbt tests approach for two reasons:

1. Existing dbt tests are too generic and lack knowledge of the underlying logic.

2. It's necessary to build the model before testing it.

Testing logic 🤔

In SQL, you might encounter numerous logical transformations, such as applying column functions, window functions, complex CASE statements, and string manipulations. Such logic is challenging to test with generic tests due to the lack of context for the transformation. However, in unit testing, you know the input, allowing you to anticipate a specific output. This makes testing such logic more viable.

Think about how often you had to modify the logic of a model because a real test case revealed that the existing logic was flawed. Reported bugs and edge cases serve as excellent input sources for unit testing. By using these, you can strengthen your models against future changes and prevent the repetition of past errors.

Unit testing is especially useful during refactoring. When refactoring affects a large number of models, it's challenging to ensure that the existing logic remains intact and no new bugs are introduced. If you have unit tests in place, you can confidently maintain the expected functionality without breaking production models.

Testing without building 👷

Another great benefit of unit testing is a testing using mock data and without building models with production data.

Using mock data in unit testing can provide faster results, as it eliminates the need to process your entire dataset. This approach allows you accelerate development cycle and simplify troubleshooting, as you don't have to wait for large amounts of data to be processed and transformed. This not only saves time but also resources, making your data warehouse bill much lower.

Finally, mock data provides a better understanding of what the output results should look like. This is because you can model the expected behavior even in an Excel spreadsheet.

Unit testing in dbt 1.8

⚠️ By the time of writing this article, unit testing is still in beta. Yet, you still could try this functionality to test-drive it. In the “Demo” section I’ll show you how to do that.

To implement unit testing in dbt, you need to take a few steps. Firstly, configure the unit test itself. Secondly, create mock data, which will serve as the input for your models and additionally the expected output. To simplify this process, the developers of dbt have combined these steps.

To define a unit test in dbt, create a config file in YAML format. This is similar to how you would document your models or define data sources. Within the /models folder, create a file named unit_testing.yaml (the name is arbitrary, you can use something else).

Inside of the file let’s define our unit test:

You need to define a few components here:

• The name and description of the unit test

• The referencing model

• The input (mock) data inside the “given” parameter

• The expected output in the “expect” parameter

In the example above, mock data is defined as JSON rows, but it also can be defined as CSV rows like this:

- input: ref('stg__users')
  format: csv
  rows: |
    user_id,email
    1,john@gmail.com
    2,mary@gmail.com

You have several options to run the test(s):

# run all tests on dim_users model, including unit tests:
dbt test -s dim_users


# run only unit tests:
dbt test -s "dim_users,test_type:unit"

# run a test by its name:
dbt test -s unit_testing__dim_users

Remember, if the model you're testing relies on another model (which is likely in 99% of cases), you need to materialize the parent model first. However, there's a workaround to bypass full materialization step. You can create empty tables for upstream models like this:

dbt run -s "stg__users stg__orders" --empty

Demo

To demonstrate how unit testing works in real life I’ve prepared a simplest demo example. Similarly to the example above, it shows a pipeline of building dim_users model out of users and orders tables:

You can find the project in this repo. You can try it in Github Codespaces, see the instructions in the README.

Next, in file _unit_tests.yml you can find a configuration of the test. It’s pretty much the same as in the example above.

To run the test, you can try dbt test:

As you can see, the test was successful since the output matched the expected result. To simulate a failed test, you could modify the dim_users model. For example, change the LEFT JOIN to an INNER JOIN. After re-running the test, you're likely to notice a difference:

The output indicates that the actual result does not match the expected one, and it even highlights the problematic rows. This is unit testing in action!

If you liked this issue please subscribe and share it with your colleagues, this greatly helps me developing this newsletter!

See you next time 👋

This year is going to be full of exciting new releases for dbt. As promised by dbt Labs in November of last year, their current focus is on ensuring the stability of dbt-core. Additionally, one of the most highly anticipated features is the native unit testing, which is expected to be introduced in version 1.8 this spring. Exciting!

Meanwhile, the developers of dbt Cloud continue to release new features for their commercial product. Today, we will discuss some of the improvements they have made to dbt Explorer, which replaces dbt docs in dbt Cloud. These new features add significant value to the platform!

New features in dbt Explorer

dbt Explorer is the successor of dbt docs for dbt Cloud users.

dbt Explorer is slick and modern. It has revamped UI, supports cross project lineage (in case you are using dbt Mesh), and gives you old good models documentation and lineage graph. And it just got a little bit better!

Meet three new features of dbt Explorer:

• Column level lineage 🤩

• Project recommendations ✍️

• Performance assessment 🚀

Column-level lineage

Yes, column-level lineage is now included in dbt!

However, there are some limitations. First, it will only be available in dbt Explorer and there will be no updates for older dbt docs. Second, currently it is only available for users on the Enterprise plan, so those on the Team plan will not have access to it.

Nevertheless, this is great news and shows that this feature is not forgotten. Hopefully, one day we will have widespread accessibility for dbt-core as well.

On a side note, Toby Mao, one of the creators of SQLmesh (dbt alternative) mentioned that they support column-level lineage in their UI as well as they have support for parsing existing dbt projects. So definitely try them out if you want to have CLL for your models.

(source)

Recommendations

Recommendations provide insights on your project and models. For example, the general overview tab displays documentation and test coverage. Additionally, each model in your project has a tab with specific recommendations.

All of that functionality is available in the open-source dbt-project-evaluator package. It appears that dbt Cloud runs all of those checks in the background, so you don't need to install the package yourself. However, if you are not a dbt Cloud user and still want to check your project for best practices, I highly recommend using this package.

Performance

Finally, the performance feature allows you to monitor the run times of your models and provides an overview of the longest running models and the models with the most number of failures.

Another interesting aspect is that you can observe the trend of past runs for a specific model, which can give you an idea of the runtime's increase over time. This enables you to keep track of models that gradually increase your costs and optimize them accordingly.

If you want to implement it yourself, take a look at the Elementary package. Essentially, it provides very similar data that allows you to monitor your project.

More on all three features you can read from the official dbt blog post.

Weekly (db)tips

Now to some tips :)

Last week, I had a discussion with students of the "Advanced dbt" course (in which I serve as a teaching assistant). We covered two topics: macros documentation and testing business rules in dbt. Here is a brief summary of both discussions.

Macros documentation

Apparently, not many dbt-folks are aware that you can document your macros in the same way you document your models and data sources.

Create a YAML file in the /macros folder and add a description to the macro and its arguments:

Having documented macros can bring the same benefits as documenting your models:

• clarity and understanding of the use case

• easier to discover and reuse in new models

• knowledge sharing with your team

Testing business rules

In dbt, we typically write tests to check data quality. For example, we check for column emptiness, uniqueness, or perform other assertion tests.

However, there are times when it is also useful to test business rules and context.

For example, in your dim_user_subscriptions table, you have two calculated columns: first_subscription_date and count_of_total_subscriptions. From a business standpoint, it is impossible to have an empty first_subscription_date while count_of_total_subscriptions is greater than 0. This breaks the logic.

Your business may expect the subscriptions table to always contain information from all payment providers, or that the data should come from both IOS and ANDROID platforms. However, there may be delays or other issues with data delivery that can go unnoticed with simple assertion tests. To identify these errors, it is necessary to understand the business context and set up appropriate tests.

I typically use dbt singular tests that can capture these tricky scenarios and alert the team before they reach the CEOs dashboard. In some cases, you could even promote singular tests to generic.

If you liked this issue please subscribe and share it with your colleagues, this greatly helps me developing this newsletter!

See you next time 👋

When you run dbt commands, they generate a significant amount of metadata. This includes storing compiled SQL code of your models without Jinja, compiled code of generic tests, and a DAG pipeline for dbt docs. These artifacts, collectively known as dbt artifacts, can be seen as byproducts of your dbt commands. Each time you run dbt commands, the artifacts are updated to reflect the latest state of your project.

There are various applications of dbt artifacts. One of the most popular example is some dbt packages. These packages rely on dbt artifacts to gather information about your project directly. Another example is data observability, as dbt artifacts provide insights into model runtimes, failed tests, and source freshness information stored conveniently.

“dbt artifacts is a way to make analytics on your analytics”

(thanks to Emily Hawking for this cleverly expressed thought)

In today's post, I will guide you through dbt artifacts that may inspire you to use them in your job, enhancing the efficiency and performance of your dbt project.

What are dbt artifacts exactly?

dbt artifacts are JSON files that are stored in the /target directory whenever you run dbt commands. Each command produces different artifacts.

There are five types of artifacts:

• manifest.json – a file that contains all info about your dbt project and its resources (models, tests, macros, etc). Each resource also contains information about configuration and properties.

• run_results.json – stores info about a completed dbt invocation, including models timing, test results, number of captured records by dbt snapshots, etc.

• catalog.json – created after running dbt docs generate and contains info from data warehouse about generated views and tables (number of rows, column types, etc).

• sources.json – produced by dbt source freshness command and contains info about sources and their freshness.

• semantic_manifest.json – stores a comprehensive information about dbt Semantic Layer.

Let's quickly dive into each file to discover any interesting findings.

manifest.json

Manifest file is produced by almost every dbt command, excluding the ones that do not work with models, like dbt debug, dbt init, etc.

It represents your dbt project, including models, tests, data sources, macros, exposures, and more. It also includes the relationships between these items, which dbt uses to visualize the DAG (Directed Acyclic Graph).

Interestingly, even if you run only a subset of models (e.g. with --select), all resources will appear in the manifest. However, some properties, like compiled_sql, might be missing for models outside of the selected, since such properties only appear after compiling and executing models.

The manifest is used to render dbt docs and perform state comparison. Having a production manifest is necessary to perform selectors like state:modified.

Since it is a simple JSON file, you can open it with any editor and explore it on your own.

Tip: By default, the file is formatted as a single line, which can make it difficult to read. Use this simple Python command to convert it to a properly formatted and indented view:

python3 -m json.tool manifest.json > manifest_formatted.json


The manifest has several top-level keys for each item type or configuration:

{
    "metadata": {...},       # metadata dbt version, run date, etc
    "nodes": {...},          # list of models, seeds, snapshots, tests
    "sources": {...},        # list of data sources
    "macros": {...},         # info macros
    "docs": {...},           # doc strings from YAML files
    "exposures": {...},      # list of exposures
    "metrics": {...},        # dictionary of metrics
    "groups": {...},         # dict of groups
    "selectors": {...},      # user-defined selectors
    "disabled": {...},       # list of disabled models
    "parent_map": {...},     # list of first-order parents (for DAG)
    "child_map": {...},      # list of first-order child (for DAG)
    "group_map": {...},      # mapping of group names to nodes
    "semantic_models": {...} # dbt Semantic Models
}

By utilizing nodes, sources, exposures, and metrics, you can obtain a comprehensive list of all the resources in the project. By using parent_map and child_map, you can construct a DAG from these resources. Additionally, the docs property allows you to identify which models and tests have documentation. The possibilities are endless!

The manifest is the most basic and commonly used artifact as it contains all the essential information about your dbt project.

run_results.json

The run_results stores the results of a dbt invocation. It only contains the result of the most recent run and does not store the history of all runs.

When you execute the dbt command, such as dbt run, this artifact will store all the information about that run. This includes the full invocation command (with flags and parameters), used selectors, as well as the results and statuses of each model, test, or other resource.

It contains four top-level keys:

{
    "metadata": {...},       # dbt metadata
    "results": [...],        # results for each model in the run
    "elapsed_time": 12.34,   # elapsed time in seconds
    "args": {...}            # info about dbt command
}

The array of results contains information about each model in the run, limited to the selector you used. It shows the used SQL, timing, name of the final table/view, and the status (success or error). The top-level elapsed_time parameter indicates the total time the command took to execute, from start to finish (in seconds).

By using run results, you can generate advanced analytics for your dbt project. For instance, consider the elementary dbt package. This package offers data observability capabilities, allowing you to extract statistics from run results artifacts, store them in your data warehouse, and create a comprehensive observability dashboard:

catalog.json

The catalog artifact is generated by running the dbt docs generate command. This file provides information (from data warehouse) about the tables and views created from your dbt models. Essentially, it includes the name of each model in the database, details about the columns and their data types, as well as statistics like the number of rows, table size, and last modified date.

It has 4 top-level keys:

{
    "metadata": {...},  # dbt metadata
    "nodes": {...},     # list of tables/views for models
    "sources": {...},   # list of tables/views for data sources
    "errors": null      # error for docs generate command
}

This artifact can be helpful for checking the percentage of documentation coverage. Also, it allows you to compare the number of columns in the model YAML and in the database, and then verify the existence of documentation using manifest.json. These and many other checks are implemented in the package called dbt_project_evaluator. (Let me know in the comments if you want t detailed tutorial on this package).

sources.json

The sources artifact displays the results of the dbt source freshness command. It records information about the freshness check for each data source with a defined check. This information includes the maximum available timestamp, query execution time, and the status of the check.

The file itself has a simple structure:

{
    "metadata": {...},     # dbt metadata
    "results": [...],      # arrays of freshness checks
    "elapsed_time": 1.23   # total execution time
}

For example, in dbt Cloud, the source freshness dashboard uses this artifact to display freshness statistics. However, you can build the same dashboard yourself by using the aforementioned elementary package to store the results of the check in the data warehouse.

semantic_manifest.json

This file is a special artifact used by MetricFlow to build metrics for the dbt Semantic Layer. It contains descriptions of the models required to work with the semantic layer. With this file, MetricFlow can understand how to build your metrics, including which models to use and which tables to join.

A semantic manifest is generated every time your project is parsed. This happens when you invoke dbt parse or any of the run, build or compile commands. Please note that the semantic layer with MetricFlow was introduced in v1.6, so you may not have this file in earlier versions.

The file structure is quite simple, with only three top-level keys:

{
	"semantic_models": [],  # entities, dimensions and measures
	"metrics": [],                   # metrics definitions
	"project_configuration": {...}   # additional configs
}

The use of this artifact outside of Semantic Layer or MetricFlow is somewhat limited. To my knowledge, I haven't come across any interesting applications of this artifact. There are some 3rd-party semantic layers, such as Cube, that have integrations with dbt (check out this link), but these layers do not rely on MetricFlow definitions. Instead, they use their own method to describe metrics. If you know of any cool usage of the semantic manifest artifact, please let me know in the comments!

Final words

dbt artifacts provide a powerful way to gain insights into your dbt project. Essentially, dbt artifacts are simple JSON files that store valuable information about your models, running times, documentation, and more. By leveraging external dbt packages, you can create a comprehensive view of your project to improve its performance and robustness.

2023 was an exciting year for the dbt and Analytics Engineering field. There were a lot of changes: different pricing for dbt Cloud, the acquisition of Transform, several dbt version upgrades, and various changes to the product itself. On top of that, the data industry and AI had a pretty fruitful year. We’ve seen a dawn of ChatGPT and LLMs, AI-based start-ups and numerous applications that really changed IT industry.

Considering all these developments, let's explore together and try to predict where Analytics Engineering is headed in 2024.

🤖 Tight(er) integration with AI

There is no doubt that AI and LLMs were a major focal point in 2023. Not using ChatGPT in your daily work is akin to digging a hole with a shovel when you have an excavator in your garage. And we’ve seen a lot of applications across all technological fields, including data engineering and analytics in general.

At some point, people started applying LLMs to dbt. As example, this package can read your dbt project and give suggestions for improvements. Well, I doubt that the quality of suggestions were matched to a qualified analytics engineer, but at least this was a start.

More promising application was writing dbt documentation. This task is more straightforward to describe and the quality of output is much better. This extension allows you to write dbt documentation with AI and gives you a column-level lineage. Now we are talking!

And finally, the most promising application, in my opinion, is smart chat-bots for analytics. Yes, we’ve seen a rise of text-to-SQL bots that could translate human request to SQL and return a resulted dataset. But most of them lacked one thing — deep understanding of the business and underlying processes. And that what separated “dumb” chat-SQL bots from human data analysts. But hear me out, there is a better and smarter way.

Jason Ganz and some other folks from from dbt Labs explained how they managed to increase the quality of answers of text-to-SQL bot by giving it Semantic Layer as an input (instead of raw data description). They’ve seen a mind-blowing 83% increase of accuracy rate!

So, yeah, AI is powerful and it can be applicable to dbt and Analytics Engineering. I hope that in 2024 we will see some quality tools that will help us to use LLMs more effectively and conveniently🤞.

🏢 dbt for Enterprise

dbt is a business, and like any business, it needs to earn money. In the software industry, particularly for dbt, big companies or enterprises present the greatest opportunity for revenue generation. And no surprises dbt Labs is rolling out a lot of features for them.

One of the key ideas that was visible throughout Coalesce 2023 was about simplifying scaling of dbt in organizations. Just watch this keynote video and count how many time they mentioned word “organization”.

Notable solutions include:

• dbt Explorer feature that will come in place of dbt docs to give more convenient experience for Enterprises (will it even be available for dbt-core users? Dunno 🤷‍♂️)

• dbt mesh — a set of solutions that will allow companies build Data Mesh architecture. dbt developers steadily converged to this point by implementing various features, such as cross-project refs (since v1.6) or data contracts (since v1.5). Finally, big customers will be able to try out this new architecture without switching the tooling.

• Finally, dbt Cloud now natively supports Microsoft Fabric and recently got a native adapter for Synapse DWH. Microsoft Fabric by itself is very interesting set of technologies and solutions for building data platforms that follows Data Fabric architecture (like, Data Mesh wasn’t enough). And many big companies who are strategically with Microsoft, are going to use Fabric, so dbt wants to be on this market.

What all this means for analytics engineers?

I think that with time we will see dbt Mesh and some other features as requirements in job descriptions once big teams adopt these features. For us means that we need to learn these technologies, even if we don’t plan to use them on current projects. This is gonna be the way to stay competitive on the job market.

📦 Further development of ecosystem

One of the things I love about dbt is the availability of numerous external packages and how easy it is to integrate them into your project. Many dbt packages require minimal initial configuration and provide significant value to dbt projects.

My point here is extremely simple — don't be afraid to use external tools to enhance your pipeline. Without them, I would have to implement a lot of features myself, and I'm probably not the best coder. Here is a few examples.

Using Elementary it was very easy to implement data observability and alerting.

Ever needed dataset comparisons, e.g. old version of the table on Prod vs new version on Dev? Try Datafold’s data-diff tool.

Wanna diff reports for Pull Requests? Try Piperider.

Wanna know how much credits your dbt model cost in Snowflake? Try dbt-snowflake-monitoring from SELECT.

I can probably go on and on, but I guess you get the point.

Of course, the core functionality of dbt will continue to evolve. Just recall how in v1.7 they integrated the date-spine macro into the core codebase. This suggests that some features will eventually become part of the core framework. However, as an analytics engineer, your job is to deliver those features today. This is where external packages can help you.

For the next year I expect to see even more new tools and ways to enhance dbt projects and make data models even better. Better testing, better alerting, better lineage (easy column-level lineage, please!) and who knows what else. Stay in touch!

💸 Better cost containments

(Special thanks to Zach for this point)

There is no single way to write dbt models. But there are some dbt practices that are suggested by the dbt itself and the community. But this hides one problem — this may be not ideal for every data stack. Here is why.

dbt is undoubtedly a great tool for abstracting the complexity of data engineering. It allows data analysts and analytics engineers to focus on data models rather than getting tangled up in the intricacies of DDL, incremental updates, and so on. However, there is a caveat. While the same code may perform exceptionally well in Snowflake, it may not work as effectively in other databases, such as BigQuery.

I once encountered an issue while working with dynamically partitioned tables in BigQuery. Despite having what seemed to be correct code, I consistently faced table full-scans. Upon investigation, I discovered that my is_incremental() condition, which I blindly copied from Snowflake code, did not function the same way in BigQuery. This triggered expensive full-scans. Reading the documentation helped me find the proper approach. Initially, the problem was not obvious since the code appeared to work and dbt did not raise any complaints. However, it was not optimal and incurred additional costs.

That’s why learning models optimization should be one of the main priorities next year. In the current microeconomics environment, cost containments are gonna be top priorities for many companies.

Okay, since we mentioned the topic of learning, let's discuss this field separately.

🧑‍🏫 More ways to learn

Next year, there will be a lot of opportunities to learn dbt and Analytics Engineering. Knowledge is becoming increasingly accessible and easier to get. It is truly an exciting time to be in data!

I would like to suggest a few methods for learning the subject.

Courses

For beginners I’d recommend start with the official free courses from dbt Labs. In there you will find a lot of topics, from fundamentals to advanced.

For middle and senior developers there are a plenty of high-quality paid courses. I can highly recommend “Analytics engineering with dbt” from Emily Hawking and “Advanced dbt” from Lindsay Murphy. I've been on both and I truly believe these are top-notch courses!

Newsletters

Newsletters are a great way to acquire knowledge without actively participating in online sessions and scheduled calls, like in online courses. You receive periodic updates and can read news and tips at your own convenience asynchronously. Also, newsletters are great regardless of your experience level. Both juniors and seniors can find something useful in them.

Here are my recommendations:

• First and foremost, subscribe to the official newsletter from dbt — “Analytics Engineering Roundup”. I’ve been subscriber of it for a long-long time and learned quite a few things out of it.

• Next, try ”Learn Analytics Engineering” from Madison Schott. This is a high-quality newsletter about everything Analytics Engineering, including tutorials and articles on general topics. Love it!

• And finally, consider subscribing to my #dbtips newsletter, where I also talk about dbt and Analytics Engineering.

Podcasts

My only suggestion for podcasts is going to be "The Analytics Engineering Podcast" from dbt Labs (Spotify link, but I think you can find it in many other places as well). New episodes are released once or twice per month, so it won't be annoying.

Feel free to recommend any other podcasts (or any other leaning resources) you found useful.

For sure, 2024 is going to be an awesome year for Analytics Engineering and for data industry in general. There are a few challenges ahead, but also cool new technologies and breakthroughs. I’ll repeat it once again — it’s really an awesome time to be in data!

Happy New Year! 🥳

When working with some technology long enough you take a lot of things as granted. But it's always a good idea to refresh your knowledge and revisit the basics. This helps to ensure that you have a solid understanding of the fundamentals and can build on them effectively.

Furthermore, I believe that revisiting the basics also helps you acquire new knowledge. For instance, you may understand how certain components or processes function, but it is likely that you only utilize them in situations where you feel confident. However, the reality is that sometimes applying "basic" concepts can be much wiser.

So, let’s take some time to review the core concepts of dbt to enhance your skills and stay up to date with the latest advancements in the field.

What is dbt?

dbt is an open-source tool which ultimate goal is to help you build modular, reusable, and high-quality data models that can help your business answer its analytics questions.

I believe the latter is very important. If you provide value to the business, you are already using dbt correctly. Everything else is just optimizing the tool's usage.

From the other perspective, dbt started in 2016 as a way to bring software engineering practices into data analytics. Most straightforward purpose of dbt is to organize your data models into understandable pipeline. So instead of a big pile of unconnected SQL scripts you get a DAG (direct acyclic graph). On top of that, you get all the practices of software engineering, like tests, version control, core review, CI/CD, etc.

How dbt works, exactly?

To understand how dbt works internally you really need to understand it’s 4 topics:

• Jinja templating

• CLI commands

• Sources and seeds

• Compiled models

Let’s refresh all of them today.

Jinja templating

Jinja is the magic behind dbt.

It allows us to write dynamic SQL. Now we can do things that are normally not possible in SQL, for example:

• conditional if statements

• for loops

• variables (to avoid hard-code)

• reusable macros

Worth mention, that Jinja is not invented by dbt, it’s a templating engine by its own. It is used in some Python web frameworks, like Flask or Django.

Jinja also has a concise syntax. There are three main templates that you will need to use:

• Expressions {{..}} are used to output a values, usually a string. You can output a variable or a result of macro.

• Statements {%..%} are used for control flows, e.g. to set up “for” loops or “if” statements.

• Comments {#..#} are Jinja comments and they won’t be compiled and rendered.

⚠️ One common mistake I see often is commenting out Jinja code with SQL comments. Such code will still be compiled!

Data warehouses do not understand Jinja. Therefore, Jinja code needs to be compiled before executing it in the database. I will discuss the compiling process in the last section, but I wanted to clarify this here to avoid any confusion about running Jinja code in the database.

One of the most useful Jinja code in dbt is ref() macro. It helps dbt understand dependencies between models and allows to avoiding table name hardcoding (because ref() macro will be translated to the real name of table/relation during compilation).

Finally, here are a few interesting pieces about ref macro:

1. It is possible to do referencing “without” ref in FROM, by specifying a special SQL comment.

2. ref() macro actually can take two arguments for more explicit referencing (first argument is a project name, useful for external packages and better visibility).

3. Since dbt models can have versions, you can add model version along with its reference name.

Here is an awesome Jinja guide from dbt team itself.

CLI commands

dbt was initially invented as CLI tool. And for vast majority of features it still is a CLI tool. That’s why it is vital to know how to use dbt in the command line.

Here is a shortlist of my most used dbt commands:

• dbt debug — used to check dbt project health and database connection

• dbt run — runs the models in a project, probably most used command in any project

• dbt compile — helps you compile models in the project, could be useful for debugging purposes

• dbt build — a command to build and test all dbt resources (models, seeds, snapshots, tests). can be very handy when running in Production

All commands above are going to run all models in your project. It may be not needed in all cases, for example during development stage. So all commands support selection syntax which allows you to run only a subset of models.

Selection is implemented with --select argument that limits commands only to the specific models:

dbt run --select model_a

# this is the same as above, but shortened
dbt run -s my_model

# run several models
dbt run -s model_a model_b

Sometimes you may want to run a model along with all its dependencies, or a model and its parents, etc. To achieve that you could use selector modifiers:

Sources and seeds

Sources are used to describe the data that already exists in your data warehouse. In dbt, you will use sources to build your models.

Sources are described in .yaml files. You can define multiple sources and multiple tables inside one source:

To use source in the model, use the following macro:

{{ source('source', 'table') }}

In the final dbt pipeline, when you want to use data source you use source() macro, when you want to use another dbt model you use ref() macro.

Each data source can be checked for freshness. You can set acceptable timeframe of data availability to consider a source “fresh”.

Freshness can be set either for the whole source set of for each table separately. To trigger freshness check you need to use the following command:

dbt freshness

Several advanced cases of advanced usage of dbt sources:

• Freshness is not checked during dbt build. If you need that, create additional step.

• It’s possible to define EXTERNAL TABLE as dbt source, however you still need to use dbt-external-tables package to update the definition in DWH.

• Data source from external package can be overwritten with overrides property.

Sometimes source data can be defined in a static file, for example CSV file with some mappings. These are called seeds in dbt.

Seeds are used to load static data into your data warehouse. Seeds are .csv files located in /seeds folder within your dbt project. Seeds are useful when you have a data that doesn't change frequently, such as dictionaries, lookup tables, or other types of static data.

For better compatibility with DWH it is recommended to create .yml file with data types and definitions.

Compiled models

Finally, let's discuss the compilation of dbt models. As mentioned earlier, our databases cannot read Jinja code directly from our models. Therefore, before running models in the database, dbt will translate Jinja code into plain SQL code and then execute it.

Compilation is done automatically whenever you use dbt run command or can be done via a special command dbt compile.

All compiled code it stored in /target folder. Its content usually not committed to Git repository, because it can be easily recreated by run or compile commands.

There will be two subfolders in target:

• /compiled  is a folder with dbt models, directly compiled to SQL. This code, however, will not be executed in the database directly, because these are only SELECT statements.

• /run is a folder of compiled dbt models with additional DDL (data definition language) code. These scripts will be executed in the database. DDL is a code that handles the creation of tables and views, as well as the insertion of new rows for incremental models. It essentially takes care of the heavy lifting that is typically done by data engineers.

You can use node selection syntax with compile command

dbt compile --select users_model

Also, you can compile an inline code, like this:

dbt compile --inline "select * from {{ ref('orders') }}"

Both options of running are interactive, meaning the result will be printed to the terminal.

⚠️ Compilation requires a working connection to the DWH! Read FAQ about that in the documentation.

I hope this refresher post was useful and you learned a thing or two about basic concepts of dbt. Take your time to to review these fundamentals and create a strong foundation to build upon.

If you liked this issue please subscribe and share it with your colleagues, this greatly helps me developing this newsletter!

See you next time 👋

Hello and welcome to the first issue of the #dbtips digest! In this digest, I will share short tips and quick news about dbt and analytics engineering.

I created this format because not all tips can be full-fledged posts, and it's difficult to include news in such posts. So, the digest format provides a solution.

In today's digest, I will discuss the new version of dbt core, provide a couple of tips about documentation (based on feedback from previous posts), and share a few dbt commands that I find useful in my work. Let's get started!

dbt Core 1.7

On November 2, dbt Labs released a new version of their dbt Core package (v1.7). In my opinion, most of the changes in this new major version are related to the MetricFlow functionality and data contracts (really hot topics!), although some other useful features were also added. Let's discuss the most interesting changes.

Since dbt 1.6, dbt natively supports MetricFlow, which is the successor of the dbt-metrics package. MetricFlow is a tool that can generate SQL from metric configurations. In other terms, it is a Semantic Layer. In the new 1.7 version, dbt continues to enhance its functionality by adding and extending a list of features. If you want to better understand what a Semantic Layer is and how it is implemented through MetricFlow, I recommend reading this article from the official documentation. It provides a good quick start.

Command dbt docs generate now supports the --select syntax. This allows you to generate documentation only for a subset of models. This feature is particularly useful for large projects when you only need to cover a portion of your project with HTML documentation. As I understand, this is also a feature needed for their dbt mesh vision.

The freshness command can now automatically detect the loaded_at_field, eliminating the need for manual configuration. However, this feature currently only works in Snowflake, so users of other adapters will need to wait for support.

There have been several changes to dbt contracts. These changes include stricter policies for model versioning and column data types, as well as access configuration for models and whole folders. If you use data contracts, you will find these changes useful.

As a bonus, the dbt-spine macro is now included in dbt Core. Additionally, you can now set a delimiter for seeds.

Using assets in documentation

After my post about improving dbt documentation, several people have asked how to display a custom logo on the main page. Here is a step-by-step instruction.

Step 1. Create a new folder called /assets inside your dbt folder. This folder can hold any assets you want to show in the documentation. For example, you can place your logo and other useful images there.

Step 2. Configure dbt to support this folder. You can do this by adding the following line to your dbt_project.yml file:

asset-paths: ["assets"]

Step 3. Use images from this folder in your markdown. To do that, simply use the following syntax:

![Logo](./assets/logo.png)

Now use can create the most awesome dbt docs ever! 🙂

Completely hiding packages from docs

Another question that several people have asked is how to completely hide dbt packages from the documentation. The issue is that if the package contains macros, it will still be visible in the documentation.

Since there is no standard solution, we will need to use some programming to achieve this. Specifically, we need to modify the manifest.json file and explicitly set the visibility to “False” for any external dbt packages.

For example, you can use the following Python script to accomplish this. It reads the content of the file, toggles the visibility of “dbt_utils” package (as example), and saves the changes:

This solution is certainly technical, but it provides a workaround. Feel free to use it as needed.

Testing macros

Finally, this trick I discovered recently when I was creating a new dbt macro. Macros can be difficult to debug because you have to use them in the model and constantly re-compile the model to see the end result. However, I found a workaround for this issue.

If you want to compile the macro itself, starting from dbt 1.5, you can use the compile command with the inline flag:

dbt compile --inline "{{ my_macro('param1', ...) }}"

This will print a compiled code of the macro to the terminal.

If you prefer to use SQL, simply write an inline SQL query. If that seems inconvenient, you can create an analysis file. Then, you can compile the analysis file as a usual model using select syntax.

dbt compile --select my_analysis_file

Or even better, you can call it with dbt show to preview the results itself:

dbt show --select my_analysis_file

Hopefully this will make your life easier!

I hope you liked this new format of content! 🔥

Let me know in the comments what do you think! 🤔

If you liked this issue please subscribe and share it with your colleagues, this greatly helps me developing this newsletter!

See you next time 👋

There are several ways to run dbt jobs. For example, if you are using Apache Airflow or Dagster, you can do it there. Alternatively, you can choose to use dbt Cloud, which will manage the job execution for you.

However, if you only starting your journey with dbt, there is an alternative way — to run dbt with GitHub Actions. Simply put, Github actions is a CI/CD tool that let’s you run any type of workflow directly from your GitHub repository. It allows you to automate tasks such as running tests, building and deploying applications, and even running dbt jobs.

Let’s dive deeper into running dbt with GitHub Actions!

Basic setup

In this post I’ll show you how to run dbt jobs with Snowflake data warehouse, but it should be easily switchable to other warehouses.

Suppose you have the following setup:

1. working dbt project in some Github repository

2. profiles.yml file configured with env variables (will provide example below)

Github Actions allow you to run any arbitrary code on their virtual machines, you only need to setup a workflow and define steps. To setup a workflow create a new file called dbt_prod.yaml inside of .github/workflow folder in your root directory. Inside you need to define a configuration of the workflow and the steps you want to perform.

The minimal workflow to execute dbt run on a schedule may look like this:

name: Daily dbt production run

# CRON job to run dbt at midnight UTC(!) everyday
on:
  schedule:
    - cron: '0 0 * * *'

# Setting some Env variables to work  with profiles.yml
env:
  DBT_PROFILE_TARGET: prod
  DBT_PROFILE_USER: ${{ secrets.DBT_PROFILE_USER }}
  DBT_PROFILE_PASSWORD: ${{ secrets.DBT_PROFILE_PASSWORD }}

jobs:
  dbt_run:
    name: Scheduled dbt refresh and test
    runs-on: ubuntu-latest
    timeout-minutes: 90
    
    # Steps of the workflow:
    steps:
      - name: Checkout
	uses: actions/checkout@v3

      - name: Setup Python environment
	uses: actions/setup-python@v4
	with:
	  python-version: "3.11"

      - name: Install dependencies
	run: |
          python -m pip install --upgrade pip
	  python -m pip install -r requirements.txt
	
      - name: Install dbt packages
	run: dbt deps
        # optionally use this parameter
        # to set a main directory of dbt project:
        # working-directory: ./my_dbt_project
	    	
      - name: Run the models
	run: dbt run
	# working-directory: ./my_dbt_project
	
      - name: Run tests
	run: dbt test
        # working-directory: ./my_dbt_project

The workflow is quite straightforward if you look closely.

Let's go through each step:

1. Our workflow will be triggered by a Cron (a scheduler that can execute tasks at specified time).

2. Since dbt requires credentials to access your database, we define them as secret Github variables. I will show you how to create them shortly.

3. The main pipeline consists of several steps: installing python dependencies, installing dbt packages, running dbt models, and finally testing them.

For the whole setup to work as expected you also need to have a proper profiles.yml file inside of your dbt project. It may look like this:

default:
  outputs:
    prod:
      type: snowflake
      account: "abc12345.us-east-1"
      user: "{{ env_var('DBT_PROFILE_USER') }}"
      role: "daily_runner"
      password: "{{ env_var('DBT_PROFILE_PASSWORD') }}"
      database: "analytics"
      warehouse: "daily_jobs_xl"
      schema: "dbt"

As you can see, for username and password we use env variables defined in our workflow. To create those secrets in Github go to Settings → Secrets and variables → Actions and then press “New repository secret”:

Once everything is ready, you can proceed to commit all the changes to the repository. At the specified time, the Cron job will trigger the workflow, which will then start updating your dbt models 🎉!

Want more? Let’s see what else can we add to the workflow.

Adding more steps

Once you have your dbt workflow up and running with Github actions, you can start thinking about making it more advanced by adding additional steps. In this section, I will provide a few examples of how to do so.

Seeds, freshness and docs

If you use dbt seeds in your project you should definitely add this step to your workflow. Just add this step before your dbt run command:

- name: Seed seeds
  run: dbt seed

Next, if you test your sources for freshness, you should definitely add freshness test:

- name: Check source freshness
  run: dbt freshness

Finally, you can generate dbt docs and send them to S3 storage, for example. To do this, simply add the AWS CLI and copy the documentation artifacts to the storage:

- name: Install AWS CLI
  run: |
    sudo apt update
    sudo apt install --assume-yes awscli

- name: Configure credentials for aws CLI
  uses: aws-actions/configure-aws-credentials@v2
  with:
    aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
    aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
    aws-region: "us-east-1"

# Don't forget to create env variable for S3 bucket name
- name: Upload the manifest.json to S3
  run: aws s3 cp ./target/manifest.json s3://${{ env.S3_BUCKET_NAME }}/dbt-docs/manifest.json

- name: Upload the sources.json to S3
  run: aws s3 cp ./target/sources.json s3://${{ env.S3_BUCKET_NAME }}/dbt-docs/sources.json

- name: Upload index.html
  run: aws s3 cp ./target/index.html s3://${{ env.S3_BUCKET_NAME }}/dbt-docs/index.html

To make dbt docs available through the browser, you will need to configure it in the AWS console. However, this task is typically handled by DevOps or Cloud Engineers.

Build instead of run

Another way to improve your workflow is to consider using the dbt build command instead of dbt run.

The build command simplifies the process by combining the seed, freshness, test, and run stages. However, depending on your specific case, you may still prefer to run each step manually and in a specified order. The choice is ultimately yours.

CI for Pull Requests

Github Actions can be useful not only for running periodic jobs, but also for checking the health of your project during the Pull Request stage.

For example, you can create a CI workflow that checks the quality of new code by running automated checks on each commit. This is similar to pre-commit hooks I described in previous post, but now it's done on the Github side.

The final workflow my look like this (don’t forget to create a new yaml file in .github/workflows folder):

name: dbt CI pipeline

# Triggers on push to main branch
# or Pull Request to any branch
on:
  push:
    branches:
      - main
  pull_request:
    branches:
      - "*"

# Set needed env variables
env:
  DBT_PROFILE_TARGET: dev
  DBT_PROFILE_USER: ${{ secrets.DBT_PROFILE_USER }}
  DBT_PROFILE_PASSWORD: ${{ secrets.DBT_PROFILE_PASSWORD }}
  
jobs:
  ci-tests:
    name: dbt run and test CI checks
    runs-on: ubuntu-latest
    timeout-minutes: 90

    steps:
    - name: Checkout
      uses: actions/checkout@v3

    - name: Setup Python environment
      uses: actions/setup-python@v4
      with:
        python-version: "3.11"

    - name: Install Python dependencies
      run: |
        pip install --upgrade pip
        pip install -r requirements.txt

    - name: Install dbt deps
      run: dbt deps    

    - name: Run seeds
      run: dbt seed

    - name: Compile queries
      run: dbt compile
    
    - name: Run SQL linter
      run: sqlfluff lint ./models
    
    # Alternatively, run pre-commit hooks.
    # For this step, however, you may
    # need to have additional configuration
    - name: Run pre-commit hooks
      run: pre-commit run

Now, every time you add new code to the repository, it will be automatically checked for common mistakes and code formatting principles.

I hope it was useful and you found one or two interesting bits for yourself! 🔥

If you liked this issue please subscribe and share it with your colleagues, this greatly helps me developing this newsletter!

See you next time 👋

“A good tool improves the way you work. A great tool improves the way you think.” Jeff Duntemann

The developers of dbt made a significant effort by incorporating software development practices into the world of data modeling. This includes version control, tests, documentation, and more. As a result, the lives of data analysts and data engineers have become much easier.

However, there are many more tools and practices that can improve the developer experience. As engineers, we want to work efficiently and focus more on business needs rather than code formatting or fixing errors in YAML files.

Today, I will show you how to improve your workflow with dbt. We will start slowly by optimizing your productivity in IDE by using “dbt Power User” plugin for VSCode. After that, I will demonstrate how to do SQL linting. And finally, will show you how to implement automatic checks of different parts of the project that will greatly enhance your developer experience.

Let's get started!

Supercharge your IDE

When developing in dbt, you typically work with a code editor (for SQL and Jinja) and a terminal (especially if you are a dbt Core user). One issue that dbt developers often face is the lack of good support for SQL + Jinja syntax in IDEs. So, let's fix that first.

I am a big fan of the VSCode. It provides excellent support for all major languages, such as Python, and offers a pleasant working experience. Additionally, it can be greatly enhanced by using plugins. Fortunately for us, there is a plugin specifically designed for dbt called "dbt Power User".

Installing the plugin is easy — basically you need to find it on the marketplace and install. Additionally, install official Python plugin, because dbt Power User depends on it. For most setups it should be sufficient. Now let’s see what is possible with it.

First of all, you are getting a contextual menu that can tell you a lot about the model: its parents and children, available tests and documentation and even a model lineage!

Additionally, you get a lot of additional perks like:

• contextual suggestions of model names:

  

• automatic generation of staging tables based on the source configuration

• data preview

• compile a single model using a key shortcut

• and many more!

This plugin is a valuable asset for all dbt developers. Master it and increase your productivity by x100 times!

Lint and format SQL

Linting and formatting of SQL is very important when you work in a team. Without actively aligning and enforcing conventions, your project can quickly become disorganized, which can negatively impact readability and overall developer experience.

To address this, I recommend using a tool called SQLFluff. This open-source tool can be used to lint and format your SQL code. It provides numerous stylistic rules and coding conventions that will help maintain a high standard for your code base.

Installing SQLFluff into your project requires several steps:

1. Install the tool itself along with an additional plugin for dbt:

   pip install sqlfluff sqlfluff-templater-dbt

2. Add .sqlfluffignore file to exclude some folders from checking:

   analysis/
   macros/
   dbt_packages/
   target/

3. Add .sqlfluff file to configure SQL rules. Start simple, add more with time:

   [sqlfluff]
   templater = dbt
   max_line_length = 80
   dialect = snowflake
   
   [sqlfluff:templater:dbt]
   apply_dbt_builtins = true
   profile = my_dbt_profile
   project_dir = ./my_dbt_project
   profiles_dir = ~/.dbt
   
   [sqlfluff:indentation]
   indent_unit = space
   indented_ctes = False
   indented_joins = True
   indented_using_on = True
   indented_on_contents = True
   tab_space_size = 4
   template_blocks_indent = True
   
   [sqlfluff:layout:type:comma]
   spacing_before = touch
   line_position = trailing
   
   [sqlfluff:templater:jinja]
   apply_dbt_builtins = true

Now you can use SQLFluff in the terminal. To lint all models from the "marts/" folder, simply run the following command:

$ sqlfluff lint models/marts



sqlfluff lint my_dbt_project/models/marts/
=== [dbt templater] Sorting Nodes...
=== [dbt templater] Compiling dbt project...
=== [dbt templater] Project Compiled.
== [my_dbt_project/models/marts/core/fact_orders.sql] FAIL
L:  26 | P:  40 | AL01 | Implicit/explicit aliasing of table.
                       | [aliasing.table]
L:  27 | P:   1 | LT02 | Expected indent of 4 spaces.
                       | [layout.indent]
L:  27 | P:  48 | AL01 | Implicit/explicit aliasing of table.
                       | [aliasing.table]
L:  28 | P:   1 | LT02 | Expected indent of 8 spaces.
                       | [layout.indent]
L:  29 | P:   1 | LT02 | Expected indent of 4 spaces.
                       | [layout.indent]
L:  29 | P:  30 | AL01 | Implicit/explicit aliasing of table.
                       | [aliasing.table]
L:  29 | P:  31 | LT01 | Unnecessary trailing whitespace.
                       | [layout.spacing]
L:  30 | P:   1 | LT02 | Expected indent of 8 spaces.
                       | [layout.indent]

You will see the output from the linter, which shows the rules that your SQL code violates. From here, you can decide whether to fix the errors or adjust the rules.

SQLFluff can also automatically reformat the code to comply with the rules:

sqlfluff fix models/marts/core/fact_user_orders.sql

And that's basically it. Now you have a functional linting and formatting tool to help you maintain clean and consistent SQL code!

Automated checks on each commit

Finally, let's discuss more advanced topic of automation — pre-commit hooks. These are small scripts that can automatically run whenever you commit your code to the Git repository. Within these scripts, we can perform various checks to identify potential issues with the project and notify you of any problems. These checks are necessary to prevent you from committing faulty code that can cause issues.

In general, pre-commit hooks can work with any type of code, such as Python, NodeJS, PHP, etc., thanks to their extensions. Fortunately, there is also an extension available for working with dbt.

What can these hooks do? They can actually check a lot of stuff, including:

• Validity of YAML files (e.g., sources and schema documentation)

• General formatting of files (e.g., add missing end-of-line, or replacing double quotes with single quotes)

• Check SQL code using SQLFluff

• Presence of documentation for all models

• Accuracy of documentation itself (e.g., ensuring all columns from a model are documented)

• Availability of tests

• and many more

To install pre-commit hooks you need to install the Python package itself and then add needed checks:

1. Install the package:

   pip install pre-commit

2. Create .pre-commit-config.yaml file and place hooks:

   repos:
     - repo: <https://github.com/pre-commit/pre-commit-hooks>
       rev: v4.4.0
       hooks:
         - id: check-yaml
         - id: end-of-file-fixer
         - id: trailing-whitespace
   
     - repo: <https://github.com/sqlfluff/sqlfluff>
       rev: 2.1.2
       hooks:
         - id: sqlfluff-lint
           additional_dependencies: [
             'dbt-snowflake',
             'sqlfluff-templater-dbt'
           ]
   
     - repo: <https://github.com/dbt-checkpoint/dbt-checkpoint>
       rev: v1.1.1
       hooks:
       - id: dbt-compile
       - id: dbt-docs-generate
       - id: check-source-table-has-description
       - id: check-model-has-tests
         args: ["--test-cnt", "1", "--"]
       - id: check-script-semicolon
       - id: check-script-has-no-table-name
       - id: check-model-has-all-columns

Next you may run all checks manually by running a command:

pre-commit run --all-files

Or by installing those hooks as preliminary step for “git commit” command with:

pre-commit install

The latter will write pre-commit configuration to the .git/hooks folder. From now on, whenever you make any commits, the pre-commit checks will be executed first.

That’s it for today!

If you liked this issue please subscribe and share it with your colleagues, this greatly helps me developing this newsletter!

See you next time 👋

One of the things I appreciate about dbt is the ability to generate documentation for my models. I find it helpful that the code and corresponding documentation can exist together in the same codebase. This makes it simpler to create and maintain.

The default documentation generated by dbt is a good starting point. It provides a foundation, including a list of models, relationships between models, and a visual DAG with dependencies. However, we can enhance it further with a few simple tweaks.

Let me share my favorite recipes for creating dbt documentation that everyone will love to use.

Upgrade your starting page

The default starting page of dbt docs provides very generic information. It can be considered as a placeholder, because it does not contain any information about your project. So it is better to change it to something more useful.

Create a file called homepage.md inside of /models folder with the following code:

{% docs __overview__ %}

# My dbt project

This is a home page for my dbt project.

Link to [Github](<https://github.com>)
Link to [Jira](<https://jira.com>)

{% enddocs %}

Inside of “docs __overview__” macro you can put any Markdown-like text that describes your project. This can include a general description of your project, useful links to other services, and any other onboarding materials.

Hide non-useful content

If you have a fairly large dbt project, you may find that the data sources and projects in the sidebar look messy and are hard to navigate. It can be especially difficult for people who do not frequently view your documentation, such as managers.

To clear the clutter in the elements tree, you can hide unnecessary items. This includes all types of dbt resources (models, seeds, sources, etc.) and external packages:

models:
  - name: stg_mysql__users
    docs:
      show: false

Similarly, you can hide folders and packages in dbt_project.yml:

models:
  my_dbt_project:
    staging: # Hide "staging" folder
      +docs:
        show: false
  
  dbt_utils: # Hide "dbt_utils" package
    +docs:
      show: false

Hidden elements will not be visible in the rendered documentation, but will still be visible in the visual DAG (they will be marked as hidden).

Use doc blocks to keep it DRY (Don't Repeat Yourself)

One common problem with documentation is that it can become repetitive very quickly. For instance, the same column may be used in many downstream models, and copying and pasting its description can become tedious.

One solution is to use the dbt_codegen package. With the generate_model_yaml macro, you can generate a YAML schema for your models and pre-populate it with descriptions from upstream models, provided that identical columns exist and have a description:

dbt run-operation generate_model_yaml --args '{"model_names": ["dim_customers"], "upstream_descriptions": True}'

While the described solution is functional, it still has its flaws. If, for instance, you need to modify the definition of a column, you must search for all instances where the column is referenced. This violates the DRY (Don't Repeat Yourself) principle.

Better solution is to use doc blocks. They allow to write a documentation in Markdown files and then reference that documentation in YAML schemas. Create a markdown file /models/docs.md (you can choose any name) and create a doc block using Jinja like this:

{% docs user_id %}

Unique identifier of a user in our system.
 
{% enddocs %}

Then simple use this block in a YAML schema:

models:
  - name: dim_users
    columns:
      - name: user_id
        description: "{{ doc('user_id') }}"

You can see here that the column description is not a hardcoded string, but rather a reference to a doc block called “user_id” using a built-in macro doc(). This approach enables you to define your documentation once, in one place, and then reuse it wherever needed.

Color code nodes

Did you know that you can change the color of nodes in the visualization graph?

Now you do! 🙂

You can change color for the whole folder of models, e.g. by defining it in dbt_project.yml:

models:
  my_dbt_project:
    staging:
      +docs:
        node_color: "#e6b530"

Or even set it for each particular model, like so:

{{
    config(
        docs={'node_color': '#682adb'}
    )
}}

This will help you set your own color-coding for the DAG, making your documentation fancier, more informative, and pleasant to look at!

That’s a wrap! I hope that you find those tips useful and will apply to your documentation to make it even better!

If you liked this issue please subscribe and share it with your colleagues, this greatly helps me developing this newsletter!

See you next time 👋

Out of the box, dbt Core provides several types of data tests that can be applied to your models. Data tests offer a simple way to check whether your data is complying with your expectations.

Built-in tests are a great opportunity to start testing your models. However, many analytics engineers are not aware that even default tests offer great flexibility and additional features that allow for very customized setups for each project.

Today, I want to share a couple of tips that could make dbt's built-in tests more powerful and useful.

Tip 1: Use WHERE conditions in generic tests

When dbt tests your model, it scans the entire table for errors. In some cases, this may be necessary, such as when the table is small or when strict testing policies are required.

However, in many cases, it is unnecessary and you may want to test only specific rows or the most recent data. Additionally, performing a full scan of the table every day can result in high costs, so it may be beneficial to save some compute power and run time.

It is possible to add WHERE clause to generic by using a config block and where property:

models:
  - name: dim_users
    columns:
      - name: user_id
        tests:
          - not_null:
            config:
              where: "created_at >= current_date - interval '7 day'"

The WHERE clause will be added to the testing SQL and executed.

Tip 2: Adjust test severity and warning conditions

Another thing you can do is to add test severity. Some test failures may be hard errors, but others are just warnings. Warnings are unpleasant, but they can be safely skipped during pipeline execution, because they are not disrupting your models.

And here is another case. Imagine you have a table with millions of rows, and suppose one row contains an error. In this case, it would be better to mark the error as a warning, since its impact is minimal.

dbt allows you set a test severity as “warning” or set a threshold when the test is considered as error:

tests:
  - not_null:
    config:
      severity: warn
        warn_if: ">0"
        error_if: ">100"

In the example above, if a column has 1 or more missing values, dbt will consider it a warning. If the number of failures is greater than 100, then the test will throw an error.

Tip 3: Custom generic tests

Default dbt project only provides you with 4 generic tests:

• not_null

• unique

• accepted_values

• relationship

You can easily create your own custom generic tests. Custom generic tests should be stored in the /tests/generic folder, or in the /macros folder (although I prefer the former). The test should be a SELECT statement that returns failed records, wrapped in the test macro.

For example, let’s create a test that checks that a column contains only positive numbers:

{% test is_positive(model, column_name) %}
    
    select *
    from {{ model }}
    where {{ column_name }} < 0

{% endtest %}

For generic tests, the first argument should always be called "model", as this identifies a resource from the project (e.g. model, data source, seed). The second argument may be optional (we will discuss why in the next tip).

Also, you can easily add more parameters and use them in the query:

{% test is_within_range(model, column_name, from, to) %}

    select *
    from {{ model }}
    where {{ column_name }} not between {{ from }} and {{ to }}

{% endtest %}

You can call such tests similarly to default generic tests:

models:
  - name: orders
    columns:
      - name: price
        tests:
          - is_positive
          - is_within_range:
              from: 29
              to: 99

Before implementing a custom generic test, it makes sense to check external packages like dbt_utils or dbt_expectations. There is a high chance that a similar test has already been implemented in one of these packages.

Tip 4: Use generic tests on a model

In some cases, you may need to access several columns in the same test. One approach is to implement a singular test that will be specific to a particular model. However, another option is to use generic tests in a more intelligent way.

Let’s imagine an example where you want to test uniqueness of two columns. You could implement such test like this:

models:
  - name: orders
    tests:
      - unique:
          column_name: "(order_id || '-' || user_id)"

Here I moved unique test from column definition to the root of the model. Then, I used column_name parameter directly to provide concatenated columns of order_id and user_id.

Do you remember when I mentioned in a previous tip that the column_name parameter can be optional in generic tests? Technically, you can completely skip it when applying a test to the whole model. But doing so can actually reduce flexibility, so pay attention.

Due to flexibility issues, it is generally recommended to use external packages for making tests truly generic. Here is the same implementation using the dbt_utils package:

models:
  - name: orders
    tests:
      - dbt_utils.unique_combination_of_columns:
          combination_of_columns:
            - order_id
            - user_id

Tip 5: Try “dbt build” instead of “dbt run”

One last tip for today is not specifically about tests, but rather about the process of testing and building models.

In some cases, I have seen people running dbt projects as a chain of two or more commands, like this:

dbt run
dbt test

What I don’t like in this approach is that quality issues in earlier models can migrate to final models.

For example, consider the case where staging model has some data quality issues, and descendant model inherits all those issues. We will know about problems with final model only after it is built and we run “test” command.

A better way to run this code might be dbt build command:

The build command sequentially builds and immediately tests models one by one. If parent model fails the test, further dependent models will not be built, which means that bugs in earlier models will not affect the final models.

This of course leads to slightly outdated final models, but al least you won’t get questions from stakeholders about incorrect data on dashboards. Additionally, if this approach is combined with alerting (e.g. a message in Slack), you will be the first to know about any problems with the data.

Also, if severity level of a failed test is “warning”, the pipeline won’t be stopped and dbt will proceed to building further models. This might be convenient is some cases.

That’s all for today!

Hopefully you learned a thing or two about using standard built-in dbt tests and will apply some of them in your project.

If you liked this issue please subscribe and share it with your colleagues, this greatly helps me developing this newsletter!

See you next time 👋

Exciting news! You have decided to use dbt as your data modeling tool, and it's a great choice. dbt not only allows you to build excellent data models, but also provides additional functionalities such as data tests, documentation, and much more.

By design, dbt is meant to be flexible, giving you complete freedom to decide how to build your project. However, without proper planning, the project can quickly become a mess of unmaintainable code. It is therefore a good idea to establish some ground rules and set some conventions.

This post aims to provide a couple of tips that can make your project more scalable and maintainable in the long run.

Tip 1: Define the project structure

dbt is a very flexible tool that allows you to organize your project in a unique and customized way. However, this can lead to problems with scaling if it is not designed in an optimal and scalable way. Although dbt doesn't enforce a standard way of organizing models, they do suggest some best practices that you could use in your project.

In the article "How we structure our dbt projects", dbt Labs recommends organizing all models into layers, with each layer serving a specific purpose and containing unique features. I refer to this approach as “modeling layers”, as it describes how to model data using layers.

There are three basic layers:

• staging layer — describes your raw tables, with light transformations on top

• intermediate layer — where you start applying transformation and first business logic, these are modular and reusable blocks

• marts layer — contains models that will be exposed to end users and BI tools

At first, this split on layers may sound redundant, but it really pays off in the long run. This approach brings many benefits to your project. Let's start with the separation of concerns: you'll know exactly how your models are organized, which leads to a clearer dependency graph. Next, you finally speaking the same language as your colleagues, which makes scaling a much easier process. It also speeds up the onboarding of new developers who are familiar with this practice.

Take your time and read the article I mentioned — you’ll find a lot of cool tips about project organization.

Next, to ensure consistency and clarity, it is important to establish rules for model naming conventions. The previously mentioned article suggests a few naming rules for each layer. In general, it boils down to a few rules:

• don’t afraid of longer names, it’s better to make you models self-descriptive

• use layer prefix, for example stg_ for staging and int_ for intermediates

• staging models should include name of the data source and described entity

• intermediate models should describe the transformation they are performing

• marts models simply describe the entity they provide

• sometimes marts are prefixed with fct_ or dim_ (facts and dimensions) to better describe what is inside

• overall, you can (and sometimes should!) adjust these rules to better suite your needs

• once rules are defined, stick to them!

Tip 2: Adjust default settings

When starting a new project, dbt_project.yml provides sensible default settings. However, these settings are too generic and may not reflect your particular situation.

Start with settings default schema and materialization type for all your assets.

For example, if you followed tip #1, you probably divided your models into different layers, which means that you can set defaults for each layer. Additionally, do the same exercise for seeds and snapshots if you want to keep them in a separate schema.

More settings you can check in the documentation.

Next, you may want to alter the default behavior of how dbt handles database schemas. By default, name of the custom schema gets concatenated to the target schema. If you want to change this behavior, just add a macro generate_schema_name that will override default functionality:

{% macro generate_schema_name(custom_schema_name, node) -%}

    {%- set default_schema = target.schema -%}
    {%- if custom_schema_name is none -%}
        {{ default_schema }}
    {%- else -%}
        {{ default_schema }}_{{ custom_schema_name | trim }}
    {%- endif -%}

{%- endmacro %}

Tip 3: Limit data usage in development

When working with large source tables, such as user events or website sessions, it can take a long time for your models to build. It could be fine for production, but not very pleasant during development. Fortunately, there is a trick that can help limit data usage during development, and it involves using a Jinja macro.

The easiest implementation is to add the following piece to your staging models:

select *
from {{ source('mysql', 'user_events') }}

{% if target.name == 'dev' %}
where date(event_timestamp) >= date_add(current_date(), interval -3 day)
{% endif %}

This macro essentially checks the current environment and, if it is development environment, limits the data to the last three days. Alternatively, you could use variable to switch this manually whenever it is necessary like so:

{% if var('is_dev', false) == true %}
where date(event_timestamp) >= date_add(current_date(), interval -3 day)
{% endif %}

So now if you run a simple “dbt run”, all rows will be processed, however if you set is_dev variable to true it will apply the limit:

dbt run --vars '{is_dev: true}'

Tip 4: Add SQL linter

If you are starting a dbt project in a team, this tip is a must: when working with several people on the same codebase, you will eventually encounter situations where your code style differs from others. In this case, you need to do two things: create a code style guide and implement a linter. For dbt projects, you specifically need an SQL linter.

Creating an SQL style guide is entirely up to you and your team. You need to mutually agree on the conventions and rules. You can borrow a few rules from dbt style guide, Gitlab’s guide, or any other set of rules out there. Simply make it your own and come to an agreement with your team.

For lining SQL code, I recommend using SQLFluff, a Python library that checks your SQL code against a list of rules that can be customized. In some cases, it can even automatically fix issues, such as leading vs trailing commas and uppercase vs lowercase keywords. More importantly, there is a plugin for dbt-styled code.

Adding SQLFluff to your project is simple:

1. Install the package along with the plugin:

pip install sqlfluff sqlfluff-templater-dbt

2. Create .sqlfluff file and configure SQL rules inside

3. Create .sqlfluffignore file with a list of folders that should be skipped during linting/fixing

4. Run linter with the command:

sqlfluff lint

Additionally, check out this nice article that provides more details and examples of SQLFluff.

Tip 5: Use Github Actions as a scheduler

When starting your dbt project, you have several options for triggering the pipeline. You could use dbt Cloud, an external scheduler like Apache Airflow or Dagster, or you could use Github Actions. The latter option is usually overlooked in favor of the first two, so let's discuss it.

Setting up Github Actions is a straightforward process. First, create a configuration file that includes the workflow and a cron schedule. Then, set the Data Warehouse credentials, and you can easily run periodic refreshes of your models.

Many people are using Github Actions for their projects to run dbt refreshes. If you search for "github actions dbt run" on Google, you will find numerous resources. Two great articles that explain how to create your own Github Actions workflow for running dbt refreshes can be found here and here.

Thanks you for reading this post till the end!

I hope you found these tips useful and helpful for your dbt project. If you want to learn more about dbt and data modeling, please subscribe to this newsletter and stay tuned for more updates.

Happy modeling! 😊