Hardcoded values are the “cockroaches” of a data platform. You think you’ve killed them all, and then you promote a pipeline to Prod and one of them again sees the light: a lakehouse GUID you forgot to swap, a connection pointing at the wrong server, a table name that was perfectly fine in Dev and is very much not fine in production.

And the worst ones don’t even announce themselves. You won’t see an error or red banner. Your Test environment all of a sudden starts reading from Prod because one identifier never got changed, and you find out three days later, when someone finally asks why the “test” numbers look suspiciously real…

I hear you, I hear you: Nikola, that’s what deployment rules in Fabric Deployment Pipelines are for, isn’t it? Well, partly. But there’s a Fabric item built specifically to put an end to this whole genre of pain, and it’s the variable library. This article is the long version: what it is, how it’s wired together under the hood, who can actually consume it, when you should reach for it, when you absolutely shouldn’t, how it compares to the other parameterization features in Fabric, and a real telco demo to make it all concrete.

What is a variable library?

Think of it like the contacts list on your phone. You don’t memorize your friend’s number and retype it every time you call. You save it once under a name, and when their number changes, you update it in one place and never think about it again. This just works for every call and every message.

A variable library does the same thing for your workspace configuration. It’s a Fabric item, sitting right there next to your lakehouses and pipelines, that holds a set of named variables and their values. Other items in the same workspace then reference those names instead of carrying the raw values themselves. So when the value changes, you change it ONCE, in the library, and every consumer follows.

The official framing is that a variable library lets you do two things: customize configuration per release stage, and share configuration across items. Both of those words earn a paragraph.

Customizing per stage means the library can hold a different value for each environment. The source connection in Dev isn’t the source connection in Prod. The lookback window in a test run might be one day, in production, it might be 30. You set these up once, and Fabric serves the right value depending on where the consuming item runs.

Sharing across items means you define a value once and reuse it everywhere. Imagine you’ve got five lakehouses in a workspace, each with a shortcut pointing at the same external source. Without a library, changing that source is five edits and a prayer. With one, it’s a single edit, and all five shortcuts follow along. So, to put it in plain English: the idea is to stop scattering your truth across a dozen items, and let promotion between environments stop being an act of manual courage.

The anatomy: variables, value sets, and the active set

To understand the Variable Libraries feature in Fabric, you need to grasp three core parts. Once you get these straight, the rest is detail.

Variables and their types

A variable is the true atom here. Each one has a name, a type, a default value, and an optional note.

Fabric splits the types into two camps. The basic ones are exactly what they sound like:

Then there are the advanced types, and these are the ones worth caring about, because they understand the Fabric world around them. An Item reference stores a pointer to an existing Fabric item (a lakehouse, a notebook, a pipeline) by holding its workspace ID and item ID. A Connection reference stores the ID of an external data connection, like Snowflake or Azure SQL, so your items can point at external resources without you pasting raw connection strings into a string variable.

That second one matters more than it looks. Pasting GUIDs and connection strings by hand is the exact behavior that gets you “Test is reading Prod”. The advanced types let the library carry that meaning properly.

One naming caveat to keep in mind: in notebooks, variable and library names are case sensitive, even though the library name is not case sensitive in the portal itself. Another reason to be consistent and to avoid spending an afternoon debugging nothing.

Value sets

Here’s where environment promotion actually happens.

A value set is a full set of values for every variable in the library. Every library has exactly one default value set with the baseline values. On top of that, you create alternative value sets – one for Dev, one for Test, one for Prod, or for A/B testing, or for swapping a data source within the same workspace.

The neat part is the override model. An alternative value set doesn’t repeat every variable. It stores only those whose values differ from the default. Anything it doesn’t mention, it inherits. This keeps your value sets lean, and it makes the differences jump off the page. You open the Prod set, you see three overridden variables, and you know instantly that those three are the only things that change in production. Everything else is shared.

The active value set

At any moment, in any single workspace, exactly one value set is active. That active set decides which values your consumers receive at runtime.

You may switch it through the UI or the API. One thing worth mentioning, though: in a deployment pipeline, each stage can have its own active value set. Dev stage runs with the Dev set, Test with Test, Prod with Prod, and the same pipeline artifact resolves the correct values in each stage without you having to touch it.

There are two rules that might surprise people. There’s always one, and only one, active value set. And you can’t delete a value set while it’s the active one. If you want to delete it, you first need to make another set active, then delete.

Under the hood: Git and CI/CD

A variable library is a regular Fabric item, and as such, it plays nicely with Git integration and deployment pipelines, and it can be driven through the Fabric public APIs.

When you commit a workspace to Git, the library shows up as its own folder. Inside, the structure mirrors the override model: a definition listing every variable with its default values, plus a separate JSON file per alternative value set holding only the overrides. A Prod value set on disk reads something like this:

{
  “$schema”: “https://developer.microsoft.com/json-schemas/fabric/item/VariablesLibrary/definition/valueSets/1.0.0/schema.json”,
  “valueSetName”: “Prod VS”,
  “overrides”: [
    { “name”: “Source_Connection”, “value”: “c0f13027-9bf4-4e8c-8f57-ec5c18c8656b” },
    { “name”: “Lookback_Days”, “value”: “30” }
  ]
}

Two overrides, and you can read the intent in a matter of seconds. Your config changes become diffable and reviewable, exactly like the rest of your platform should be.

One subtle detail that may confuse people: which set is active is not stored in these files. The active selection lives as an item state at the workspace level. So deploying or importing the library doesn’t stomp on whichever set a target workspace already had active. Your Prod workspace stays pointed at Prod even when a fresh deployment rolls in. Which, essentially, is the behavior you want.

Who can actually consume from the library?

A library is only as useful as the items that can consume it.

As of today (June 2026), these items can consume variables from a variable library:

• Data pipelines, through library variables referenced as dynamic content

• Lakehouse shortcuts, parameterizing the target workspace, item, and path (string-type variables only, for now)

• Notebooks, two ways: the notebookutils library, and the %%configure magic command

• Dataflow Gen2

• Copy job

• User data functions

Look at the shape of that list. It leans hard toward data movement and data engineering: pipelines, dataflows, copy jobs, notebooks, shortcuts. That’s your first real signal about where variable libraries shine today, and where they don’t. But, more on that later…

And please, as a word of warning, mind the supported items list as it evolves! Microsoft has been widening it steadily since the feature first showed up, and something that’s missing today may well be supported by the time you read this. Check before you architect around an assumption.

When variable libraries are the right tool…

Ok, it’s time to get opinionated, because that’s why you’re here and not on the docs page:)

Reach for a variable library when you’re promoting the same artifact across Dev, Test, and Prod, and you’re tired of editing it by hand in each one. The same goes if a single configuration value is consumed by several items and you’d rather change it in one place than chase it across five, or if the value matters enough that you want it versioned in source control instead of buried in an item’s properties. Switching connections or data sources by stage is another classic fit, and connection reference variables were practically built for the multi-environment migrations many of us live in. It’s also the controlled, auditable way to let automation, or a teammate, flip configuration without opening the item and editing it live.

…and when they’re not

This is equally important part, because reaching for the wrong tool is how a platform rots from the inside.

A variable library is not a vault. Connection reference variables help you avoid embedding raw connection strings, but the variable values themselves aren’t secret material. Passwords, keys, tokens… They belong in Azure Key Vault, and proper connection management, and that’s the end of that conversation.

It’s also not the place for values computed during a run. A watermark, a current timestamp, a row count from the previous activity, that’s what a pipeline’s own variables and expressions handle. The library holds configuration, not runtime state. Going back to our contacts-list analogy: the library is the saved contact, not the live phone call.

And it does nothing for items that can’t consume it yet. If your target isn’t on the supported list, a library buys you nothing there. One more: with a hard ceiling of one active set per workspace, this is a stage-based configuration engine, not a per-user or multi-tenant fan-out machine.

Head-to-head: the variable library versus its neighbors

Fabric has several ways to avoid hardcoding things, and they overlap enough to confuse many of the practitioners. So, let’s try to put these capabilities heead-to head in the table below:

Deployment pipeline rules and variable libraries are partners far more than rivals, although the distinction might confuse almost everyone at first glance. A deployment rule rewrites a specific value at the moment you click deploy, so it’s a last-mile override during promotion. The library is a living item that any supported consumer reads at runtime based on the active value set. A clean pattern is to let the library carry the bulk of your stage-aware configuration and keep deployment rules for the narrow cases it can’t yet cover. If you find yourself maintaining a small forest of deployment rules, that’s usually a sign that a library wants to absorb most of them.

A pipeline’s own parameters and Set variable activities are a different beast, even though both have “variable” in the name. They’re about a single run: inputs passed in, state computed, and changed as the pipeline executes. The library lives above any single run and is shared across items and stages. The rule of thumb: if the value is decided before the run and is the same for every run in that stage, it’s library material. On the flip side, if it’s born during the run, it’s pipeline-variable material.

Semantic model parameters sit one layer down, scoped tightly to a single model or report, driving its data source and query behavior. They’re the right tool inside the BI layer, and as of today, a variable library doesn’t parameterize semantic models the way it does pipelines and notebooks. So these two rarely compete. You’ll often use both, one per layer.

Spark environments are about compute, not business config: which libraries are installed, Spark settings, the default lakehouse for a session, etc. The library is about which source, which table, which window, which toggle. They meet in exactly one place – %%configure can pull a library variable to set a notebook’s default lakehouse, but their jobs stay distinct. Don’t stuff compute settings into a library, and don’t try to express environment-specific business config purely through Spark environments.

A fresh arrival: schedule parameters

This feature is worth a short detour, because it shows where this whole thing is heading. Fabric Data Factory now lets you pass parameter values straight into pipelines through scheduled triggers. Instead of hardcoding values in pipeline logic, or leaning on a Set variable plus utcNow() workaround, you define the parameters once in the pipeline and supply the values at each scheduled execution. Different config for the morning run than the end-of-day batch? Set up two schedules and pass different values to each.

You can supply those values in two ways: a direct value typed into the schedule, or a variable library reference that pulls a centralized value, which is the cleaner option for environment promotion.

However, there’s one caveat here to be aware of. The parameter names in your schedule must match the parameter names in your pipeline exactly. If you get them wrong, Fabric ignores the values at runtime with no error at all… so write the names down and check them twice. It works well alongside variable libraries, and it closes a gap that scheduled runs have had for a while.

A real-life demo: One pipeline, three countries, three stages

Ok, enough theory, let’s now examine the kind of scenario that pulls everything we covered so far together, based on the world of multi-country telco platforms – a world I spend a fair bit of my time in.

Let’s imagine a telecom group, call it Helios Telecom, running in three markets. Every night, a single ingestion pipeline pulls subscriber usage data from an operational source and lands it in a lakehouse for downstream analytics. That same pipeline has to run in Dev, Test, and Prod, and in each stage, several things must change. The source connection differs from one environment to the next, along with the destination lakehouse, the source table name, the lookback window in days, and a data-quality gate that’s off while developing and on in production. Five different things, three environments. That’s fifteen chances to mess up a value by hand. Let’s use Variable libraries to take it to zero.

First, in the Fabric workspace, create a new item and pick Variable library. Name it Helios_Ingest_Config.

Next, define six variables, choosing types on purpose instead of dumping everything into strings. The connection and the lakehouse are referenced by their GUIDs, so they’re Guid variables, not the advanced reference types. You can pull the connection ID from the Manage connections and gateways. Open the connection settings and grab the object ID:

Then build the value sets. The default set is your Dev config. Add two alternatives, Test VS and Prod VS, overriding only what actually changes:

Notice how Batch_Wait_Seconds is identical everywhere? In Test and Prod, it simply inherits the default and never appears in those override files at all. And Source_Table is shared between Test and Prod but different in Dev. Each value set tells you only what’s special about it. Nothing more.

Now consume the variables in the pipeline. Drop a Copy data activity on the canvas. On the pipeline canvas, look at the bottom properties panel where the Parameters and Variables tabs live, and select the Library variables tab. Click + New. In the pop-up, choose your variable library, then pick the Source_Connection_Id variable.

You’ll assign it a reference name here, and that reference name is what the expression uses. Now go back to the Copy data activity, Source > Connection > Add dynamic content. Don’t type the expression. Instead, select the three dots next to Functions and choose Library variables, then select your reference.

Fabric inserts @pipeline().libraryVariables.Source_Connection for you.

Then, for the source table, choose Enter manually, click on the Table name, and then select Add dynamic content:

Do the same for the destination.

The pipeline now holds zero hardcoded environment values. It asks the library, and the library answers based on whichever set is active.

A downstream notebook can read the very same library, with no code changes between environments:

vl = notebookutils.variableLibrary.getLibrary(”Helios_Ingest_Config”)

run_dq   = vl.Enable_DQ_Gate
lookback = vl.Lookback_Days
table    = vl.Source_Table

if run_dq:
    print(f”Running DQ gate on {table} over {lookback} days”)
else:
    print(”DQ gate disabled for this environment”)

Let me share three things from experience here. The notebookutils.variableLibrary API reaches libraries in the same workspace only, it always reads the active value set, and the library is read-only from the notebook – you change values in the UI or API, never from code. (Service principals aren’t supported for these notebook utilities yet, so keep that out of the scope of an SPN-driven run.)

If instead you want a notebook to bind its default lakehouse per environment, that’s the %%configure route, pulling the variable by reference path:

%%configure
{
  “defaultLakehouse”: {
    “name”: { “variableName”: “$(/**/Helios_Ingest_Config/Destination_Lakehouse_Name)” },
    “id”:   { “variableName”: “$(/**/Helios_Ingest_Config/Destination_Lakehouse_Id)” }
  }
}

Last, wire it into a deployment pipeline with Dev, Test, and Prod stages, assign the workspaces, and for each stage, set the matching value set as active: Dev uses the default, Test uses Test VS, Prod uses Prod VS. This is a one-time setup.

First, while in the Test stage, open the Variable library item:

Then, click on three dots and set the Test VS value as active:

And, that’s it. From here on, promotion is straightforward. You deploy the same pipeline and notebook up the stages, and each environment resolves the correct source, lakehouse, window, and DQ toggle by itself. And straightforward, in release management, is the highest compliment there is:)

Limits, gotchas, and the permissions elephant!

Before you rebuild your whole platform on this, a few words about smaller warnings and one bigger warning that earns its own heading.

The size limits: up to 1,000 variables and up to 1,000 value sets per library, as long as the total cells across alternative sets stay under 10,000 and the item stays under 1 MB. Notes and value-set descriptions cap at 2,048 characters each. Value sets also appear in creation order, and you can’t reorder them in the UI. To change the order, you need to edit the JSON in Git directly.

And, now the elephant. Because a variable library is its own Fabric item, it carries its own permission set, and that set can differ from the permissions on the items that consume it.

Wait, what?! A user could have write access to the library while having no access at all to the pipelines and notebooks that depend on it. Let that one sink. Someone who can edit a value can change the behavior of items they can’t even see. Flip a connection reference, and a consuming pipeline suddenly starts reading from somewhere else. That’s a real attack vector, whether the change is malicious or just a Friday-afternoon mistake.

So treat write access to variable libraries as the privileged thing it is. Lock it down to trusted users and service identities, audit those assignments on a schedule, and let your items reference only libraries you’ve explicitly decided to trust. Centralizing your configuration is powerful precisely because everything leans on it, which is the same reason a careless permission grant here does so much damage.

A few habits worth keeping

• Pick types deliberately – connection reference and item reference exist to save you from pasting GUIDs by hand.

• Keep libraries focused and name them like a human will read them at midnight (Helios_Ingest_Config, not vars2_final_REAL).

• Use the note field – context is free.

• Override only what differs, so each set reads as a clear statement of what makes that stage special.

• Put the library in Git from day one – it’s a de-facto standard for a reason.

• Guard those write permissions as they matter, because they do.

Wrapping up

The variable library looks modest and turns out to be foundational. It takes the most error-prone, least glamorous job on a data platform – moving the same thing across environments without breaking it – and trades a recurring evening of GUID-hunting for a one-time setup. For centralized, stage-aware, source-controlled configuration across your engineering layer, it’s the tool Fabric built for exactly this job. So, go centralize your truth, and let “Test is reading Prod” become a story you only tell at conferences.

Thanks for reading!

Thanks to Claude for creating these nice-looking visuals.

A few months ago, I was sitting in a session at FabCon Atlanta. It was an amazing session about Direct Lake semantic models and various optimization tips and tricks, delivered by true masters, Christian Wade and Phil Seamark (both from Microsoft). Among many fantastic topics, the one that immediately caught my attention was the new feature that Christian Wade introduced: User-context-aware calculated columns.

Although we all know that DAX calculated columns are the “last island” in what are considered recommended data modeling practices (“Roche’s Maxim”, etc.), this one still stood out for me as something that might be super useful in certain scenarios.

I was so excited that I immediately shared the news on social media. And, reactions on that post confirmed that Power BI practitioners were quite confused about this feature (so was I).

To be honest, I never heard of a concept called: User context. We all know about the row context and the filter context in DAX, but what on Earth is now the user context?!

In simple words, it allows you to capture the user’s locale settings (like culture, for example) and bake this information into your semantic model. It relies on the USERCULTURE() DAX function. If you think of the USERCULTURE as one of the new, fancy DAX functions, you can’t be more wrong! In fact, this function has been here for a while, but up until now, we were able to use it only in DAX measures to dynamically calculate measure results based on the user’s locale settings (language code-country code, such as “en-US”, “de-DE”, “it-IT”, and so on).

You can read more about the USERCULTURE() function in Microsoft’s official documentation.

One of the most common examples was to leverage this function to dynamically translate the visual titles, based on the user locale settings. Here is the basic example:

SWITCH (
  USERCULTURE(),
  “de-DE”, “Umsatz nach Produkt”,
  “fr-FR”, “Ventes par produit”,
  “Sales by product”
)

In the code above, the measure obtains the user culture settings (let’s say, it’s German “de-DE”), and then returns the “Umsatz nach Produkt” string. This happens dynamically during the measure evaluation.

There were many creative implementations of this pattern, like this one from Gerhard Brückl, where he explains how to extend this approach and make it even more robust in synergy with the Field parameters feature.

Word of warning: The USERCULTURE function behaves differently between measures and calculated columns. As mentioned on the Microsoft docs page, the USERCULTURE() function only returns the current user’s culture when called inside a measure. If you use USERCULTURE() in a calculated table or calculated column, it returns the culture name of the semantic model’s default language instead, because those expressions are evaluated at model load time.

Now, to the new feature:) Which is, by the way, still in preview at the moment of writing, so you first need to enable it in the Preview features tab in the Options and settings.

Once I open Power BI Desktop, I’ll connect to the local Excel file that consists of two sheets:

1. Months – this is the sheet with month names in various languages (English, German, Spanish, etc.)

2. Sales – a typical fact table that contains data about sales

As you may tell, I’m using German locale settings (since I’m based in Austria). So, the idea is to automatically display month names in German for me, but if my colleague is using, let’s say, Italian locale settings, their months should be displayed in Italian.

The first step is to create a new calculated column:

I’ll use the following DAX expression:

Month Name User Culture = 
VAR _UserCulture = USERCULTURE()
RETURN
SWITCH (
    _UserCulture,
    “en-US”, Months[English],
    “fr-FR”, Months[French],
    “es-ES”, Months[Spanish],
    “de-DE”, Months[German],
    “pt-PT”, Months[Portuguese],
    “it-IT”, Months[Italian],
    Months[English] --Default
)

The key thing for this to work is to switch the context from Standard to User context. Otherwise, the expression will throw an error. This way, Power BI knows that we plan to use this calculated column in a non-standard way.

Now, when I start building my visuals on the report canvas, the new column will automatically switch to display month names in German:

But, when my Italian colleague opens the same report, they will see month names displayed in Italian (of course, assuming that their locale settings are Italian:)) – How cool is that!

A couple of remarks worth keeping in mind before you go crazy finding more creative ways to implement this feature (especially important to understand the difference between the Import and DirectQuery models’ behavior):

• DirectQuery – the result is determined by the language (locale) specified in Language Settings in the Power BI service. The default in Language Settings specifies locale is determined by the user’s browser language setting, which means the same calculated table or column can return different results depending on the browser language settings for each user

• Import mode – the result is statically determined during refresh and will not change at query time. For managed refreshes, such as scheduled or interactive, the locale is not based on the user’s browser language setting but instead uses an invariant locale

What are your thoughts about the user context calculated columns? Any practical use cases (aside from translations, which is a big win on its own) you can think of right now? Please let me know in the comments.

Thanks for reading!

I am launching a new live workshop:

AI-Native Data Engineering with Microsoft Fabric

It runs on September 7, 2026, live online, from 5pm Central Europe / 11am EDT / 8am PT.

You can find the full page here:

https://courses.data-mozart.com/

But before the promotional part, I want to explain why I built it.

Because I think the conversation around coding agents and data engineering is slightly broken.

Most demos show the easy part.

You type a request. The assistant generates a notebook, a pipeline, a SQL query, maybe even deployment commands. Everything looks clean. The code has comments. The folder structure looks professional. The demo data works.

Then the demo ends.

Real data engineering starts after that.

Data bugs are quiet

In application development, many bugs are loud. A page fails to load. A test breaks. A service throws an exception. A user cannot complete a workflow.

Data bugs are often more polite.

They produce a table.

The dashboard refreshes.

The number looks close enough.

Nobody notices until the downstream decision has already been made.

That is why generated data engineering work needs a different level of skepticism. A coding agent can write PySpark that looks perfectly reasonable and still contains a dangerous assumption:

• an inner join that silently drops unmatched transactions

• schema inference where a contract was needed

• overwrite behavior that should have been append or merge

• a deduplication rule that keeps the wrong event

• a one-to-many join that duplicates financial facts

• a missing-source behavior that reports success instead of stopping

These are not science-fiction problems. They are normal data engineering problems, accelerated by better tooling.

The risk is not that the assistant is useless.

The risk is that it is useful enough to create a lot of plausible output quickly.

The new skill is not prompt theater

I don’t think the serious skill here is “prompt engineering” in the shallow sense.

The deeper skill is engineering direction:

• specifying what should be built

• giving the agent bounded context

• defining what it may and may not touch

• making it stop at review checkpoints

• reading generated code critically

• validating behavior with evidence

• deciding what is safe to automate and what needs human approval

That is the muscle I want to practice in this workshop.

Not “look, the model wrote a notebook.”

More like:

“Here is the scaffold it generated. Now find the assumption that would break production.”

That is a much more useful exercise for working data engineers.

Why Microsoft Fabric?

The hands-on implementation is in Microsoft Fabric because concrete beats abstract.

Fabric gives us a realistic surface area:

• workspaces

• lakehouses

• notebooks

• medallion architecture

• Fabric CLI

• Copilot and related assistant-driven experiences

• capacity and licensing questions

• deployment and tenant-safety concerns

It is also where a lot of teams are currently asking practical questions:

“Can we use coding agents for Fabric work?”

“Can we scaffold items instead of clicking through the portal?”

“How do we stop an agent from doing something dangerous?”

“What does production readiness look like when parts of the code were generated?”

But I don’t want to oversell the course as vendor-agnostic. It is not.

The lab is Fabric.

The exact CLI commands, security model, resource definitions, runtime behavior, and deployment path are Fabric-specific.

What transfers is the workflow:

• repo-level agent instructions

• bounded tasks

• metadata contracts

• generated scaffolds

• validation checklists

• test and documentation generation

• LLM enrichment boundaries

• cost review

• deployment checkpoints

• human approval rules

If your team also uses Databricks or Snowflake, you will still need to adapt the platform layer. But the judgment pattern is the same.

The dangerous part is rarely the button.

It is the unchecked assumption.

What we will build

The workshop is four hours and code-first.

We will build a Microsoft Fabric medallion scaffold for a card payments scenario. I chose payments deliberately because it creates useful validation pressure without needing an enormous domain model.

Payments data gives us duplicates, late reference data, status changes, fraud signals, disputes, reconciliation questions, and one-to-many joins.

In other words: the kind of data where “looks fine” is not good enough.

During the workshop, we will create:

• repo instructions for GitHub Copilot and Claude Code

• bounded task templates

• a Fabric workspace scaffold

• lakehouse and notebook structure

• a metadata-driven ingestion framework

• Bronze/Silver/Gold transformation stubs

• a controlled LLM enrichment pattern

• generated tests and documentation

• a production-readiness checklist

• a deployment, trust, cost, and security review

The centerpiece is the metadata-driven ingestion framework.

That is where coding agents are genuinely useful, because they can produce repetitive framework code quickly.

It is also where they are genuinely dangerous, because one bad framework assumption repeats across every table it drives.

So we will not just generate the framework.

We will audit it.

What this workshop is not

This is not a no-code tour of Fabric screens.

This is not a promise that you will build a production-grade payment platform in four hours.

That would be theater.

The honest outcome is better:

You will leave with a working scaffold, a reusable way of directing coding agents, and a much sharper review lens for generated data engineering work.

You should be comfortable reading Python and SQL. You do not need to be a Fabric expert already, but you should be willing to inspect code, question assumptions, and think about validation.

If your goal is to let an agent freely modify production workspaces, this workshop will probably frustrate you in the right way.

Production deployment, deletes, permission changes, secret handling, overwrite operations, and financial reconciliation logic are not things I want agents doing without human review.

This is not anti-automation.

It is professional automation.

Why now?

Because the tools are getting good enough that ignoring them is not a serious strategy.

But they are also not good enough to remove engineering responsibility.

The data engineer’s role is shifting from manually writing every line to designing patterns, generating first drafts, reviewing assumptions, validating behavior, and deciding what earns trust.

That is a better use of human judgment.

It is also a higher bar.

If you can’t tell whether the generated transformation is preserving the right grain, the assistant did not make you faster. It made you riskier.

That is the space this workshop lives in.

Details

Workshop: AI-Native Data Engineering with Microsoft Fabric
Date: September 7, 2026
Time: 5pm Central Europe / 11am EDT / 8am PT
Format: Live online
Duration: 4 hours
Instructor: Nikola Ilic, Data Mozart
Founding cohort price: €179 Standard
Pro option: €449, limited to 15 seats, includes office hours and a 30-minute 1:1 review call

Registration page:

https://courses.data-mozart.com/

If your team is trying to understand where coding agents fit in real data engineering work, I built this for you.

Not the keynote version.

The field version.

Logical Data Management series

1. Logical Data Management for mere mortals

2. Logical Data Management - Storage Was Always the Easy Part

3. Logical Data Management: Why “Just a quick number” is never quick...

4. Logical Data Management: One Definition to Rule Them All! (this article)

Let me take you back to Sara’s Tuesday for a second.

If you read the previous article, you remember how it went. A simple request for “revenue by region” that should have taken twenty minutes, and instead ate the entire day, not because the data was missing (it was all there, sitting in the lakehouse exactly as designed), but because the meaning was missing. Eleven datasets called revenue. Two definitions that disagreed by eight percent. A forty-line measure with a comment that said “talk to Mike”, and Mike, of course, left the company back in 2023.

Now let me show you a different version of that same Tuesday.

Sara gets the same message at 9:14. Opens the same tool and types “revenue by region”. And this time, there is exactly ONE thing called Revenue - clearly defined (order-based, canceled orders excluded), owned by the finance team, certified three weeks ago. Region is defined once, the same way every single report in the company uses it. Sara drags the two onto the canvas, gets the number, writes a short note, and hits send before the coffee gets cold.

Same data. Same tools. Same Sara. The only difference is that something now sits between Sara and the raw tables and tells her, with authority, what things actually mean.

That something is the semantic layer. And if you take away only one idea from this entire series, please, let it be this one.

Ok, so what the heck is a semantic layer?

Let me give you the plain definition first, and then we’ll take it apart together.

A semantic layer is an abstraction that sits between your physical data and the people (or agents:)) consuming it, translating technical structures into business meaning.

I know, I know - that sentence is dense. So, to put it in plain English: on one side, you have the world of storage. Tables, columns, joins, foreign keys, partitions. A column called rev_amt in a table called fct_ord_ln, joined to dim_geo on a key named geo_sk. Lovely stuff, if you happen to be a database. On the other side, you have actual human beings, who think in words like revenue, region, customer, margin, and quarter, and who have absolutely no idea what fct_ord_ln is, nor should they ever have to.

The semantic layer is the translator standing between those two worlds. When someone asks for “Revenue by Region”, the semantic layer is the thing that knows which table to hit, which column to grab, how to aggregate it, how to join to geography, and which business rule takes care of stripping out the canceled orders. The consumer asks for a business concept. The semantic layer turns it into the correct physical query.

The user sees Revenue. The engine sees forty lines of logic. The semantic layer is what stands between them.

Ok, that’s nice, but it’s just pretty naming, right?

This is exactly where most people undersell the whole thing, so let me push back on it directly, because I used to think the same.

It would be easy to read all of the above and conclude that a semantic layer is basically a friendly naming layer, something like a way to rename ugly columns into pretty ones. Useful? For sure. But, hardly the most important thing in your platform.

That undersells it badly. Renaming is just the surface. What the semantic layer really does, and. honestly, this is the part that took me incredibly long to fully appreciate, is give meaning a single home.

Going back to Sara and those eleven datasets... why were there eleven in the first place? Because every team that needed revenue built their own version, in their own report, with their own slightly different logic, for the simple reason that there was nowhere central to define it once. The meaning got duplicated because it was homeless.

A semantic layer gives meaning a home. Revenue, defined once. Region, defined once. The fiscal calendar, the customer hierarchy, the canceled-orders rule… All defined once, in one place, owned by someone, versioned, documented. And every report, every dashboard, every query that asks for revenue gets the SAME revenue. Not because everybody nobly remembered to use identical logic, but because there is only one logic to use.

This is the whole difference between consistency by discipline and consistency by design. Consistency by discipline is what most of us have lived with: “please, everyone, remember to define revenue the same way.” Keep in mind, this approach fails every single time, because it depends on hundreds of people independently making the same choice, forever, without slipping even once. Consistency by design means the system itself makes inconsistency hard, because the definition lives in one governed place, and everything else just draws from it.

A semantic layer holds way more than names!

So what, concretely, does a semantic layer actually hold? Worth being specific here, because “meaning” is a slippery word, and I don’t want to leave it floating.

A mature semantic layer captures a few distinct things. It holds definitions: what is revenue, what is an active customer, what counts as churn. It holds relationships: how customers connect to orders, orders to products, everything to time. It holds calculations and metrics: not just stored columns, but the derived stuff, like year-over-year growth or rolling averages or margin percentages, the math that turns raw numbers into the figures people actually ask for. It holds hierarchies: how region rolls up to country to continent, how day rolls up to month to quarter to year. And it holds business rules: canceled orders don’t count, internal transfers are excluded, revenue is recognized on shipment for finance but on order for sales.

And then there’s one more thing it holds, increasingly: context for machines. Descriptions, synonyms, constraints - the kind of thing that lets a query engine or an AI agent understand not only the shape of your data but what it actually means and how it’s allowed to be used.

But, more on that later... that thread deserves an entire article of its own, and it’s coming. For now, just tuck it in the back of your mind, because it’s going to end up mattering more than anything else on this list.

Buy it, or build it? Wait - wrong question!

If there is one single question that trips people up, this is the one: Is the semantic layer a product you buy, or a place in your architecture?

The honest answer is that it’s a place, and several very different kinds of products can move into it.

For years, the semantic layer was basically trapped inside the BI tool. Your revenue and region lived inside Power BI, or inside a cube (hello to those of you with a few gray hairs, or no hair, like me), or inside one specific reporting tool. That worked, with one big catch - the meaning was only available to that tool. Want the same revenue definition in a notebook? An API? A different BI tool? Redefine it. The meaning was real, but it was locked in a box.

The big shift in modern data platforms is the move to set the semantic layer free from any single consumer. Meaning shouldn’t belong to Power BI, or to one application. It should sit in a shared layer that anyone can draw from - a Power BI report, a Python notebook, a SQL client, an external app - all asking the same layer for “revenue”, all getting back the same answer. You’ll hear this called a universal semantic layer, or “headless BI” (because the definitions get separated from the visualization “head” that displays them). The labels keep shifting, but the principle doesn’t: define meaning once, in a neutral place, and let every consumer share it. Kurt Buhler (Tabular Editor) has written some of the clearest material out there on semantic models and what they really are. It’s well worth your time if you want to go deeper than the scope of this article allows. And if you want the platform-by-platform view of how this decoupling plays out - how Microsoft, Databricks, Tableau, and others each implement the layer differently - Ruben Van de Voorde's "What is a semantic layer?" is the clearest map I have seen.

I won’t pretend the industry has fully arrived. We’re mid-transition, where some semantic layers are still welded to their BI tool, some are fully independent, and most real organizations run a messy mix of both. But the direction of travel is not subtle. And the reason it matters circles right back to that thread I planted above: the moment the consumers of your data stop being only humans-with-dashboards and start including AI agents, the question of whether your meaning lives in a shared, governed, queryable layer stops being a nice-to-have and becomes the single thing that decides whether those agents can be trusted at all.

Why I’ll die on this hill

I said at the very top that the semantic layer is the most important component in a modern data platform. That’s a big claim, and I’d rather defend it properly than just wave it around.

So let’s run a quick thought experiment. What happens when each layer of your platform is mediocre?

If your storage layer is mediocre, you’ve got a performance and cost problem. Painful, but fixable. Throw hardware at it, tune the queries, restructure the tables.

If your ingestion is mediocre, you’ve got a freshness and reliability problem. Also painful, also fixable, with better pipelines and some monitoring.

But if your semantic layer is mediocre or simply not there, you get a problem that no hardware on earth fixes: NOBODY agrees on what the numbers mean. Two executives walk into the same meeting carrying two different revenue figures. An AI assistant cheerfully reports a metric using the wrong definition. A board slide contradicts the official financials. The data is flawless, the queries are lightning-fast, and the organization still can’t trust a single number it produces.

That, not slow queries, is the failure that actually destroys confidence in a data platform. Disagreement about meaning. And it’s precisely the failure that no storage technology, however brilliant, was ever built to prevent.

This is why I keep banging the same drum across this whole series. We’ve spent twenty years perfecting everything beneath the semantic layer: faster storage, cheaper storage, more flexible storage, unified storage. All real, all good progress. But the one layer that decides whether anyone can trust the output, the layer where meaning actually lives, got treated as an afterthought, bolted onto whichever BI tool happened to be lying around. Logical Data Management is, more than anything else, the argument that this layer has earned the right to be treated as first-class. Designed on purpose, owned by name, governed properly, and shared across every consumer that comes knocking.

Back to Sara, one last time

Sara’s good Tuesday (the imaginary one we opened with) had nothing to do with better data or better tools. Same data. Same tools. The one thing that changed was that meaning finally had a home.

That’s the whole promise of the semantic layer, squeezed into a single analyst’s morning. The data was always there. What was missing was the layer that tells everyone, clearly and with authority, what it all means.

Next time, we get our hands dirty. A single shared layer of meaning sounds wonderful right up until somebody asks how it performs when millions of queries come crashing in, and whether all this lovely abstraction quietly costs you something. The idea is the easy part. Making it fast is the engineering...

Thanks for reading!

Thanks to Claude for creating these nice-looking illustrations.

Logical Data Management series

1. Logical Data Management for mere mortals

2. Logical Data Management - Storage Was Always the Easy Part

3. Logical Data Management: Why “Just a quick number” is never quick... (this article)

Sara is a data analyst. She has been in her current position for three years, and she’s doing a quite good job. Sara is the kind of person other people in the company definitely rely on when they need something figured out. It’s Tuesday. Nothing special about it.

At 9:14 in the morning, a message arrives from the VP of Sales.

“Hey, quick one. Can you get me revenue by region for the last quarter? Board meeting Thursday. Thanks!”

Sara reads it and thinks: sure. Twenty minutes, maybe thirty. We have a whole data platform for exactly this. There is a lakehouse. There is a semantic model. The model is even certified. This is the easy kind of request.

Sara is wrong, but does not know it yet.

9:30. The first fork in the road.

Sara opens the BI tool and searches for “revenue.”

Eleven results come back. Eleven datasets, each with some version of a revenue measure. There is Sales_Reporting. There is Finance_Actuals. There is Revenue_Dashboard_v3. There is something called FINAL_revenue_USE_THIS that was last refreshed four months ago, which is not reassuring given the name.

Which one is right?

This is the first thing nobody tells you about self-service analytics. The hard part was never running the query. The hard part is knowing which of the eleven nearly-identical things you are allowed to trust.

Sara picks the one that looks most official, Finance_Actuals, because finance sounds authoritative, and opens it up.

9:50. The definitions do not agree.

The numbers in Finance_Actuals do not match the numbers Sara remembers from a report last month.

Not by a little. By about eight percent.

Sara frowns and starts digging. After a while, the reason surfaces: Finance_Actuals recognizes revenue on shipment, while the sales team has always talked about revenue on order. So the finance number and the number the VP is expecting in their head are measuring two different things. Both are correct. They just answer different questions.

Now Sara has a problem that has nothing to do with data and everything to do with meaning. Which definition does the VP actually want? Probably the sales definition, since it is the VP of Sales asking. But the board might expect the finance definition, because that is what shows up in the official financials.

Sara does not know. So Sara sends a message asking, and waits.

11:20. Still waiting, so let us try the other path.

The VP has not replied. They are, presumably, in meetings, because VPs are always in meetings:)

Sara decides to hedge and build both versions. Sales definition and finance definition, side by side, so the VP can pick. Reasonable plan.

Except the sales definition is not in a certified semantic model anywhere. It lives in the logic of an old report that someone built years ago. Sara opens that report to see how the measure was defined, and finds a DAX expression forty lines long, full of nested conditionals, with a comment near the top that just says “TEMP FIX, talk to Mike before changing.”

Mike left the company in 2023.

Let us pause here for a second.

I want to step out of the story, because I suspect a lot of you reading this stopped seeing Sara a while ago and started seeing yourself.

Here is what has actually happened in the two hours since that cheerful 9:14 message. Sara has not yet produced a single number. And it is not because Sara is slow, or because the tools are bad, or because the data is missing. The data is all there. It has been centralized, modeled, and made available. The platform is doing its job.

What is missing is everything around the data. Which definition is the right one? Who owns it? Whether the logic can be trusted. What the business actually means by “revenue” in this specific context for this specific audience.

None of that lives in the lakehouse. It lives in conversations, in tribal knowledge, in the head of a person named Mike who does not work here anymore.

This is what I meant, in the earlier articles in this series, when I said the bottleneck moved. We solved the problem of where the data lives. We did not solve the problem of what it means and who decides. And self-service analytics, for all its promise, quietly handed that unsolved problem to the analyst.

We told people: here are the tools, here is the data, go serve yourself.

What we did not say is that serving yourself also means navigating, alone, a maze of undocumented definitions, orphaned logic, and competing versions of the truth, with no map.

Self-service became self-navigation.

1:45. A reply, of sorts.

The VP responds: “Just the normal revenue number, whatever we usually use. You know the one!”

Sara does not, in fact, know the one. “Whatever we usually use” is precisely the thing that does not exist in any single, agreed, documented place. That is the whole problem, compressed into five words by someone who has no idea they have just described one of the hardest unsolved problems in the company’s data estate.

Sara makes a decision. Sales definition, because the audience framing suggests it, with a footnote noting the finance figure differs and why. It is a judgment call. A reasonable one. The kind analysts make a dozen times a day without anyone realizing a call was even made.

3:30. It is mostly done. Is it right?

The report looks good. Revenue by region, last quarter, clean and clear.

Sara stares at it for a moment with the specific unease that every analyst knows. The numbers are plausible. They are probably correct. But “probably correct” is doing a lot of work in that sentence, and Sara is aware that on Thursday this slide goes in front of the board, and if the number is off, or if it somehow uses a definition the CFO disagrees with, the person whose name is attached to it is Sara.

There is no way to be sure. There is no system that will confirm “yes, this is the trusted regional revenue figure, blessed by finance, current as of this quarter.” There is only Sara’s judgment, a footnote, and a small knot of doubt.

Sara sends the report. Adds a careful note: “Used order-based revenue to match how the sales team reports. Happy to switch to the finance basis if the board prefers, just let me know.”

Then Sara closes the laptop and goes home.

So, did it work?

Honestly, I do not know. And that is the point.

Maybe the number was exactly right and the board nodded along and nobody thought about it again. Maybe someone in the room said “that does not match the figure in the financials” and there was an awkward few minutes. Maybe the report sat in an inbox unopened. Most analytics requests end this way, not in triumph or disaster, but in a kind of unresolved haze where you ship something defensible and move on to the next fire.

What I am certain of is this. A request that everyone, including Sara, assumed would take twenty minutes consumed most of a working day. Not one second of that day was spent on anything a database would recognize as hard. The query itself would run in under a second. Every minute was spent on meaning: finding it, questioning it, reconciling it, and finally just guessing at it.

Multiply that Tuesday across every analyst in your organization, across every “quick one” that lands in their inbox, and you start to see the real cost. It does not show up in your cloud bill. It does not appear on any dashboard. It hides inside the gap between “the data is available” and “the answer is trusted,” and in most organizations that gap is enormous, and almost entirely invisible.

We have spent a decade making the data easier to reach. Sara’s Tuesday is what is left over. It is the part we have not solved yet.

In the next article, we will finally start talking about how to close that gap, beginning with the idea that has been lurking behind this whole series: the semantic layer.

Thanks for reading!

Thanks to NotebookLM for creating these nice-looking illustrations.

Logical Data Management series

1. Logical Data Management for mere mortals

2. Logical Data Management - Storage Was Always the Easy Part (this article)

3. Logical Data Management: Why “Just a quick number” is never quick...

In the first article of this series, I made a claim that I want to pick apart now: that we spent twenty years getting very good at moving data, and somewhere along the way, the real problem quietly changed shape.

If you missed that one, here is the short version. We solved the movement problem. Data used to be scattered across systems, so we built warehouses, lakes, and lakehouses to bring it together. And it worked. But once everything was in one place, a new problem appeared that storage alone could not fix: the meaning of the data was still scattered, this time across people’s heads instead of across systems.

Fair enough. But that is a big claim to drop and then walk away from. So in this article, I want to do something harder and more honest. I want to look at the three dominant architectures, the warehouse, the lake, and the lakehouse, and ask a specific question about each one.

Not “is it good?” Of course, they are good. Millions of organizations run on them every day.

The question is narrower: what does each one solve really well, and what does it leave on the table?

Because, before we proceed, let me share the key thing: None of these architectures is broken. They are excellent answers. The problem is that some of the questions we are asking in 2026 are not the questions they were designed to answer.

The data warehouse: order at the cost of flexibility

Let us start with the oldest of the three.

The data warehouse was built around a simple, powerful idea: if you define the structure of your data up front, everything downstream becomes easier. Schema-on-write. You decide what the data looks like before it lands, you enforce that structure rigorously, and in return, you get consistency, performance, and trust.

And it delivers on that promise. A well-designed warehouse is a thing of beauty. Queries are fast, the data is clean (well, at least, it should be), and relationships are enforced. When the finance team pulls a report, the numbers tie out because the structure guarantees they tie out. Decades of accumulated engineering, from star schemas to columnar storage to query optimizers, have made the warehouse extraordinarily good at what it does.

So what does it leave on the table?

Flexibility. The same rigid structure that makes the warehouse trustworthy also makes it slow to change. Every new data source needs to be modeled, transformed, and fitted into the existing schema before anyone can use it. Want to bring in semi-structured data, like JSON event logs or sensor readings? You will spend significant effort flattening and structuring it first. Want to explore data before you know what questions you will ask? The warehouse is not built for that. It rewards knowing the questions in advance.

For decades, this was an acceptable trade. Most analytical questions were known in advance, and most data was structured anyway. But the shape of data changed. More of it became messy, nested, high-volume, and fast-moving. And the warehouse, for all its strengths, started to feel like a beautifully organized library that only accepts books in one specific format.

The data lake: flexibility at the cost of order

So the industry swung the other way.

If the warehouse was too rigid, the data lake offered the opposite: store everything, in any format, and figure out the structure later. Schema-on-read. Dump the raw data into cheap object storage, structured or not, and impose meaning at the moment you query it rather than the moment you store it.

This solved real problems. Suddenly you could store data that did not fit neatly into tables, like images, logs, clickstreams, audio, nested JSON. Storage was cheap, so you could keep everything, even data you were not sure you needed yet. Data scientists loved it, because they could access raw data in its original form rather than a pre-modeled version that had already had decisions baked into it.

The lake gave us flexibility and scale that the warehouse could not match.

And then the bill came due.

The same freedom that made lakes powerful also made them dangerous. When you can store anything in any format with no enforced structure, you very often end up storing everything in every format with no enforced structure. The industry even coined a term for what lakes frequently became: data swamps. Lakes full of data that nobody could find, nobody could trust, and nobody quite remembered the meaning of.

Schema-on-read sounds liberating until you are the fifth analyst trying to read the same poorly documented file, each of you interpreting it slightly differently, each of you producing a slightly different number. The structure did not go away. It just moved from the storage layer into the head of whoever happened to be querying the data that day.

Sound familiar? It should. This is the “meaning in too many heads” problem in its purest form.

The lakehouse: can we have both?

By now, you can probably see the pattern, and you can probably guess what came next.

The warehouse gave us order without flexibility. The lake gave us flexibility without order. So the obvious question became: why not both?

That is the lakehouse. The pitch is genuinely compelling: keep your data in cheap, open storage like a lake, but add a transactional metadata layer on top that brings warehouse-like structure, reliability, and performance. Technologies like Delta Lake, Apache Iceberg, and Apache Hudi made this real. You get ACID transactions on top of object storage. You get schema enforcement when you want it and schema flexibility when you do not. You get one copy of the data that can serve both the data scientist exploring raw files and the analyst running a governed report.

I want to be clear: this is a real architectural achievement. The lakehouse is not marketing fluff. The convergence of lake economics with warehouse reliability practically resolved a tension that had forced organizations to maintain two separate systems and copy data between them. Microsoft Fabric, Databricks, Snowflake, and others have built serious, capable platforms on this foundation.

So, with the lakehouse solving the order-versus-flexibility tension, are we done? Did the lakehouse finally answer the question?

It answered a question. A big one. But not the one this series is about.

Here is what the lakehouse still does not give you. It is, fundamentally, still a place where data lives. A very good place, with excellent properties. But the lakehouse is an answer to “where should the data be and how should it be stored?” It manages tables, files, transactions, and schemas brilliantly. What it does not manage, because it was never designed to, is meaning.

A lakehouse will happily store seventeen tables that each contain a column called revenue. It will enforce the schema of each one flawlessly. It will give you ACID guarantees on every write. And it will tell you absolutely nothing about which of those seventeen revenue columns finance actually trusts, what business rules sit behind each one, who owns the definition, or whether “revenue” in this table includes canceled orders.

That is not a criticism of the lakehouse. It is a statement about what layer the lakehouse operates at. You would not criticize a warehouse for not making coffee.

The common thread

Now, I want you to take a step back and look at all three together, and something becomes clear.

The warehouse, the lake, and the lakehouse are all answers to questions about storage and structure. Where does the data live? How is it formatted? How do we query it efficiently? How do we keep it consistent? These are important questions, and the progression from warehouse to lake to lakehouse represents real, genuine progress in answering them.

But none of these three architectures was designed to answer a different category of question entirely:

What does this data mean? Which definition is the trusted one? What business rules turn a raw column into a metric someone can make a decision on? Who owns that meaning? How do we make sure a human analyst and an AI agent both interpret “active customer” the same way?

These are not storage questions. They are not solved by a better file format, a smarter query engine, or a more elegant transactional metadata layer. They live one level up, in a layer that sits between where the data is stored and how people and systems actually consume it.

And that, finally, is the gap this series is really about.

I am not going to pretend that logical data management makes warehouses, lakes, or lakehouses obsolete. That would be both wrong and a little insulting to the genuinely brilliant engineering behind all three. You will still need a place to store your data, and a lakehouse may well be the best place yet invented to do it.

The point is narrower, and I think more interesting: the storage problem and the meaning problem are different problems. We have spent twenty years getting very, very good at the first one. We have barely started on the second.

In the next article, I want to leave architecture diagrams behind for a bit and spend a day in the life of the people who feel this gap most acutely: the data users. The analysts, the business users, the people who were promised self-service and got self-navigation instead. Because the limitations I described here are abstract until you watch someone actually run into them on a Tuesday afternoon.

Thanks for reading!

Thanks to Claude for creating these nice-looking diagrams.

Logical Data Management series

1. Logical Data Management for mere mortals (this article)

2. Logical Data Management - Storage Was Always the Easy Part

3. Logical Data Management: Why “Just a quick number” is never quick...

Two years ago, I was approached by a publisher to write a short book (~100 pages) about Logical Data Management.

I did not write it.

Not because the topic was weak (quite the opposite). The topic was strong, maybe even stronger than I fully realized at the time. But I had this strange feeling that the story was not ready yet. It sounded like an architecture topic. Useful, yes. Important, hell, yes. But still one (or two) step away from the pain that real people feel every Monday morning when someone asks them for “just a simple number”.

And, of course, that number is never simple.

“Can you give me revenue by customer segment for last quarter?”

Sounds like a piece of cake, right?.

Then you open the first report and realize that “customer segment” means one thing in CRM, another thing in the data warehouse, something completely different in the finance model, and absolutely nothing in the Excel file that the sales team has been updating manually since 2019.

Welcome to modern data management.

Everything is centralized.

Everyone is still waiting.

We solved the movement problem

For the last 20 years, a huge part of data management was about movement.

Move data from operational systems into a data warehouse. Move it from the warehouse into marts. Move it from files into tables. Move it from on-premises into the cloud. Move it from a lake into a lakehouse. Move it from one platform into another platform because the previous platform promised to solve everything, but apparently everything needed a new SKU.

I am exaggerating a bit.

Only a bit.

The logic was easy to understand. Data was scattered across systems, and scattered data is painful. If sales data lives in CRM, product data lives in ERP, finance data lives in another system, and web analytics lives somewhere else, then the business cannot ask questions across those systems without a lot of glue.

So, we built the glue.

ETL pipelines. ELT pipelines. Data warehouses. Data lakes. Lakehouses. Medallion design pattern. Data marts. Semantic models. Cubes, if you have a few gray hairs (or no hair at all) like me.

The industry became very good at moving data.

And that solved many problems. I do not want to pretend otherwise. Centralized analytical platforms brought consistency, scale, performance, governance, and a much better foundation for reporting. Without them, many organizations would still be living in spreadsheet archaeology.

But moving data into one place did not automatically make it usable.

That is the part we did not always say loudly enough.

The report is fast. The answer is slow.

There is a funny thing that happens in many data projects.

The technical team celebrates that the warehouse query now runs in 4 seconds instead of 40 minutes. And they should celebrate! That is real progress.

Then the business user says: “Great. But is this number correct?”

Silence.

Because the query may be fast, but trust is not fast. Meaning is not fast. Ownership is not fast. Security approval is not fast. The conversation about whether canceled orders should be included in net revenue is definitely not fast.

This is the less obvious truth: the speed of the database is only one part of the speed of analytics.

If the report runs in 4 seconds, but it takes 4 days to understand which table to use, which column is certified, whether the definition changed last month, and who owns the metric, then the business is not experiencing self-service.

It is experiencing a faster waiting room.

Self-service became self-navigation

I have always liked the idea of self-service analytics. Power BI, semantic models, dataflows, Fabric, Databricks, Snowflake, all of it. Give people the tools. Give them governed data. Let them answer more questions without opening a ticket every time.

Beautiful idea.

But in many organizations, self-service became something else: self-navigation.

This is the data lake. Good luck.

Here are 600 tables. Good luck.

Here are 80 semantic models, 17 of which contain a measure called Revenue. Good luck.

Here is the certified dataset, but the person who built it left the company last year. Good luck.

But, you might say: “Nikola, this is not a technology problem. This is a governance problem.”

Yes…

…And no.

It is a governance problem. It is also an architecture problem. It is also a product problem. It is also a language problem.

The business does not think in tables, pipelines, partitions, notebooks, and bronze-silver-gold layers. The business thinks in customers, orders, invoices, churn, margin, inventory, risk, claims, campaigns, and targets.

If our data platform exposes only technical objects, we should not be surprised when business users keep asking technical people to translate.

Centralization created a new bottleneck

Traditional data architecture had an obvious bottleneck: data lived in too many places.

Centralization attacked that bottleneck. Good.

Then a new bottleneck appeared.

The meaning of data lived in too many heads.

This is the part that makes logical data management interesting to me. And, it’s not because it is another architecture label. We have enough labels to wallpaper a small apartment. It is interesting because it asks a better question:

How do we manage the logic around data, not only the data itself?

That logic includes definitions, relationships, business rules, security policies, ownership, lineage, quality signals, delivery patterns, and the semantic layer that allows humans to ask questions without memorizing physical storage.

In plain English: if the data platform is the kitchen, logical data management is the menu, the recipe book, the allergy warnings, and the person who knows that “the usual” means black coffee with no sugar.

The ingredients matter.

But nobody wants to eat ingredients.

Read more

If your social media feed has been making you feel like you missed the bus on Microsoft Fabric - take a breath. You didn’t miss anything. You’ve been riding the bus the whole time. It just got a new name and a few extra stops.

I’ve been talking to a lot of Power BI developers in the last 12 months who are truly anxious about Fabric. Truth to be said, the messaging from Microsoft has been hype-heavy, and the community has been moving fast. And every time you open LinkedIn or Reddit, somebody is announcing a new Fabric feature you’ve never heard of, with a screenshot of a Lakehouse you don’t understand, talking about MLVs and Direct Lake on OneLake and Fabric IQ as if everyone already knows what those things are.

I get it. It feels like the ground is shifting under your feet.

So, the first thing I’m telling everyone (and all of you who have ever attended my trainings/live sessions already heard this from me): your Power BI skills are intact. Your reports still work. Your DAX matters more than ever. And you do not need to learn Spark this weekend.

What you DO need is a map. One that tells you what actually changed for YOU as a Power BI developer, what you can safely ignore for now, and where to start when you’re ready to take the next step. This article aims to be that map. By the way, I already wrote about Power BI in the era of Microsoft Fabric here.

What didn’t change (start here and breathe)

Before we talk about anything new, let me confirm what’s still exactly the same. Because in all the noise about Fabric, this part rarely gets said.

Power BI’s authoring experience is still the same. Whether you build models and reports in Power BI Desktop or in the browser using web modeling (which is now essentially at parity with Desktop), the workflow you know hasn’t changed. You still load data, build a model, write DAX, design a report, and publish. Same skills, same patterns.

DAX is still “the language” for semantic models, and it’s not going anywhere. If anything, it’s MORE important now - Copilot generates DAX, Direct Lake queries run on DAX, and every new AI feature in Fabric depends on well-written measures. The better your DAX, the better everything downstream works.

Power Query and the M language still handle data transformation across Desktop, web modeling, and Dataflows. And, although the old Dataflows Gen1 will be officially deprecated, the Gen2 version is here to stay, and it relies on Power Query/M skills. Your existing knowledge of Power Query patterns - merging, appending, conditional columns, parameters - all still applies.

Semantic models are still the same thing. Same VertiPaq columnar storage, same relationships, same calculated columns, same measures, same Best Practice Analyzer rules…

The Power BI service is now part of the Fabric portal, but if you log in tomorrow morning, you’ll find the same workspaces, the same reports, the same scheduled refreshes, the same Apps experience. Even the URL didn’t change.

Import mode and DirectQuery still work exactly as they did before. You don’t HAVE to use Direct Lake. If your reports run fine on Import mode with a daily refresh, that’s a perfectly valid architecture in 2026.

Row-level security, deployment pipelines, gateways, Apps - all still there. Nothing was removed.

OK, so your foundation is solid. But some things DID change, and a few of them actually matter to you. Let me walk you through the ones that are worth your time.

What changed and actually matters to you

I’ve identified five things. If you understand these five changes, you understand 90% of what’s different about being a Power BI developer in Fabric.

1. Your license changed (and it’s fine)

In January 2025, Microsoft retired the Power BI Premium P SKUs. If your organization was running on Premium - P1, P2, P3 - your environment has already been moved to Fabric capacity (F SKUs). P1 is roughly equivalent to F64. P2 is roughly F128, and so on.

The key thing to remember is that nothing broke during that transition. Reports work, semantic models work, paginated reports work, scheduled refreshes work… Your existing investment is preserved.

What you GAINED was access to the full Fabric platform under the same license - Lakehouses, Warehouses, pipelines, Real-Time Intelligence, all of it. You don’t have to use them, but they’re sitting there, included, ready when you want them.

One thing to keep in mind: below F64, report consumers still need Power BI Pro licenses to view content. At F64 and above, viewer access is included in the capacity license, which is why F64 is the commercial threshold most organizations target if they have more than 300 report consumers.

2. OneLake exists, and your data lives there now

OneLake is the unified storage layer for everything in Fabric. Think of it as “OneDrive for data.” One lake. One location. One copy.

When you create a Lakehouse, a Warehouse in Fabric, the data is stored in OneLake as Delta format - the open table format that’s become the industry standard for analytics. Your semantic models, Lakehouse tables, Warehouse tables, they all live in the same place underneath.

Why does this matter to you as a Power BI developer? Because of the next change…

3. Direct Lake is a new storage mode option

You already know two storage modes: Import (load data into the model, refresh on a schedule) and DirectQuery (query the source live every time). Each has trade-offs you’ve probably learned the hard way: Import is fast but stale, DirectQuery is fresh but slow.

Direct Lake is a third option. It reads Delta tables directly from OneLake into memory - no scheduled refresh, no live query overhead. Fast queries AND fresh data. Direct Lake is the closest thing to “you can have both” that Power BI has ever offered.

The key thing to bear in mind is that the experience of building reports doesn’t change. You still author DAX the same way, you still design visuals the same way. The mechanics underneath changed (your model is reading directly from Delta in OneLake instead of holding its own copy), but as a developer, the workflow looks the same.

When should you care about Direct Lake? If your semantic models are large (millions of rows, multi-GB models) and refresh times are painful, Direct Lake is worth investigating. If your models are small and refresh takes 30 seconds, Import mode is still perfectly fine. No urgency.

I wrote about the two flavors of Direct Lake - Direct Lake on SQL and Direct Lake on OneLake, so you may want to read that one as well.

4. Copilot is everywhere

Copilot in Power BI has expanded significantly in the last year. You’ll find it in Desktop (generates report pages, generates DAX, suggests measures, explains formulas), in the service (report-level chat, creates visuals from natural language), and on mobile (as of April 2026, conversational report exploration on your phone or tablet).

I built an entire Pluralsight course on Copilot in Power BI, so here’s my honest take: it’s useful, not magical. Copilot is really good at generating starter DAX measures, summarising what’s on a report page, and helping you ask exploratory questions of a model. It’s NOT that good at writing complex measures from scratch, understanding nuanced business logic, or working with poorly-designed semantic models.

The pattern I’ve seen is consistent: the better your semantic model, the better Copilot performs. Good table names, good column names, properly defined measures, clear relationships… If those are in place, Copilot becomes a real productivity tool. If they’re not, Copilot generates plausible-looking nonsense.

If there’s interest, I’ll write a full, honest field report on Copilot in a future post. Let me know in the comments.

What you can safely ignore (yes, I really mean it)

Now, the section that nobody else will write for you:) The Fabric ecosystem is enormous: Spark, KQL, Real-Time Intelligence, Data Factory, Data Activator, MLVs, Fabric IQ, Data Agents… And the social media-driven pressure to learn ALL of it RIGHT NOW is misplaced.

You can safely ignore most of it. I’ll try to share the breakdown:

Spark notebooks and PySpark. Unless your role is expanding into data engineering, you don’t need these. Your Power Query skills are still the right tool for Power BI data prep. If a data engineer on your team builds a Spark notebook that lands clean Delta tables in OneLake - great. You consume them. You don’t need to write them.

KQL databases and Real-Time Intelligence. Only relevant if you’re working with streaming data, IoT telemetry, or high-volume event logs. Most PBI developers aren’t. If your data lives in a database and refreshes daily, Real-Time Intelligence is not your problem.

Data Activator. Event-driven automation. Honestly, it’s very interesting, but not core to report development. File this under “explore when curiosity pulls you there.”

Fabric IQ and Data Agents. The new AI/natural language layer that lets users ask questions of their data in plain English. Worth keeping an eye on, but you don’t need to architect around it yet.

Materialized Lake Views (MLVs). Brilliant for data engineers building medallion architectures. Not something Power BI developers need to create, though you’ll happily consume the silver and gold MLVs that data engineers build for you. If you’re curious, my MLV deep dive explains them in detail. But you don’t need to read it to keep doing your job well.

The key message: learn these when your role demands it or when your curiosity genuinely pulls you toward them. Not because the algorithm told you to be scared.

The “start here” path

OK, you’re ready to take the first concrete step. Here’s the sequence I’d recommend - the one I give to Power BI developers on my team when they ask where to begin.

Step 1: Understand your capacity. Find out whether your organization is on Fabric capacity (an F SKU), Power BI Pro, or Premium Per User. This single fact determines what’s available to you.

Step 2: Explore OneLake and a Lakehouse. Create a Lakehouse in a workspace. Upload a CSV file. Watch it become a Delta table. Query it from the SQL analytics endpoint. Connect Power BI to it - from Desktop or directly in the browser, your choice. This single hands-on exercise demystifies 80% of the Fabric terminology you’ve been seeing. The best thing? It takes an afternoon.

Step 3: Try Direct Lake on a non-production dataset. Create a semantic model from your Lakehouse using Direct Lake storage mode. Build a small report. Notice that there’s no refresh button - you just publish, and the data is always current. Compare query performance to your existing Import model. Get a feel for how Direct Lake behaves.

Step 4: Let Copilot help you with something real, but verify. Open Copilot - it’s available wherever you author Power BI content. Ask it to generate a DAX measure for something specific to your model. Check its work. Iterate. Then ask Copilot to summarise a report page and see if the summary is accurate. Build your own sense of where Copilot is trustworthy and where it isn’t.

Each step takes an afternoon. Not a month or quarter.

The career question nobody wants to ask

And, to wrap this up, let me address the elephant in the room: “Is my Power BI career safe?”

Short answer: yes. But the long answer is more interesting and more nuanced. I already wrote an article about the future of Power BI from the rapid rise of AI-generated dashboards. Here, I want to focus on the Power BI future, looking through the Fabric lenses.

Power BI developers are NOT being replaced by Fabric developers. The roles are different. Power BI development - semantic modeling, DAX, report design, business-facing analytics, is a distinct and valuable skillset that the platform fundamentally depends on.

What IS changing is that the best Power BI developers in 2026 will be the ones who understand where their data comes from (OneLake, Lakehouses, Warehouses) and how it gets there. You don’t need to BUILD the pipeline. But understanding the architecture around your reports makes you significantly more effective and significantly more valuable in conversations with data engineers, architects, and stakeholders.

If you are looking for certifications as a method to prove your skills, there are a few certifications to keep on your radar. The PL-300 (Power BI Data Analyst) is still the core PBI certification and is being kept current - it’s still the foundation. The natural next step for Power BI developers moving into Fabric is the DP-600 (Fabric Analytics Engineer Associate). DP-600 bridges semantic modeling, DAX, and the broader Fabric platform from a BI perspective. If colleagues on your team are heading into data engineering, DP-700 (Fabric Data Engineer Associate) is their path - different role, different exam. For Power BI developers, PL-300 followed by DP-600 is the combination I’d recommend.

Another point that is worth saying out loud: your DAX and modeling skills are MORE valuable in Fabric, not less. Direct Lake performance depends on well-designed Delta tables and clean semantic models. Copilot performs better with clean models. Data Agents generate better answers from clean models. Fabric IQ surfaces better insights from clean models. EVERYTHING downstream depends on the quality of the semantic layer, and the semantic layer is YOUR domain.

The best career move you can make right now isn’t learning Spark. It’s getting even better at semantic modeling and DAX. Master Power BI’s modeling fundamentals, get fluent in advanced DAX patterns, learn how to design models for Direct Lake performance, and then, when you’re ready, add Fabric context around it.

Wrapping up

Fabric isn’t a replacement for Power BI. It’s the platform Power BI now lives inside. Your skills are intact. Your reports work. Your DAX matters more than ever. And you don’t need to learn the entire Fabric stack to be excellent at your job.

Start with the four steps. Take them at your own pace, ignore the parts that don’t apply to your role yet, and if you want to go deeper, I’ve been writing about Fabric architecture, Direct Lake, materialized lake views, and the Warehouse vs. Lakehouse decision - all from a working architect’s perspective.

You haven’t missed the bus. You’re on it. The destination is just a little further than you thought.

Thanks for reading!

Thanks to Claude for creating these nice-looking illustrations.

If I had a euro for every time someone asked me “should I use the Warehouse or the Lakehouse in Microsoft Fabric?” - I’d have enough to fund an F64 for a month:) And, my answer has been the same for a while now: you’re asking the wrong question.

Let me explain why - even if you were asking the right question a year ago, the answer has changed. Three things happened in the last 12 months that fundamentally shifted the calculus: materialized lake views went GA at FabCon Atlanta in March 2026, SQL databases in Fabric went GA, and the Warehouse got many limitations lifted in the meantime.

The old framing - “Lakehouse for data engineering, Warehouse for BI” - doesn’t hold anymore. In 2026, the question isn’t which one. It’s which combination, and where does each layer sit.

Let me walk you through what changed and what I’m doing differently in client engagements because of it.

The ground truth: same lake, same engine

Before we get into the shifts, let me make sure we’re on the same page about what these things actually share, because this is the most important fact and the most underappreciated one.

Both the Warehouse and the Lakehouse store data as Delta tables in OneLake. Same storage, same format, same lake. When you create a table in a Lakehouse and then query it from a Warehouse using a three-part name, no data moves. You’re reading the same Delta files from the same OneLake location.

And, both use the same SQL engine under the hood - the Polaris engine. The SQL analytics endpoint that Fabric auto-generates for every Lakehouse? That IS the Warehouse engine, just in read-only mode. So the compute is the same too.

If storage is the same and the SQL engine is the same… What’s actually different?

The write model and the transformation interface. That’s it. That’s the real fork in the road.

The Warehouse gives you full T-SQL DML: INSERT, UPDATE, DELETE, MERGE, stored procedures, views, functions - the complete toolbox that SQL Server developers have used for decades. You write data with T-SQL. You transform data with T-SQL. You serve data with T-SQL.

The Lakehouse SQL endpoint is read-only. If you want to write data to a Lakehouse, you go through Spark - PySpark notebooks, Spark SQL, Scala, or R. The Lakehouse is a Spark-first environment that happens to expose a SQL endpoint for querying.

Ok, so far so familiar. But, bear with me, now it gets interesting, because three things happened in the last year that blurred these boundaries in ways most comparison articles haven’t caught up with yet.

What changed: three shifts that broke the old framing

Shift 1: MLVs gave the Lakehouse a transformation layer

Before materialized lake views, the Lakehouse’s transformation story was: write notebooks, build pipelines to orchestrate them, configure schedules, add custom data quality logic, and wire up monitoring. Five different surfaces, five different things to debug when something broke.

The Warehouse, meanwhile, had stored procedures, views, and T-SQL - a mature, self-contained transformation story that SQL developers already knew.

That gap was real, and it was the single strongest argument for choosing the Warehouse: “use the Warehouse because it has a proper transformation layer.”

Then MLVs went GA at FabCon Atlanta. I already wrote about MLVs in my previous article. Now? The Lakehouse has declarative SQL transformations with automatic lineage inference, optimal refresh (skip / incremental / full), built-in data quality constraints, and multi-schedule support. You can build a full medallion pipeline ( bronze to silver to gold) without writing a single notebook. Just SQL statements.

The implication is significant: the Lakehouse’s transformation story isn’t better or worse than the Warehouse’s anymore - it’s just different. MLVs are declarative and dependency-aware. The engine figures out the execution order from your SQL. T-SQL stored procedures are procedural and manually orchestrated. You tell the engine what to do and in what order. Different tools for different thinking styles.

I wrote a full deep dive on MLVs recently - if you haven’t read it, start there. It covers the syntax, optimal refresh internals, the five GA changes, and the limitations you need to know.

Can it get better than this? Actually, yes.

Shift 2: SQL databases brought OLTP into Fabric

Before SQL databases in Fabric, the platform was analytics-only. If you needed an operational data store, a reference data hub, or a transactional backend for a Power App, you went to Azure SQL Database outside Fabric. Your OLTP and OLAP worlds lived on separate planets. You built pipelines to move data between them.

If you want to learn more about SQL database in Fabric, I have you covered in this article.

Then Fabric databases went GA in late 2025. SQL database in Fabric is essentially a SaaS Azure SQL DB that lives inside Fabric, backed by the same SQL Server engine that’s been trusted for 30+ years. But, with one crucial enhancement: it automatically mirrors its data to OneLake as Delta, making it instantly available for analytics across Fabric, without you building a single pipeline.

I covered this in detail on my blog, but the short version is: the old two-player game (Warehouse vs. Lakehouse) is now a three-player game. SQL database handles the use cases that neither Warehouse nor Lakehouse was designed for: high-frequency writes, OLTP workloads, reference data management, operational data stores, and even reverse ETL scenarios where curated analytics data gets pushed back into operational systems.

I hear you, I hear you: Nikola, isn’t it confusing to have yet ANOTHER data store in Fabric? I thought the whole point was simplification! And, you’d be right to feel that way. But think of it like this: the stores aren’t competing with each other. They’re specialized. The Lakehouse handles batch engineering and declarative transformations. The Warehouse handles T-SQL serving and procedural logic. The SQL database handles transactional operations. All three sit on the same lake, all three store Delta, and all three can query each other using three-part names.

And then there’s the third shift...

Shift 3: The Warehouse got real developer ergonomics

This one flew under the radar because it wasn’t a single headline announcement. It was a steady accumulation of features across the January-April 2026 updates that collectively made the Warehouse feel like a proper development environment:

The Warehouse now supports explicit user transactions for ALTER TABLE (April 2026). Schema migrations that combine ALTER TABLE with other DDL/DML can be executed atomically in a single transaction. Previously, ALTER TABLE couldn’t participate in explicit transactions, which meant schema changes were risky and non-atomic. That’s fixed now.

Git integration got meaningfully better: branched workspaces give developers clearer visual cues when working with feature branches. Selective branching lets you pick only the items you need for a feature, reducing workspace clutter. And notebook auto-binding means notebooks resolve their Lakehouse references automatically as they move across Git-connected workspaces (dev -> test -> prod).

Cross-database queries work across Warehouses, Lakehouses, SQL databases, and mirrored databases using standard three-part naming. One SELECT can join a dimension from your Warehouse with a fact table in your Lakehouse. No shortcuts needed, no data movement.

And SQL Pool Insights now gives you pool-level telemetry for monitoring and troubleshooting - tracking pressure events, configuration changes, and capacity utilization over time.

The implication: the Warehouse is no longer “just” a query engine. It’s becoming a proper development platform with CI/CD support, transactional DDL, cross-store queries, and observability built in.

So if all three stores are getting more capable and the boundaries are blurring, how do you actually decide?

The decision framework - 4 questions

I don’t use comparison tables in client engagements anymore. There are 50 of those on the internet already, and they all say the same thing. Instead, I ask four questions. The answers tell me which combination fits.

Read more

For the longest time, building a medallion architecture in Microsoft Fabric meant stitching together a small orchestra of moving parts: notebooks for the transformations, pipelines for orchestration, schedules for refresh, custom code for data quality checks, and the Monitor Hub for keeping an eye on whether anything actually worked. Every layer worked - until something didn’t, and then you had to figure out which layer broke, why, and which downstream layers got affected along the way.

If you’ve ever tried to debug a silver layer that didn’t update because the bronze notebook failed three hours ago, you know exactly what I’m talking about.

Then, at FabCon Atlanta in March 2026, materialized lake views (MLVs) went generally available. And the story they’re telling is simple: what if your entire medallion pipeline could be a few SELECT statements?

Let me walk you through the whole thing - what they are, how they work, what changed between preview and GA, and where they fit (and where they don’t) in your architecture.

Materialized Lake View - WHAT?

A materialized lake view is a persisted, automatically refreshed view defined in Spark SQL or PySpark. You write a SELECT query that describes the transformation you want, and Fabric takes care of execution, storage, refresh, dependency tracking, and data quality enforcement.

The result is stored as a Delta table in your lakehouse. So downstream consumers, such as Power BI Direct Lake, Spark notebooks, SQL endpoints, can query it just like any other Delta table. No special handling, no different syntax.

To put it in plain English: an MLV is nothing else but a SELECT statement that learned to materialize itself, manage its own dependencies, schedule its own refresh, and check its own data quality.

OK, that’s nice. But what does that actually replace?

That’s a fair question. Before MLVs, building a single bronze-to-silver-to-gold flow looked roughly like this: you’d write a notebook for each transformation, set up a Data Factory pipeline to call them in the right order, configure schedules, build custom validation logic, and then wire up the Monitor Hub to watch for failures. Five different surfaces, five different things to debug when something breaks.

With MLVs, all of that collapses into declarative SQL. You describe what you want. Fabric figures out the rest.

The four stages of an MLV’s life

Every MLV moves through four stages. According to the Microsoft documentation, understanding them is the foundation for everything else:

1. Create - You write the Spark SQL (or PySpark) that defines the transformation. Fabric stores the definition and materializes the initial result as a Delta table.

2. Refresh - When source data changes, Fabric chooses the optimal strategy: incremental (process only changes), full (rebuild), or skip (no changes detected).

3. Query - Any application or tool reads the materialized result. They don’t know - and don’t need to know - that it’s an MLV.

4. Monitor - Refresh history, execution status, data quality metrics, and lineage are all tracked and visualised natively in Fabric.

Now let’s dive into each piece.

Create: the syntax

Here’s the full Spark SQL pseudo-code syntax for creating an MLV, straight from the Microsoft Learn reference:

CREATE [OR REPLACE] MATERIALIZED LAKE VIEW [IF NOT EXISTS]

[workspace.lakehouse.schema].MLV_Identifier

[(CONSTRAINT constraint_name CHECK (condition) [ON MISMATCH DROP | FAIL], ...)]

[PARTITIONED BY (col1, col2, ...)]

[COMMENT “description”]

[TBLPROPERTIES (”key1”=”val1”, ...)]

AS select_statement

A real example - cleaning order data joined from products and orders, with a data quality constraint and partitioning:

CREATE OR REPLACE MATERIALIZED LAKE VIEW silver.cleaned_order_data

(

CONSTRAINT valid_quantity CHECK (quantity > 0) ON MISMATCH DROP

)

PARTITIONED BY (category)

COMMENT “Cleaned order data joined from products and orders”

AS

SELECT

p.productID, p.productName, p.category,

o.orderDate, o.quantity, o.totalAmount

FROM bronze.products p

INNER JOIN bronze.orders o ON p.productID = o.productID

Two things worth flagging right away. First, MLV names are case-insensitive (MyView becomes myview). Second, all-uppercase schema names (like MYSCHEMA) aren’t supported, so use either mixed or lowercase.

You also need a schema-enabled lakehouse and Fabric Runtime 1.3 or higher. If your lakehouse doesn’t have schemas turned on, MLVs aren’t available, that’s the very first prerequisite.

Refresh: the brain of MLVs

Here’s where MLVs stop being clever and start being smart.

When source data changes, Fabric’s optimal refresh engine looks at every MLV in the lineage and asks a series of questions: Did anything actually change? Can I process just the changes? Or do I need to rebuild from scratch?

Three possible outcomes:

• Skip refresh - source data hasn’t changed. Don’t waste compute. Move on.

• Incremental refresh - process only the new or changed rows. Fast, cheap, ideal.

• Full refresh - rebuild the whole thing. Slowest path, used when incremental isn’t safe or possible.

But, and this is important, incremental refresh isn’t free. It has prerequisites:

1. The Delta change data feed (CDF) must be enabled on every source table referenced by the MLV (delta.enableChangeDataFeed=true).

2. The source must be a Delta table. Non-Delta sources always get a full refresh.

3. The data must be append-only. If your source has updates or deletes, Fabric falls back to a full refresh.

4. The query must use only supported SQL constructs (more on this in a moment).

Without CDF enabled, optimal refresh can only choose between skip and full. With CDF on, the full incremental path opens up. Enabling CDF on your source tables has no measurable storage or performance impact for append-only workloads, so there’s very little reason not to turn it on:

ALTER TABLE bronze.orders SET TBLPROPERTIES (delta.enableChangeDataFeed = true);

ALTER TABLE bronze.products SET TBLPROPERTIES (delta.enableChangeDataFeed = true);

Can it get better than this? Actually, yes! And, this is where the GA story really begins.

What’s new in GA - The FabCon Atlanta story

MLVs were introduced in preview at Build 2025. Between then and GA in March 2026, Microsoft closed the most important gaps. Five major changes turned MLVs from “interesting” into “production-ready”:

1. Multi-schedule support

2. Broader incremental refresh coverage

3. PySpark authoring (preview)

4. In-place updates with Replace

5. Stronger data quality controls

Let me take them one at a time.

1. Multi-schedule support

In preview, you could only refresh all MLVs in a lakehouse on a single schedule. Need finance to update hourly, but analytics to update every six hours? You had to work around it with notebooks, which broke dependency awareness, error reporting, and retry logic. Notebook-triggered refreshes don’t surface MLV error details. Failures appear only in cell output, and dependent views have no awareness of them. Errors can persist week after week without anyone knowing the pipeline is broken.

Now you can define named schedules within a single lakehouse, each targeting a specific subset of views. Finance pipeline hourly. Analytics every six hours. Marketing every 15 minutes. All in the same lakehouse, no custom code.

Read more

You’ve probably spent the last year hearing about Lakehouses and Warehouses until your ears are ringing. But today, we’re talking about another house in Microsoft Fabric – the Eventhouse.

I’ve recently had a chance to work with a large client on the Real-Time Intelligence project implementation, and was amazed by the power of Eventhouse!

If a Lakehouse is a massive library and a Warehouse is a high-end bank vault, the Eventhouse is a 24-hour Air Traffic Control Tower. In technical terms, an Eventhouse is a logical container tailored for streaming data. It’s built on the Kusto (KQL) engine. While the rest of Fabric is busy thinking about ‘data at rest’ (data that’s sitting in a file waiting to be read), the Eventhouse is obsessed with ‘data in motion’. It’s where your logs, your IoT sensor data, and your real-time clicks live. It doesn’t just store data – it catches it mid-air.

“The newspaper” analogy

You might be thinking, ‘Can’t I just put my sensor data in a Lakehouse?’ Well, sure you can. But there’s a massive catch: latency.

Think about it like this:

• The Warehouse/Lakehouse approach is like reading the morning newspaper. It’s great! It’s detailed, it’s curated, and it tells you exactly what happened yesterday. But if your house is currently on fire, a newspaper delivered tomorrow morning isn’t going to help you much.

• The Eventhouse approach is the live smoke alarm. It’s scanning the air every second. It doesn’t care about the history; it cares about what is happening right now.

If you have a million rows of data coming in every minute from a factory floor, you don’t want to wait for a Spark job to spin up, ‘shuffle’ the data, and write a Delta table. You want to query it the millisecond it hits the wire. That is why we need an Eventhouse.

The difference in architecture

Warehouses and Lakehouses are built for throughput (moving big chunks of data at once). The Eventhouse is built for concurrency and low latency (handling thousands of tiny ‘events’ coming in at the same time and letting you query them instantly).

The reason Eventhouse is so fast is the engine: KQL (Kusto Query Language). If SQL is a precise surgical scalpel, then KQL is a high-powered leaf blower. It is designed specifically to scan through billions of rows of text and logs in seconds.

In an Eventhouse, you aren’t worried about complex joins between 50 tables. You’re doing ‘Search’ and ‘Filter’ across massive streams. It’s built to index everything by default, so when you hit ‘Run,’ the answer is there before you’ve finished clicking the mouse.

So, the rule of thumb for your next project:

1. If your data is coming in batches and you want to do standard reporting, go Warehouse.

2. If you have massive raw unstructured data, such as PDFs, images, video, or audio files, and you want maximum flexibility, go Lakehouse.

3. If your data is streaming, comes in high volumes, and you need to react in seconds, go Eventhouse.

Additionally, the big differentiator between the Lakehouse/Warehouse vs. Eventstream approach could be: use Lakehouse/Warehouse for analytical workloads, and use the Eventhouse for operational workloads.

It’s not about which one is ‘better’ – it’s about choosing the right tool for the speed of your business.

Eventhouse vs. KQL Database

This is a point of confusion for a lot of people because, when you look at the workspace, they look very similar. But there is a clear parent-child relationship here.

Think of it like this: The Eventhouse is the apartment building, and the KQL Database is the apartment.

The Eventhouse is the high-level container. It’s the architectural “host.” When you create an Eventhouse, you aren’t just creating a place to put data, you are also setting up the compute and management layer.

• Shared resources: The Eventhouse manages the “capacity” (resources) for everything inside it.

• Centralized management: You use the Eventhouse to manage things like data ingestion, networking, and security at scale across multiple KQL databases.

• Think of it: The building provides the electricity, the water, the security guard at the front desk, and the overall structure. Without the building, the individual apartments have nowhere to exist.

The KQL Database is where the actual work happens. This is where your data is stored and where your queries run.

• Storage and tables: This is where you create your tables, your functions, and your materialized views.

• Data isolation: You might have one KQL Database for “Production Logs” and another for “Test Logs” inside the same Eventhouse.

• Think of it: This is your actual living space. It has your furniture (tables) and your rules (functions). You can have many apartments in one building, but they all share the building’s foundation.

Eventhouse OneLake availability

You’ve probably heard about the OneCopy principle in Fabric, right? This means, we create a single copy of the data in OneLake, and all the Fabric engines can read the same copy of the data. This principle is based on the unified, open table format, called Delta Parquet. However, when you store the data in Eventhouse, it’s not stored in Delta Parquet format. Which defeats the purpose of the OneCopy principle, right?

Well, not really. There is a special feature, called Eventhouse OneLake availability, that allows you to automatically create a logical copy of the Eventhouse data in Delta Parquet format, and then query this data using any other Fabric workload, including Direct Lake storage mode for Power BI semantic models. This feature can be enabled on both the entire KQL database level and on the individual table level.

Key strengths

Let me now quickly introduce some of the key strengths of the Eventhouse in Microsoft Fabric. In this article, I’ll just briefly introduce them, and then I plan to cover them in more detail in the future.

Shortcuts

I’ve already written about shortcuts in Microsoft Fabric. As a reminder, a shortcut is a OneLake object that points to another storage location. Essentially, it allows you to create a virtual copy of the data without physically moving it from its original location.

Shortcuts in a KQL database conceptually work the same as in the Lakehouse, but there are also some specifics when it comes to their physical implementation. As already mentioned, I’ll cover KQL database shortcuts in detail in one of the next articles.

Eventhouse endpoint for Lakehouse/Warehouse

The Eventhouse endpoint is a capability in Microsoft Fabric that lets users query lakehouse or warehouse data in a high-performing and real-time querying way, using both KQL and SQL.

Auto-indexing

One of the biggest headaches in database history is indexing. In the old days, you had to guess which columns people would query and build indexes manually. If you missed one, your query would crawl.

In an Eventhouse, everything is indexed by default. Because it’s built on the Kusto engine, it indexes data as it arrives. You don’t ‘tune’ an Eventhouse, you just feed it.

Partitioning

Then there’s partitioning. Usually, you have to decide: “Do I partition by year? By month? By department?” If you get it wrong, moving data is a nightmare.

The Eventhouse handles partitioning based on ingestion time automatically. It’s like having a closet that reorganizes itself every night so the clothes you wear most often are right in front.

And the compression? It’s amazing. Eventhouse automatically squishes data down.

Native time-series

Now, let’s talk about time series. Most databases treat time like just another number. But the Eventhouse understands time.

If you want to see a trend, find an anomaly, or predict where a sensor reading is going, you usually need a math degree and 200 lines of Python. In Eventhouse, we use KQL, which has “Time Series” baked into its DNA.

Standard SQL is like looking at a series of polaroid photos. You have to flip through them one by one to see what changed. Eventhouse is like having a video editing suite. You can use one command like make-series to fill in missing data points, or series_decompose to strip away the “noise” and see the actual trend. It’s built to answer “What’s normal?” and “When did it stop being normal?” in milliseconds.

Complex JSON and XML parsing

Let’s wrap it up by talking about the messy stuff: nested JSON and XML. We’ve all seen those JSON files that look like a giant tangled ball of Christmas lights – arrays inside objects inside other arrays.

In a traditional warehouse, you’d spend three days writing a flattening script. The Eventhouse has a feature called Dynamic Columns. You just dump the JSON in. You don’t even have to define the schema first! It’s like a high-end wood chipper. You throw in a whole tree (a complex, nested JSON object), and the Eventhouse shreds it into perfectly organized wood chips (columns) that you can query immediately. You want to reach deep into a nested array? You just use a simple dot notation (like Log.Details.Sensor.Value). It’s that easy.

Stay tuned for more about Eventhouse and Real-Time Intelligence workloads in Microsoft Fabric!

Thanks for reading!

Your data model isn't about tech specs. It's about thinking like a business. Consider it the blueprint for your entire analytics house. If the blueprint's chaotic, your house crumbles. If it's structured and organized, your team finds insights fast.

You’re staring at a spreadsheet full of customer orders, product prices, and sales dates. It’s messy. Your “dashboard” is slow. You’ve tried to answer a simple question like: How much revenue did pizza make last quarter? and ended up with numbers that don’t add up. Why? Because your data model is a pure mess.

In this blog post - adapted from a chapter of my book Analytics Engineering with Microsoft Fabric and Power BI, which I’m writing with Shabnam Watson for O’Reilly, I’ll walk you through the core data modeling concepts that every analytics engineer should know. Forget Power BI and Microsoft Fabric for a second. This is about core principles - the why behind models. These ideas work regardless of the tool.

Let’s start by introducing the challenge. Imagine you run a tiny pizzeria. Your “database” is a single Excel sheet: Order ID, Customer Name, Address, Pizza Type, Quantity, Price. Looks simple enough, right? The problem? John Smith’s address is repeated in every single order. If he moves, you’ve got to edit 37,000 rows of orders just to update his address. Doesn’t look smart, right?

Your data model is the fix. It literally says: Customers live in their own table. Orders link to Customers without copying addresses. It’s not about how you’re going to visualize the data. It’s about organizing it so the data makes sense when you ask questions.

However, data modeling begins long before your data is stored in a spreadsheet or in a real database. In the following sections, we will introduce core data modeling concepts - the ones that should be implemented in every single data modeling scenario, regardless of the modeling approach you plan to take or the tool you are going to use for the physical implementation.

Just as an architect doesn’t go straight from an idea to a finished building, a data modeler doesn’t create a database schema in a single step. The process progresses through three levels of increasing detail, each serving a distinct purpose and audience. Think of it as progressing from a conceptual sketch, to a detailed architectural blueprint, to a final construction plan used by the builders.

Conceptual model: The napkin sketch

Every great data model begins not with code or tables, but with a conversation. The conceptual model is the very first, highest-level view of your data. It’s completely non-technical and focuses solely on understanding and defining the business concepts and the rules that govern them.

The conceptual model identifies the main things or entities a business cares about and how they relate to each other. It creates a vocabulary for talking with business stakeholders to ensure you’re speaking the same language.

Imagine an architect meeting with a client at a coffee shop. The client might say something like: I want a family home that feels open and connected. The architect takes a napkin and draws a few bubbles: Kitchen, Living Room, Bedrooms, and draws lines between them labeled “connects to” or “is separate from.” There are no dimensions, no materials, no technical details. It’s just about capturing the core idea and ensuring that everyone agrees on the fundamental concepts. That napkin sketch is the conceptual data model.

Let’s look at a real example - events at a stadium. In a conceptual model for this scenario, you’d identify several entities: Stadium, Event, Customer, Attendee, and Ticket. You’d also notice how these entities are interconnected. This high-level overview provides a simplified picture of the business workflow within the organization.

A Stadium has a name and is located in a specific country and city, which uniquely identifies it. This Stadium may host many events, and there can be many attendees coming to these events. An Event cannot exist outside of the Stadium where it is scheduled. An event can be attended by an attendee, and there can be many attendees for one event. An Attendee is the entity that attends the event. They can also be a Customer of the Stadium entity - for example, by visiting a stadium museum or buying at a fan shop - but that doesn’t make them an attendee of a specific event. Finally, a Ticket represents confirmation that the attendee will attend a specific event. Each Ticket has a unique identifier, and one Attendee can purchase multiple Tickets.

Now, you might be wondering: why is this important? Why should someone spend time and effort describing all the entities and relations between them?

Remember, the conceptual data model is all about building trust between business and data personas - ensuring that business stakeholders will get what they need, explained in a common language, so that they can easily understand the entire workflow. Setting up a conceptual data model also provides business stakeholders with a way to identify a whole range of business questions that need to be answered before building anything physical. Questions like: Are the Customer and Attendee the same entity (and why are they not)? Can one Attendee buy multiple Tickets? What uniquely identifies a specific Event?

Additionally, the conceptual data model depicts very complex business processes in an easier-to-consume way. Instead of going through pages and pages of written documentation, you can take a look at the illustration of entities and relationships, all explained in a user-friendly way, and quickly understand the core elements of the business process.

Logical model: The blueprint

Once business and data teams align on the conceptual data model, the next step is designing a logical data model. In this stage, we are building upon the previous step by identifying the exact structure of the entities and providing more details about the relationships between them. You should identify all the attributes of interest for each entity, as well as relationship cardinality.

Note that, just as during the conceptual data modeling phase, we’re still not talking about any specific platform or solution. The focus is still on understanding business requirements and how these requirements can be efficiently translated into a data model.

There are several steps to ensure that the conceptual model successfully evolves into a logical model. You need to identify entity attributes - the specific data points each entity should contain. Then identify candidate keys - which attribute, or set of attributes, uniquely identifies a specific entity. From there, choose primary keys based on the findings from the previous step. You’ll also apply normalization or denormalization as appropriate (more on that later). Next, set relationships between entities, validating how entities interconnect and, if needed, breaking down complex entities into multiple simpler ones. Then identify the relationship cardinality — defining how many instances of one entity relate to instances of another. There are three main types: one-to-one (1:1), one-to-many (1:M), and many-to-many (M:M). Finally, and critically, iterate and fine-tune. In real life, it’s almost impossible to find a data model that suits everyone’s needs immediately. Ask for feedback from business stakeholders and fine-tune the logical data model before materializing it in physical form.

The potential gains of the logical model are significant. First, it serves as the best quality assurance test, identifying gaps and issues in understanding the business workflow, thereby saving you a significant amount of time and effort in the long run. It’s much easier and less costly to fix issues at this stage, before locking into a specific platform. Building a logical data model can be considered part of the agile data modeling cycle, which ensures more robust, scalable, and future-proof models. And ultimately, it serves as a blueprint for the final physical implementation.

Physical model: The construction plan

A physical data model represents the final touch: how the data model will actually be implemented in a specific database. Unlike conceptual and logical data models, which are platform and solution-agnostic, physical implementation requires defining low-level details that may be specific to a certain database provider.

There’s a whole list of necessary steps to make your physical data model implementation successful. You need to choose the platform - this decision shapes your future design principles. Then translate logical entities into physical tables - since a real database doesn’t support the abstract level of a logical entity, you need to define the data type of each attribute: whole number, decimal number, or plain text. Additionally, each physical table should rely on keys (primary, foreign, unique) to ensure data integrity.

You must also establish relationships based on the key columns. Apply normalization or denormalization as appropriate - remember, in OLTP systems, tables should be normalized (typically to 3NF) to reduce redundancy and support write operations efficiently, while in OLAP systems, data may be denormalized to eliminate joins and make read operations more performant.

Define table constraints to ensure data integrity - not just keys, but logical checks too. For example, if your table stores student grades in the range of 5 to 10, why not define that constraint on the column, preventing the insertion of nonsensical values?

Create indexes and/or partitions - these are special physical data structures that increase the efficiency of the data model. Table partitioning, for example, splits one big table into multiple smaller sub-tables, reducing scanning time during query execution. A classic approach is partitioning by calendar year. And finally, extend with programmatic objects - stored procedures, functions, triggers — that are de facto standard in almost every data platform solution.

The main benefit of the physical data model is to ensure efficiency, optimal performance, and scalability. When we talk about efficiency, we have in mind the two most precious business assets: time and money. Unless you think that time = money, then you have only one asset to consider. The more efficient your data model, the more users it can serve, the faster it can serve them, and that, in the end, brings more money to the business.

Why bother with all three? Because fixing a gap in the conceptual model costs a conversation. Fixing it in the physical model costs a sprint. The earlier you catch issues, the cheaper they are to resolve.

OLTP vs. OLAP: Writing vs. Reading

Online Transaction Processing Systems (OLTP)

To be a successful analytics engineer, you must first understand where your data comes from. The vast majority of business data isn’t created for analytics. It’s created by applications that run the daily operations of the business: a point-of-sale system, a customer relationship management (CRM) tool, an e-commerce website’s backend database, and many more.

These source systems are called online transaction processing (OLTP) systems. They are designed and optimized for one primary goal: to process a high volume of transactions quickly and reliably. OLTP systems need to instantly confirm a customer’s order or update their shipping address. Speed and data integrity for writing data are paramount.

To achieve this, OLTP systems use a relational data model that is highly normalized. Let’s dive deep into what that actually means.

Normalization: The library card catalog

Normalization is the process of organizing data in a database to minimize data redundancy and improve data integrity. In simple terms, it means you don’t repeat information if you don’t have to.

Imagine a library in the pre-computer era. Each and every book has an index card. If the author’s full name, nationality, and date of birth had to be written on every card for every book an author wrote, it would be tedious. You’d be writing “William Shakespeare, English, 1564-1616” on the cards for Hamlet, Macbeth, and Romeo and Juliet. And if you discovered a mistake in Shakespeare’s birth year, you’d have to find and correct every single card for every book he wrote. It’s almost guaranteed you’d miss one.

A smart librarian would use normalization. They would create a separate Authors card catalog. The card for Hamlet would just say “Author ID: 302.” You would then go to the Authors catalog, look up ID 302, and find all of William Shakespeare’s details in one single place. If you need to make a correction, you only have to do it once.

Normal Forms

This is the essence of normalization: breaking data down into many small, discrete tables to avoid repeating ourselves. The rules for doing this are called normal forms (1NF, 2NF, 3NF...). There are seven normal forms in total, although in most real-life scenarios, normalizing data to the third normal form (3NF) is considered optimal.

Let’s briefly break down the key principles behind the first three normal forms. First normal form (1NF) eliminates repeating groups. Each cell should hold a single value, and each record should be unique. Second normal form (2NF) builds on 1NF and ensures that all attributes depend on the entire primary key - this is mainly relevant for tables with composite keys. Third normal form (3NF) builds on 2NF and ensures that no attribute depends on another non-key attribute. This is the library example: AuthorNationality doesn’t depend on the book; it depends on the author. So, you move AuthorNationality to the Authors table.

Let’s look at a before-and-after example. Imagine a non-normalized spreadsheet for tracking orders: OrderID, OrderDate, CustomerID, CustomerName, CustomerCity, ProductID, ProductName, Qty, UnitPrice - all in one flat table. Notice all the repetition? John Smith’s name and city are repeated. Widget A’s name and price are repeated. To update Widget A’s price, you have to change it in two places, and that’s just in a tiny sample.

To normalize this data to 3NF, we break it down into four separate tables: a Customer table (CustomerID, CustomerName, CustomerCity), a Product table (ProductID, ProductName, UnitPrice), an Orders table (OrderID, OrderDate, CustomerID), and an OrderDetails table (OrderID, ProductID, Qty). Now if John Smith moves to Los Angeles, we update his city in exactly one place. If the price of Widget A changes, we update it in exactly one place. This is perfect for the OLTP system.

However, life is not a fairy tale. And here’s the twist. While this normalized structure is brilliant for writing data, it’s inefficient for analyzing it. To answer a simple question like “What is the total sales amount for products in the ‘Widgets’ category to customers in New York?” you’d have to perform multiple complex JOIN operations across all these little tables. With dozens or even hundreds of tables, these queries become incredibly slow and a nightmare for your business users to write.

This leads us to the core job of an analytics engineer: transforming data from a model optimized for writing (OLTP) to one optimized for reading (OLAP).

Online Analytical Processing Systems (OLAP)

If OLTP systems are for running the business, online analytical processing (OLAP) systems are for understanding the business. Our main goal as analytics engineers is to build OLAP systems. These systems are designed to answer complex business questions over large volumes of data as quickly as possible.

Denormalization: The strategic reversal

Let’s kick it off by explaining denormalization. As you may rightly assume, with denormalization we are strategically reversing the process of normalization that we previously examined. We intentionally re-combine many small tables into a few larger, wider tables, even if it means repeating some data and creating redundancy.

Denormalization is essentially a trade-off: we are sacrificing a bit of storage space and update operation efficiency to obtain potentially massive gains in query performance and ease of use. Denormalization is one of the core concepts for implementing dimensional data modeling techniques, covered next.

Dimensional modeling: The Star schema and beyond

A dimensional model represents the gold-standard paradigm when designing OLAP systems. Before we explain the dimensional aspect, let's have a brief history lesson. Ralph Kimball's book The Data Warehouse Toolkit (Wiley, 1996) is still considered a dimensional modeling bible. In it, Kimball introduced a completely new approach to modeling data for analytical workloads - the so-called bottom-up approach. The focus is on identifying key business processes within the organization and modeling these first, before introducing additional business processes.

Kimball’s approach is elegant in its simplicity. It consists of four steps, each based on a decision:

Read more

If you’ve been working in enterprise data for a while, you’ve probably experienced the following scenario: the marketing team calls a customer anyone who’s ever downloaded a whitepaper from your company’s website. Sales defines a customer as someone with an active contract. And finance? They only count someone as a customer after the first payment clears. Same word, three different meanings, and chaos in every cross-functional dashboard. Welcome to the world of semantic drift - the silent killer of data initiatives that Microsoft Fabric IQ is designed to eliminate.

Announced at Microsoft Ignite 2025, Fabric IQ represents a fundamental shift in how Microsoft Fabric operates. While the rest of the platform has been laser-focused on where your data lives (lakehouses, warehouses, eventhouses), Fabric IQ is obsessed with what your data means.

Ontology at the core

At its core sits the Ontology item - think of it as the Rosetta Stone for your business. An ontology is a shared, machine-understandable vocabulary that defines the things in your environment (represented as entity types), their facts (represented as properties), and the ways they connect (represented as relationships). If you’re thinking “this sounds like a fancy data model,” you’re not far from the truth. However, there is one critical difference: unlike a data model that lives in a single Power BI semantic model or database schema, an ontology sits above all your physical data sources and provides a single source of truth that every tool, every analyst, and every AI agent can reference.

The following illustration shows the simplified overview of the Fabric IQ architecture:

The power of Binding

The power of Fabric IQ happens through a concept called binding. When you create an ontology, you’re not just drawing boxes and arrows in a diagram - you’re rather connecting those abstract business concepts to real data living across your OneLake ecosystem. A single entity type like Customer can be bound to columns from your lakehouse tables, eventhouse streams, and even existing Power BI semantic models. The ontology handles the heavy lifting: it maps data types, defines identity keys, describes how columns map to properties, and establishes how those keys create relationships across multiple sources. This means when your marketing automation system, your CRM, and your billing platform all have slightly different customer records, the ontology can reconcile them into a single, coherent business entity. Updates in upstream sources require manual refresh, but once refreshed, every downstream consumer, whether it’s a Power BI report, a Fabric Data Agent, or an Operations Agent, sees the same unified view.

You might be wondering: “Can’t I just standardize my data upfront in a data warehouse and call it a day?” Well, sure you can. But, that only works if you control every single data source and can enforce perfect consistency at ingestion time. In the real world, you’re dealing with acquired companies running different systems, third-party APIs with their own schemas, and legacy databases that predate your entire career. The ontology approach says: “Let the data be messy at the source level, but enforce semantic consistency at the consumption layer.” It’s the difference between trying to make everyone speak the same language versus having a really good translator.

What makes Fabric IQ particularly powerful for Real-Time Intelligence workloads is its integration with Graph capabilities in Microsoft Fabric. While traditional analytics asks “what happened?” and “why did it happen?”, graph-enabled ontologies let you traverse complex relationships to answer questions like “Which stores have fewer ice cream sales when their freezer temperature rises above -18°C?” or “Find all shipments exposed to risky routes and the related cold chain breaches that resulted.”

The ontology declares which things connect and why, while the Graph engine stores and computes those traversals with sub-second performance. This isn’t just about drawing pretty network diagrams. It’s more about enabling AI agents and analysts to follow cause-and-effect chains across domains that traditional SQL joins simply can’t handle efficiently.

The real-world applications of Fabric IQ are already emerging across industries. Some companies are leveraging ontology in Fabric IQ to unify transmission and distribution grid data, breaking down the silos created by traditional relational databases. By structuring relationships between assets, events, and operational domains, such as weather systems, outage management systems, GIS data, and telemetry data, they’re building a foundation for advanced forecasting and risk evaluation.

In the retail sector, companies are using ontologies to link sales transactions, supply chain events, IoT sensor data, and customer behavior into unified business entities that Operations Agents can monitor in real-time. When a freezer temperature sensor crosses a threshold, the agent doesn’t just send an alert - it understands the downstream implications for inventory spoilage, sales impact, and customer satisfaction, because all those concepts are explicitly modeled in the ontology.

Despite its emerging popularity, Fabric IQ is not a silver bullet. If your organization struggles to agree on basic definitions now, an ontology won’t magically create consensus. What it will do is make those disagreements painfully obvious. The power comes when you commit to treating the ontology as a living, governed artifact with clear ownership, approval workflows, and version control. When that happens, you unlock benefits that cascade through your entire data ecosystem: consistent semantics across tools, fewer one-off models, less duplicated logic, and most importantly, AI agents that can move beyond answering questions to taking safe, auditable actions.

Because when business rules and constraints live in the ontology rather than scattered across 47 different Power BI semantic models and Fabric notebooks, your AI systems finally have the context they need to not just be helpful, but to actually understand your business.

Build your first Ontology

Let’s quickly see Fabric IQ in action. In this section, we will build our first Ontology. At the moment of writing, there are two options for creating an ontology:

• Generate Ontology from semantic model - automatically generates an ontology on top of the existing Power BI semantic model

• Build directly from OneLake - manually create the ontology by binding properties directly from OneLake

In our example, we will build an Ontology directly from OneLake. Before we start, there are certain prerequisites - you must enable specific settings on your Fabric tenant. All these settings can be found under the Tenant settings tab of the Admin portal.

• Enable Ontology item

• User can create Graph

• Allow XMLA endpoints and Analyze in Excel with on-premises semantic models - required only if you want to generate an ontology from a semantic model

The following settings are required only if you want to use ontology with a Fabric Data agent:

• Users can create and share Data agent item types

• Users can use Copilot and other features powered by Azure OpenAI

• Data sent to Azure OpenAI can be processed outside your capacity’s geographic region, compliance boundary, or national cloud instance

• Data sent to Azure OpenAI can be stored outside your capacity’s geographic region, compliance boundary, or national cloud instance

For this example, we will need some sample data. You can download all the necessary files from here.

Next, let’s make this data available in the lakehouse. Let’s upload the files to the Files section of the lakehouse, and then transform them to Delta tables in the Tables section, as shown below. We will do that for all the files except the FreezerTelemetry file.

Next, we will create a new Eventhouse in the same Fabric workspace. Once the eventhouse is created, click on the three dots next to the default KQL database, choose Get data, and then Local file, as illustrated below:

Read more

Before we dive into the technical details, let’s talk about what data architecture really is. At its core, data architecture is just the set of rules, standards, and patterns that govern how data flows through your organization. It’s the blueprint that determines the following:

• Where different types of data live

• How data moves from system to system

• What happens to data as it gets transformed

• Who has access to what data

• How data quality is maintained

• How new data sources get integrated

Think of it as the layout of a city. You have residential areas, commercial districts, industrial zones, and transportation networks that connect everything together. Each zone serves a specific purpose, and the way they’re connected determines how efficiently the city functions.

In the data world, your architecture determines whether your organization is like a well-planned city with efficient highways and clear zoning, or like a metropolis that grew without any planning and now has traffic jams everywhere.

Here’s a story that illustrates this perfectly. We once worked with a company that had grown rapidly through acquisitions. They had inherited data systems from each company they bought, and nobody had ever taken the time to think about how all of it fit together.

Their data “architecture” looked something like this:

• Customer data residing in five different CRM systems

• Financial data split between three different ERPs (Enterprise Resource Planning)

• Marketing data scattered across dozens of disparate solutions

• Each system had its own definition of basic concepts like customer and revenue

• Getting answers to simple questions required manually combining data from multiple sources

The outcome? Their “weekly” business review actually took two weeks to prepare because analysts spent most of their time just figuring out how to join the data together. Leadership couldn’t make quick decisions because they never trusted the numbers. And the data team was constantly putting out fires instead of providing strategic value.

Share Head First Data

Six months later, after implementing a well-thought-out data architecture, the same company could do all of this:

• Generate their weekly business review in under two hours

• Answer ad-hoc questions from leadership in minutes instead of days

• Trust their metrics enough to base major business decisions on them

• Onboard new data sources in days instead of months

The difference wasn’t the latest technology or huge budgets. It was simply having a thoughtful approach to how data should be organized and managed.

And as an analytics engineer, understanding data architecture isn’t just helpful – it’s absolutely essential. Here’s the reasoning: every single day, you’re going to make decisions about how to structure data, where to store it, how to transform it, and how to make it accessible to others. These decisions might seem minor in the moment—Should I create this as a view or a table? Where should I put this transformation logic? How should I organize these datasets?but they add up together to create the foundation that your entire analytics ecosystem sits on.

In the following sections, we’ll introduce the core data architectural types and their main concepts, so that once you start designing your own analytics solutions, you’ll have a thorough understanding of both the strengths and weaknesses of each of these architectures. Please bear in mind that this is by no means a definitive guide to data architectures, as we don’t have the space to delve into the nitty-gritty details of each type. If you want to know each data architecture type in more depth, we recommend the book Deciphering Data Architectures by James Serra (O’Reilly, 2024).

Fasten your seatbelts, as we are now taking you on a journey through the history of data architectures!

The Data Architecture Evolution

The current data architecture landscape has a long history. As with any kind of technology, data architectures have been evolving to support the latest innovations and requirements. In the following sections, we will discuss key characteristics of the following data architectures:

• Relational databases

• Relational data warehouses

• Data lakes

• Data lakehouses

• Data fabric

• Data mesh

• Event-based architecture

This is the excerpt from the early release of the “Analytics Engineering with Microsoft Fabric and Power BI” book, which I’m writing together with Shabnam Watson for O’Reilly. You can already read the early release, which is updated chapter by chapter, here.

Relational Operational Database

Relational databases may be rightly considered the fine old wine in the data architecture realm. They have a long history, going all the way to the 1970s, when Edgar F. Codd proposed the relational model as a way to organize data.

At its core, a relational database is a highly organized, digital filing cabinet. For decades, it’s been the gold standard for storing business data for an extremely simple reason: it’s incredibly reliable. Think of each table in the database as a drawer in the cabinet, dedicated to one thing - say, customers or orders. Each row in a table is a single file folder (one customer, one order), and each column is a specific piece of information on a form inside that folder (a customer’s name, their email address, the order date, and so on).

The relational part is where the power comes from. It means the database understands how these tables are connected. It knows that the customer X in the Customers drawer is the same Customer X who placed the order in the Orders drawer. This structure is what allows us to ask complex questions using a language called SQL (Structured Query Language). SQL is just how you tell the database: Go get me all the orders placed by customer X.

When you work with these organized systems, you have to follow a strict rule called schema-on-write. Think of building a house: you must have a detailed blueprint (the schema) before you can even start laying the foundation. You define every room, every window, and every doorway upfront. The data must fit this blueprint perfectly when you save - or write - it. This upfront work ensures everything is consistent and the data is trustworthy.

The opposite approach, common in more flexible systems like data lakes, covered in one of the following sections, is schema-on-read. This is like dumping all your building materials - bricks, wood, windows - into a big pile. There’s no blueprint to start. You only decide how to structure it all when you need to build something, or when you read the data. It’s incredibly flexible and great for storing all kinds of messy, unstructured information, but it puts the responsibility of making sense of it all on the person analyzing it later.

Let’s break down a typical relational database structure. Figure below displays a simplified relational model that consists of four tables: Customer, Product, Orders, and OrderDetails.

If you don’t understand each piece in this illustration, don’t worry too much. In the next article, we will cover various data modeling concepts in depth.

Let’s briefly look at a couple of key points from the figure above:

• Each table stores the information about a specific type of object

• Each column represents the property, or attribute of the object defined within a table (e.g., FirstName, ProductName, OrderDate, etc.)

• Some columns are different from others. These columns are used to uniquely identify records and connect tables:

  • Primary key (PK): A unique identifier for each record within a table

  • Foreign key (FK): A field in one table that uniquely identifies a record in another table, and as such, is used to connect two tables

• Relationships are connectors between tables. In relational databases, relationships are enforced through schema design and constraints, not by creating separate relationship objects like in some other systems. For example, the CustomerID from the Customer table is also stored in the Orders table as a foreign key, allowing us to see all orders placed by a specific customer. In this particular case, one customer may place multiple orders, so we are talking about the one-to-many (1:M) relationship here. We cover relationships in detail in the next chapter

That example illustrates what’s known as a normalized structure - a design approach that organizes data efficiently by minimizing redundancy. This normalized structure - and, again, don’t stress if you don’t know what normalized structure is yet, as the next chapter explains it in detail - is highly efficient in scenarios when you need to frequently write or update the data. For example, a product name is stored only once in the Product table. Therefore, even if that product is sold five million times, you only need to update its name in one single place within the database, and the change will be immediately applied everywhere.

Relational Data Warehouse

In bygone days, relational databases were the undisputed kings of the data world. They were, and still are, fantastic at running the daily operations of a business, like processing orders, managing inventory, or updating customer records. We refer to these as operational or transactional (aka OLTP, which stands for online transactional processing) systems, and they must be lightning-fast. When a customer places an order, you can’t have them waiting around.

But this created a huge challenge. We call this the Don’t touch the live system! challenge. Business leaders needed answers to questions such as How many widgets did we sell last quarter? And Which products are most profitable? They needed to analyze the data to find insights.

But if you tried to run a big, complex analytical query on the same live database that was busy processing thousands of transactions a minute, everything would grind to a halt. It was like trying to do a deep, detailed inventory count in a busy supermarket during peak hours - you’d just get in everyone’s way and slow the whole operation down. This conflict between fast ongoing transactions and slow, heavy analysis was a major headache. The solution was to create a separate playground for analysts.

The invention of the relational data warehouse (RDW) solved the problem brilliantly. The concept was simple yet powerful: instead of running reports on live systems, you create a copy of all your important data from various operational systems (Sales, Inventory, CRM, and so on) and centralize it in a separate, dedicated database built specifically for analysis.

This was a true game-changer for two main reasons:

• It protected the live systems. Analysts could now run massive, complex queries in their own playground (the data warehouse) without any risk of slowing down the daily business operations.

• It created a single source of truth. For the first time, you could pull data from your sales system, your inventory system, and your customer relationship system into one place. This allowed you to answer much more sophisticated questions, like What is the lifetime value of customers who buy Product X? as shown in the diagram.

The illustration below best describes the need to implement a data warehouse. You may notice how complicated and inefficient it is to run a report that retrieves the data from multiple applications, where each business unit creates a report that retrieves the data from all the databases connected with each application. It’s fair to say that this is total chaos.

On the other hand, when you copy the data from all the applications into a data warehouse, life becomes much easier, as shown below.

Conceptually, a data warehouse is just a copy of the data stored in transactional processing systems. But, technically, it’s more than just a copy. It’s a special type of relational database optimized for high-speed analytical queries on huge volumes of data. It comes with its own powerful compute engine designed to slice, dice, and aggregate information, giving businesses a comprehensive view of their performance that was impossible before.

Essentially, the data warehouse gave businesses the best of both worlds: rock-solid, fast operational systems and a powerful, dedicated environment for deep data analysis.

In a typical scenario, a data warehouse is built by leveraging data pipelines that perform three core steps, or actions, called extract, transform, and load (ETL). Chapter 1 already discussed the shift from ETL to ELT paradigm, when we examined the circumstances that enabled the rise of analytics engineering.

Before examining the two most common approaches to creating a data warehouse, we’ll first highlight one of the main pitfalls in numerous real-life data warehouse implementations. As in many other scenarios, a lack of planning can easily come back to haunt you. Data warehouses are not an exception to this rule. What usually starts as a simple, one-off solution for a small number of users may later expand and become a real data monster. Thus, you should spend some time planning the data warehouse implementation.

We can segregate two fundamental approaches for building a data warehouse, and the following sections explore them in depth.

Top-down approach

This method starts with designing the overall data warehouse architecture and then creating data marts for specific business units, as shown in the illustration below.

This method was introduced by Bill Inmon, widely recognized as the father of the data warehouse. It begins with building a centralized, company-wide data warehouse. The data is normalized to avoid redundancy. This centralized repository serves as a single source of truth for managing and analyzing the data across the entire organization.

Read more

I’m extremely thrilled to announce the brand-new initiative I’m launching in early 2026: FabriQuiz! As you may assume based on its name, FabriQuiz is a live show with a quiz related to Microsoft Fabric - the fastest-growing analytics and AI platform in the world.

In this article, I’ll try to break down all the details about FabriQuiz, but, of course, if you have any further questions, don’t hesitate to reach out.

What is FabriQuiz?

FabriQuiz is an engaging and interactive way to learn about Microsoft Fabric, while having some fun and enjoying time with like-minded people. It is a live, online event streamed on the Data Mozart YouTube channel. Everyone is welcome to join! Even if you don’t plan to compete and participate in the quiz, you are free to attend and discuss various Fabric (and data in general) topics in the chat. The only “prerequisite” that I ask for is: be kind and respect others.

When is the FabriQuiz?

The inaugural edition of FabriQuiz will be on Sunday, January 11th at 7 PM CET/10 AM PST/1 PM EST. From there, we will gather every second Sunday (so, the 2nd episode will be streamed on January 25th), always at the same time. If there are any changes to the upcoming episodes, I’ll make sure to announce them in advance. In any case, the best way to stay up to date is to subscribe to the Data Mozart YouTube channel and set a reminder for FabriQuiz.

Why FabriQuiz?

Back in the days, and I’m sure a lot of you from the community remember it, we had a Power BI Quiz, run by Just Blindbæk. It was such great fun, but also an extraordinary way to learn about various aspects of the Power BI as a platform. Personally, this was one of my favorite ways to spend 45-60 minutes on Sunday evenings. This quiz doesn’t exist anymore, although my friend Ricardo Rincón is running a Spanish version (I wish I knew Spanish so I could join😉).

Hence, I was thinking: why not revive such an engaging live show, but this time targeting everything around Microsoft Fabric?

How will FabriQuiz work?

Each edition of FabriQuiz will consist of 10 questions. In most cases, these will be classic, multiple-choice questions with one or multiple correct answers. The winner is the player with the highest number of points (not necessarily the one with the most correct answers), because we are measuring both accuracy (the most important) and speed (also very important).

The quiz itself is hosted and run on a dedicated platform similar to Kahoot.

I plan to invite special guests to FabriQuiz (the same as Just did for Power BI quiz) to provide their unique perspective on Fabric. Guests might be from both the Microsoft Fabric product team and the data community. The first episode will be without special guests, and then I’ll see how it goes from there:)

Sponsors wanted!

Just to make this crystal clear - I don’t plan to make any money out of FabriQuiz, but I’m looking for sponsors for winners’ prizes. It can be a gift card, free license, or subscription to your product (of course, if the product is relevant to the quiz topic. I definitely won’t accept any cosmetics, sports equipment, or similar😊). If we manage to attract multiple sponsors, I’m happy to provide prizes for the top N players:)

For the first two episodes, prizes are already secured, and we’ll see how it goes from February onwards:)

Each sponsor gets a proper shout during the show, both verbally and using visual elements (logo, promo message, etc.).

Key facts about FabriQuiz

❓WHAT: A fun and interactive way of learning about Microsoft Fabric, with a chance to grab some cool prizes

📆 WHEN: January 11th, 2026, 7 PM CET/1 PM EST/10 AM PST (the inaugural episode). Then, bi-weekly (every second Sunday)

📍WHERE: Online, Data Mozart YouTube channel

Enjoy the holidays and looking forward to seeing you on January 11th! Let’s have some Fabric fun:)

In the ocean of announcements from the recent FabCon Europe in Vienna, one that may have gone under the radar was about the enhancements in performance and cost optimization for Dataflows Gen2.

Before we delve into explaining how these enhancements impact your current Dataflows setup, let’s take a step back and provide a brief overview of Dataflows. For those of you who are new to Microsoft Fabric – a Dataflow Gen2 is the no-code/low-code Fabric item used to extract, transform, and load the data (ETL).

A Dataflow Gen2 provides numerous benefits:

• Leverage 100+ built-in connectors to extract the data from a myriad of data sources

• Leverage a familiar GUI from Power Query to apply dozens of transformations to the data without writing a single line of code – a dream come true for many citizen developers

• Store the output of data transformation as a delta table in OneLake, so that the transformed data can be used downstream by various Fabric engines (Spark, T-SQL, Power BI…)

However, simplicity usually comes with a cost. In the case of Dataflows, the cost was significantly higher CU consumption compared to code-first solutions, such as Fabric notebooks and/or T-SQL scripts. This was already well-explained and examined in two great blog posts written by my fellow MVPs, Gilbert Quevauvilliers (Fourmoo): Comparing Dataflow Gen2 vs Notebook on Costs and usability, and Stepan Resl: Copy Activity, Dataflows Gen2, and Notebooks vs. SharePoint Lists, so I won’t waste time discussing the past. Instead, let’s focus on what the present (and future) brings for Dataflows!

Changes to the pricing model

Let’s briefly examine what’s displayed in the illustration above. Previously, every second of the Dataflow Gen2 run was billed at 16 CU (CU stands for Capacity Unit, representing a bundled set of resources – CPU, memory, and I/O – used in synergy to perform a specific operation). Depending on the Fabric capacity size, you get a certain number of capacity units – F2 provides 2 CUs, F4 provides 4 CUs, and so on.

Going back to our Dataflows scenario, let’s break this down by using a real-life example. Say you have a Dataflow that runs for 20 minutes (1200 seconds)…

• Previously, this Dataflow run would have cost you 19.200 CUs: 1200 seconds * 16 CUs

• Now, this Dataflow run will cost you 8.100 CUs: 600 seconds (first 10 minutes) * 12 CUs + 600 seconds (after first 10 minutes) * 1.5 CUs

The longer your Dataflow needs to execute, the bigger the savings in CUs you potentially make.

This is amazing on its own, but there is still more to that. I mean, it’s nice to be charged less for the same amount of work, but what if we could make these 1200 seconds, let’s say, 800 seconds? So, it wouldn’t save us just CUs, but also reduce the time-to-analysis, since the data would have been processed faster. And, that’s exactly what the next two enhancements are all about…

Modern Evaluator

The new preview feature, named Modern Evaluator, enables using the new query execution engine (running on .NET core version 8) for running Dataflows. As per the official Microsoft docs, Dataflows running the modern evaluator can provide the following benefits:

• Faster Dataflow execution

• More efficient processing

• Scalability and reliability

The illustration above shows the performance differences between various Dataflow “flavors”. Don’t worry, we will challenge these numbers soon in a demo, and I’ll also show you how to enable these latest enhancements in your Fabric workloads.

Partitioned Compute

Previously, a Dataflow logic was executed in sequence. Hence, depending on the logic complexity, it could take a while for certain operations to complete, so that other operations in the Dataflow had to wait in the queue. With the Partitioned Compute feature, Dataflow can now execute parts of the transformation logic in parallel, thus reducing the overall time to complete.

At this moment, there are certain limitations on when the partitioned compute will kick in. Namely, only ADLS Gen2, Fabric Lakehouse, Folder, and Azure Blob Storage connectors can leverage this new feature. Again, we’ll explore how it works later in this article.

3, 2, 1…Action!

Ok, it’s time to challenge the numbers provided by Microsoft and check if (and to what degree) there is a performance gain between various Dataflows types.

Here is our scenario: I’ve generated 50 CSV files that contain dummy data about orders. Each file contains approximately 575.000 records, so there are ca. 29 million records in total (approximately 2.5 GBs of data). All the files are already stored in the SharePoint folder, allowing for a fair comparison, as Dataflow Gen1 doesn’t support OneLake lakehouse as a data source.

I plan to run two series of tests: first, include the Dataflow Gen1 in the comparison. In this scenario, I won’t be writing the data into OneLake using Dataflows Gen2 (yeah, I know, it defeats the purpose of the Dataflow Gen2), as I want to compare “apples to apples” and exclude the time needed for writing data into OneLake. I will test the following four scenarios, in which I perform some basic operations to combine and load the data, applying some basic transformations (renaming columns, etc.):

1. Use Dataflow Gen1 (the old Power BI dataflow)

2. Use Dataflow Gen2 without any additional optimization enhancements

3. Use Dataflow Gen2 with only the Modern evaluator enabled

4. Use Dataflow Gen2 with both the Modern evaluator and Partitioned compute enabled

In the second series, I’ll compare three flavors of Dataflow Gen2 only (points 2-4 from the list above), with writing the data to a lakehouse enabled.

Let’s get started!

Dataflow Gen1

The entire transformation process in the old Dataflow Gen1 is fairly basic – I simply combined all 50 files into a single query, split columns by delimiter, and renamed columns. So, nothing really advanced happens here:

The same set of operations/transformations has been applied to all three Dataflows Gen2.

Please keep in mind that with Dataflow Gen1 it’s not possible to output the data as a Delta table in OneLake. All transformations are persisted within the Dataflow itself, so when you need this data, for example, in the semantic model, you need to take into account the time and resources needed to load/refresh the data in the import mode semantic model. But, more on that later.

Dataflow Gen2 without enhancements

Let’s now do the same thing, but this time using the new Dataflow Gen2. In this first scenario, I haven’t applied any of these new performance optimization features.

Dataflow Gen2 with Modern Evaluator

Ok, the moment of truth – let’s now enable the Modern Evaluator for Dataflow Gen2. I’ll go to the Options, and then under the Scale tab, check the Allow use of the modern query evaluation engine box:

Everything else stays exactly the same as in the previous case.

Dataflow Gen2 with Modern Evaluator and Partitioned Compute

In the final example, I’ll enable both new optimization features in the Options of the Dataflow Gen2:

Now, let’s proceed to the testing and analyzing results. I will execute all four dataflows in sequence from the Fabric pipeline, so we can be sure that each of them runs in isolation from the others.

And, here are the results:

Partitioning obviously didn’t count much in this particular scenario, and I will investigate how partitioning works in more detail in one of the following articles, with different scenarios in place. Dataflow Gen2 with Modern Evaluator enabled, outperformed all the others by far, achieving 30% savings compared to the old Dataflow Gen1 and ca. 20% time savings compared to the regular Dataflow Gen2 without any optimizations! Don’t forget, these savings also reflect in the CU savings, so the final CU estimated cost for each of the used solutions is the following;

• Dataflow Gen1: 550 seconds * 12 CUs = 6.600 CUs

• Dataflow Gen2 with no optimization: 520 seconds * 12 CUs = 6.240 CUs

• Dataflow Gen2 with Modern Evaluator: 368 seconds * 12 CUs = 4.416 CUs

• Dataflow Gen2 with Modern Evaluator and Partitioning: 474 seconds * 12 CUs = 5.688 CUs

However, I wanted to double-check and confirm that my calculation is accurate. Hence, I opened the Capacity Metrics App and took a look at the metrics captured by the App:

Although the overall result accurately reflects the numbers displayed in the pipeline execution log, the exact number of used CUs in the App is different:

• Dataflow Gen1: 7.788 CUs

• Dataflow Gen2 with no optimization: 5.684 CUs

• Dataflow Gen2 with Modern Evaluator: 3.565 CUs

• Dataflow Gen2 with Modern Evaluator and Partitioning: 4.732 CUs

So, according to the Capacity Metrics App, a Dataflow Gen2 with Modern Evaluator enabled consumed less than 50% of the capacity compared to the Dataflow Gen1 in this particular scenario! I plan to create more test use cases in the following days/weeks and provide a more comprehensive series of tests and comparisons, which will also include a time to write the data into OneLake (using Dataflows Gen2) versus the time needed to refresh the import mode semantic model that is using the old Dataflow Gen1.

Conclusion

When compared to other (code-first) options, Dataflows were (rightly?) considered “the slowest and least performant option” for ingesting data into Power BI/Microsoft Fabric. However, things are changing rapidly in the Fabric world, and I love how the Fabric Data Integration team makes constant improvements to the product. Honestly, I’m curious to see how Dataflows Gen2’s performance and cost develop over time, so that we can consider leveraging Dataflows not only for low-code/no-code data ingestion and data transformation requirements, but also as a viable alternative to code-first solutions from the cost/performance point of view.

Thanks for reading!

Thanks to my friend Sandeep Pawar from the Fabric CAT team, I’ve been pointed to the great video by Pat Mahoney (Sandeep’s colleague from Microsoft), which provides more details about configuring the partitioning in Dataflows Gen2.

Imagine this: It’s Monday morning, and you’re finding yourself in yet another meeting where the finance team is arguing with the sales team about whose numbers are “right.” The CFO insists that lead conversion rates are up by 23% this quarter, while the VP of Sales swears they’re only seeing an 11% increase. Meanwhile, the CEO is getting frustrated because nobody can agree on basic facts about the business.

Sound familiar? If you’ve worked in any data-driven organization in the last five years, you’ve probably witnessed this exact scenario multiple times.

But, let’s be honest - this isn’t really (only) about the numbers being wrong. It’s about something much more fundamental: the gap between those who understand the data and those who need to use it to make decisions. And that gap, dear reader, is exactly where analytics engineering comes in.

The Making of Analytics Engineering

The story about analytics engineering doesn’t start as a great plan in someone’s head in a Silicon Valley boardroom. It was more like... well, like most good ideas - born out of pure necessity and a healthy dose of frustration.

Back in early 2019, a group of passionate data folks were hanging out in an online community. They were sharing stories about their day-to-day jobs, and a pattern started to emerge. Everyone was dealing with the same problem: traditional data teams were cracking under the pressure of modern data demands.

Data engineers were drowning. They were supposed to be building a robust and resilient data infrastructure, but more often than not, they were also pulled into every data transformation request. “Hey, can you just quickly add this metric to our dashboard?” “Can you modify this report to show regional breakdowns?” “Why doesn’t our customer count match between these two systems?”

Meanwhile, data analysts were stuck. They knew the business inside and out, they understood what questions needed answering, but they couldn’t access the data they needed without going through the already overwhelmed engineering team. It was like being a chef who wasn’t allowed to enter the kitchen.

And that’s when someone in that online community said the magic words: “What if we had someone whose job was specifically to bridge the gap between technology and business? Someone who could speak both languages and focus purely on making data usable?”

That “someone” was the analytics engineer.

The idea quickly took off. The team at dbt Labs, the company behind the data build tool that’s become synonymous with analytics engineering, started evangelizing the role. Companies started to hire for the role. Universities began offering courses on it. And here we are today, with analytics engineering being one of the fastest-growing roles in tech.

This is the excerpt from the early release of the “Analytics Engineering with Microsoft Fabric and Power BI” book, that I’m writing together with Shabnam Watson for O’Reilly. You can already read the early release, which is updated chapter by chapter, here.

What exactly does an Analytics Engineer do?

We’re glad you asked! But seriously, if you asked five different analytics engineers to describe their job, you’d probably get six different answers. Even the role name can be a bit of a head-scratcher at first. And professionals working in the data field sometimes get a bit confused about where one role ends and another begins. It’s like trying to explain the difference between a latte, a cappuccino, and an espresso to someone who just wants a good coffee – they’re all coffee, but the nuances also matter.

The analytics engineer role is still evolving, and it looks different at every company. But here’s our take on it, based on long hours of working with many outstanding analytics engineers and seeing the value they create:

Analytics engineers are the translators between the world of raw data and the world of business decisions.

Think of analytics engineers as skilled craftspeople who step into the messy world of data storage. They don’t just clean that space, but also build well-organized, clearly labeled shelves ( data models), ensure all the items are clean and reliable ( data quality and transformation), and create clear pathways so other people can easily find exactly what they need. They are the bridge between the super-technical “how the data gets collected and stored” part and the “what does this data actually mean for the business?” part.

Now, you might be wondering: “Wait, isn’t that what a Data Engineer does? Or a Data Analyst?” And that’s a great question! While analytics engineers work closely with both, the Analytics Engineer role has its own distinct flavor. They are not always building massive, foundational data pipelines from scratch as many data engineers do (though a sound analytics engineer will understand how they work). And they are not usually the ones deep-diving into specific datasets to create the final dashboard or build the business report as a data analyst often does (though a skilled analytics engineer will enable them to do it significantly better).

An Analytics Engineer is situated in the specialized space in between, focusing on making data trustworthy, accessible, and optimized specifically for analytics. They ensure that when five different people ask “What was our revenue last month?”, they all get the same, correct answer, without wanting to pull their hair out trying to find it.

However, what makes analytics engineers special is that they don’t just move data around. They apply the same strict principles that software engineers leverage to build applications: version control, testing, documentation, and continuous integration. Imagine if every data transformation were as reliable and well-documented as the well-established software you use on a daily basis. That’s what analytics engineers bring to the table.

Let us give you a real-life example. Imagine you work for an e-commerce company, and your CEO wants to know the answer to a simple question: “What’s our customer lifetime value by acquisition channel?”

In the old world, this would have been a nightmare. An analyst would have had to:

1. Figure out where customer data lives (probably five different systems)

2. Figure out where transaction data lives (probably two additional systems)

3. Figure out where the acquisition channel data lives (good luck with that!)

4. Write a bunch of complex SQL to join it all together

5. Hope they didn’t make any mistakes in the logic

6. Pray that the data was actually accurate and up-to-date

7. Spend days validating the results

With an analytics engineer on the team, the following illustration displays what happens instead:

1. The analytics engineer has already built a reliable pipeline that brings together customer, transaction, and acquisition data

2. They’ve tested and documented all the transformation logic

3. They’ve created a clean, business-ready dataset that updates automatically. The analyst can simply query the prepared dataset and get their answer in minutes

4. Everyone trusts the results because they know the process is reliable

The data analyst gets to focus on the fun part - finding insights and telling stories. The data engineer gets to focus on infrastructure instead of ad-hoc data requests. And the business gets reliable answers to their questions. It’s not a win-win scenario, but rather a win-win-win.

The Big ETL to ELT (R)evolution

Now that we have explained what analytics engineers are supposed to do, let’s understand one of the biggest shifts in the data ecosystem, which made analytics engineering not just possible, but necessary - the transition from ETL to ELT. Don’t worry if you don’t know what these acronyms stand for, as we will break them down here.

Read more

There is a LOT of Microsoft Fabric training nowadays!

Some of them are free, while the others are paid. And, it's hard to navigate through the noise and make the right decision - not just in terms of money, but also in terms of the time you plan to invest in learning the fastest-growing analytics platform.

TL;DR: If you want to skip reading this post and find out all the details about the “24 Hours of Fabric” training, you can jump here

Why should you join the "24 Hours of Fabric" training?

• It's carefully tailored based on the vast experience of delivering numerous Fabric trainings for hundreds of learners

• It's curated based on the feedback I received from previous trainings - incorporating all the positives and removing all the negatives provided by your fellow Fabric learners

• It's built with YOU and YOUR priorities in mind! Let me elaborate on this: An 8-hour training can be exhaustive...3 consecutive days of 8-hour training may be an overkill even for most dedicated learners! Thus, I decided to break the learning content into smaller, reasonable chunks - to keep you engaged and focused all the time, while giving your brain a fair share of time to refresh and absorb everything we learn. Not only that. I completely understand if your daily job doesn't allow spending the whole day away. Maybe you have some important meetings, calls, or other responsibilities. Or, maybe you want to go for a family party AFTER your training, still saving enough energy to share with your loved ones. So, there will be 3 sessions of 2 hours each week - on Monday, Wednesday, and Friday. 4 weeks x 3 days x 2 hours = 24 hours in total!

• You'll get a chance to put your hands right on Fabric! It's nice listening to me talking and showing how cool Fabric is in demos, but it's a whole different story getting YOUR hands dirty with various Fabric components and workloads. You'll be asked to complete 10+ hands-on labs during the training, to help you get a real-life feeling about the Fabric platform

• You'll also get a chance to ask a question and get an answer - of course, assuming that I know the answer:) But, I promise, if I don't know the answer immediately, I'll get back to you as soon as possible. This is completely different from free online tutorials or massive open online courses with hundreds of participants, where communication is unidirectional, and your question may not get answered or your concerns may not get clarified. We work in a small group (max. 10 participants, as I want you to get the most value out of this training)

• Working in a small and focused group - the maximum number of participants is capped at 10 per group. No exceptions! This guarantees that I can dedicate a proper amount of time to each and every one of you trying to learn Fabric, and that you have enough space to ask your questions

• You'll get access to the recording of each training session that will be available for the next 365 days for you to revisit and reinforce learning on demand

• You'll get a FREE 1-hour "1-on-1" learning mentoring session with me, where we discuss your plans, future opportunities to upskill in Fabric, as well as additional learning resources to shape your knowledge and advance your career. Please bear in mind that this is NOT a consulting session (I won't solve your production-level Fabric problems), but you can always hire me if you need help with that matter:)

• You'll learn directly from me:) Why from me, I hear you ask? I've been working with Microsoft Fabric since its very early days - before it was even called Fabric! As part of the early private preview program, I had an opportunity to get involved in the platform evolution together with Microsoft product teams. Additionally, I've been working on various real-life Fabric projects, supporting numerous organizations in their Fabric implementation journey. Last but not least, I've been officially recognized by Microsoft as an MVP for Microsoft Fabric. I also co-authored the "Fundamentals of Microsoft Fabric" book (with Ben Weissman). Check the full list of benefits for the training, as one of the benefits is a free copy of the book.

Sign me up!

Why you should NOT join the "24 Hours of Fabric" training?

• If you are looking for an easy path to becoming a Fabric expert without investing any effort, this training is probably not for you. I will push you to the limits and ask you not only to complete hands-on labs, but also to read and learn beyond the topics we cover during the training session. I know that this might sound scary to some participants, but I can GUARANTEE that anyone who completes the entire "24 Hours of Fabric" training may work with the platform confidently

• If you have never worked with any data analytics tool or platform before, this training might feel overwhelming to you. With that being said, I'll ASSUME that you have a basic understanding of the fundamental data analytics and data modeling concepts, since we need to focus on learning Fabric in a limited amount of time. However, if you feel you still have what it takes to take this training anyway, I'd be happy to provide you with the list of learning resources to prepare for the training

What will you learn in the “24 Hours of Fabric” training?

Here is the high-level curriculum (some tweaks may be possible):

1. Navigating the Microsoft Fabric ecosystem

   1. WHY MS Fabric?

   2. WHAT is MS Fabric?

   3. Tenant -> Capacity -> Domains -> Workspaces -> Items

   4. Core components (high-level overview)

   5. Pricing model

2. OneLake

   1. Overview of data architectures

   2. Overview of OneLake - concepts and technical implementation

   3. Inside the Delta Parquet storage format

   4. All roads lead to OneLake

   5. Introducing Shortcuts and Mirroring,

3. Ingest or Access Data

   1. Data Factory as a workload

   2. Dataflows Gen2

   3. Pipeline

   4. Copy data activity

   5. Copy job

   6. Data loading methods: use cases and common scenarios

   7. Apache Airflow

   8. Scheduling and triggers

4. Data Engineering

   1. Fundamentals of data engineering

   2. Fundamentals of a lakehouse

   3. Understanding the Spark engine fundamentals

   4. Medallion design pattern fundamentals

   5. Notebooks

   6. Querying and working with the data in the lakehouse

   7. Limitations and considerations

5. Data Warehousing

   1. Fundamentals of data warehousing

   2. Warehouse vs Lakehouse: Which one is for me?

   3. Warehouse core elements: schemas, tables, views, sprocs

   4. Advanced features (clone table, time travel, etc.)

   5. Query and work with the warehouse data

   6. Limitations and considerations

6. Direct Lake for Power BI

   1. What is Direct Lake?

   2. Direct Lake inner workings

   3. Two types of Direct Lake

   4. Core concepts in Direct Lake (syncing, framing, transcoding, temperature)

   5. Analyze query processing

   6. Creating semantic models and reports with Direct Lake

7. Real-Time Intelligence

   1. Stream processing vs. batch processing

   2. Real-time hub

   3. Eventhouse

   4. KQL database

   5. KQL queryset

   6. Real-time dashboard

   7. Activator

   8. KQL fundamentals (data transformation, data processing, etc.)

8. Administration, Monitoring, Security, CI/CD

   1. Administration: Admin portal and various settings

   2. Monitoring: different "levels" of monitoring

   3. Security: current security model + OneLake security

   4. CI/CD options: Workspace Git integration, Deployment pipelines

   5. Best practices for lifecycle management

9. AI Features

   1. Copilots

   2. Data agents

10. Prepare Data for Fabric Workloads

    1. Choose between the lakehouse, warehouse, or eventhouse

    2. OneLake integration for semantic models and eventhouse

    3. Create views, functions, and stored procedures

    4. Enrich data by adding new columns or tables

    5. Implement a star schema for the lakehouse or warehouse

    6. Denormalize, aggregate, and merge data

    7. Identify and resolve duplicates and missing values

    8. Change column data types

11. Advanced Concepts and Techniques

    1. Incremental refresh with pipelines, notebooks, Dataflows

    2. Metadata-driven pipelines

    3. Implementing a medallion design pattern in MS Fabric

    4. Semantic link

    5. Configure Spark settings

12. Optimizing Fabric Workloads

    1. Optimizing lakehouse tables: OPTIMIZE, VACUUM, V-order

    2. Optimize SQL queries for a warehouse

    3. Optimize Direct Lake model performance

    4. Optimize Power BI semantic models

Book Your Seat!

Training start and format

• The training starts on January 5th (Monday) and ends on January 30th (Friday)

• Sessions are scheduled every Monday, Wednesday, and Friday (2 hours each)

• Sessions start at 8 AM PT / 11 AM EDT / 5 PM CET

• Training is delivered virtually via Zoom

Bonus resources

• Access to all training materials: decks, notebooks, queries, files

• Access to a recording of each session for the next 365 days

• Complimentary copy of the "Fundamentals of Microsoft Fabric" book

• Certificate of completion

Still not sure if this is for you? Check the full FAQ section and let me know if you have any additional questions.

Subscribe now

Data transformations are the bread and butter of every analytics project. Let’s be honest – raw data is usually – well, raw – which in many cases means messy, invalid, and inconsistent. Before you can extract valuable insights, you need to transform the data into something clean, structured, and ready for analysis.

By this point, if we talk about Microsoft Fabric specifically, you are probably aware that the same task can be done in multiple different ways. I’ve already written about how to pick the most suitable analytical engine. But, even then, the possibilities are tremendous. By that, I mean that choosing the analytical engine doesn’t prevent you from mixing more than one programming language for data transformations.

In the lakehouse, for example, you can transform the data by using PySpark, but also Spark SQL, which is VERY similar to Microsoft’s dialect of SQL, called Transact-SQL (or T-SQL, abbreviated). In the warehouse, you can apply transformations using T-SQL, but Python is also an option by leveraging a special pyodbc library. Finally, in the KQL database, you can run both KQL and T-SQL statements. As you may rightly assume, the lines are blurred, and sometimes the path is not 100% clear.

Therefore, in this article, I’ll explore five common data transformations and how to perform each one using three Fabric languages: PySpark, T-SQL, and KQL.

Extracting date parts

Let’s start by examining how to extract date parts from the data column. This is a very common requirement when you have only the date column, and you need to extend date-related information with additional attributes, such as year, month, etc.

Imagine that we have a source data that contains the OrderDate column only, and our task is to create the Year and Month columns out of it.

from pyspark.sql.functions import *

 # Create Year and Month columns
 transformed_df = df.withColumn("Year", year(col("OrderDate"))).withColumn("Month", month(col("OrderDate")))

The same can be done using T-SQL…

--Extracting date parts
SELECT SalesOrderNumber
    , OrderDate
    , YEAR(OrderDate) as OrderYear
    , MONTH(OrderDate) as OrderMonth
    , DATENAME(WEEKDAY, OrderDate) as DayOfWeek
FROM dbo.orders

…and KQL:

//extracting date parts
Weather
| extend dateYear = datetime_part("year", StartTime)
         , dateMonth = datetime_part("month", StartTime)
         , dayOfWeek = dayofweek(StartTime)

The only difference compared to T-SQL is that the dayofweek KQL function will return the number of days since the preceding Sunday, as a timespan.

Uppercase/lowercase transformation

Let’s learn how to apply uppercase/lowercase transformation in PySpark:

df_transformed = df.selectExpr(
    "SalesOrderNumber as order_ID",
    "upper(CustomerName) as customer_name_upper"
)
df_transformed.show()

Now, with T-SQL:

-- Alias, UPPER transformation
SELECT SalesOrderNumber as order_id
    , UPPER(CustomerName) as upper_cust_name
FROM dbo.orders

And, finally, let’s lowercase all the names by using KQL:

//select and rename columns
Weather
| project State
          , LowerState = tolower(State)

Conditional column

• PySpark – in the following example, we are creating a conditional column price_range, based on the value in the existing UnitPrice column. If the value is greater than 1000, we want to set the value of the price_range to “High”. If it’s greater than 500, then the price_range is “Medium”. For all the other values, the price_range will return “Low”. You might be thinking: wait, if there is a Unit Price value of, let’s say, 2000, this can be both High and Medium. However, the expression evaluation happens in the order you defined in the statement. First, it will check if the particular value is greater than 1000. If this evaluates to true, no further checks will be performed.

from pyspark.sql.functions import col, when

df = df.withColumn(
    "price_range",
    when(col("UnitPrice") > 1000, "High")
    .when(col("UnitPrice") > 500, "Medium")
    .otherwise("Low")
)
df.select("SalesOrderNumber","OrderDate","CustomerName","Item","Quantity","UnitPrice", "price_range").show()

• T-SQL

--Conditional column
SELECT SalesOrderNumber
    , OrderDate
    , CustomerName
    , Item
    , Quantity
    , UnitPrice
    , CASE 
        WHEN UnitPrice > 1000 THEN 'High'
        WHEN UnitPrice > 500 THEN 'Medium'
        ELSE 'Low'
    END AS PriceRange
FROM dbo.orders

• KQL

//conditional column
Weather
| extend DamagePropertyClass = case (
    DamageProperty > 49999, "High"
    , DamageProperty > 9999, "Medium"
    , "Low"
)

Pivot transformation

Sometimes, we need to tweak the structure of the underlying data. Pivot transformation is a common example of transforming rows into columns. In this case, I’m creating a new dataframe df_pivot, which first groups the data by the order date, then pivots values from the price_range column, remember those highs, mediums, and lows from the previous example, and calculates the sum of the unit price.

df_pivot = df.groupBy("OrderDate").pivot("price_range").sum("UnitPrice")

df_pivot.show()

The T-SQL implementation requires slightly more complex code, as you may notice in the following snippet:

--PIVOT Transformation
SELECT 
    OrderDate 
    , [High] 
    , [Medium] 
    , [Low] 
FROM (
    SELECT 
        OrderDate
        , UnitPrice
        , CASE 
        WHEN UnitPrice > 1000 THEN 'High'
        WHEN UnitPrice > 500 THEN 'Medium'
        ELSE 'Low'
    END AS PriceRange
    FROM dbo.orders
) AS SourceTable
PIVOT (
    SUM(UnitPrice)
    FOR PriceRange IN ([High], [Medium], [Low])
) AS PivotTable;

Unlike PySpark and T-SQL, which both provide a built-in pivot function, there is no such function in KQL. However, there is a pivot plugin, and we can mimic PIVOT behavior by combining the summarize and evaluate operators. In this example, we are summing injuries direct per year, month and state, and then pivoting states to be columns.

//Pivot transformation
Weather
| summarize Total = sum(InjuriesDirect) by datetime_part ("year", StartTime), datetime_part ("month", StartTime), State
| evaluate pivot(State, sum(Total))

Aggregation and grouping

Let’s wrap it up by examining how to perform aggregation and grouping operations when transforming the data. This is one of the most frequent requirements in analytical scenarios.

First, PySpark. I’m grouping the data by the year and month columns, and then I’m performing the sum aggregate function on the Unit price column. In the last step, we can also rename the column if needed, which I did here.

sales_by_yearMonth = orders_df.groupBy("Year", "Month").agg(
    {"UnitPrice": "sum"}
).withColumnRenamed("sum(UnitPrice)", "Unit_Price")

sales_by_yearMonth.show()

Now, T-SQL:

--Aggregate and group data
SELECT YEAR(OrderDate) as OrderYear
    , MONTH(OrderDate) as OrderMonth
    , count(*) as TotalOrders
    , SUM(UnitPrice) as TotalUnitPrice
FROM dbo.orders
GROUP BY YEAR(OrderDate)
    , MONTH(OrderDate)

And, finally, KQL:

//aggregate and group data
//Use summarize operator
Weather
| summarize TotalRecords = count() by State

Conclusion

By understanding and leveraging the strengths of PySpark, T-SQL, and KQL, you can efficiently handle numerous data transformation tasks in Microsoft Fabric. Whether it’s filtering, aggregating, pivoting, cleaning, or enriching the data, choosing the right language for each scenario will enable faster and more resilient workflows, enhance productivity, and deliver more value from the raw data.

Thanks for reading!

Microsoft Fabric is all about the options. The same task can be completed in multiple different ways. And, it doesn’t necessarily mean that one way is better than the other. Sometimes, it boils down to the skillset of the individuals in your organization performing the data task or maybe a personal preference. However, sometimes, these decisions might have a larger impact on the entire data platform workload. Hence, it is of paramount importance to understand the implications of choosing Option A vs. Option B, for example,to complete Task XYZ in Microsoft Fabric.

In this article, I will do my best to provide you with guidance on the most common Fabric dilemmas. Please bear in mind that the list of dilemmas is not definitive – I simply handpicked some of the challenges that Fabric practitioners face most frequently, and where I believe that such guidance would make the lives of these practitioners much easier in their day-to-day work with Microsoft Fabric.

General recommended practices and edge-case scenarios…

Before we proceed, an important disclaimer: the guidance I’m providing here is based on both my experience with implementing Microsoft Fabric in real-world scenarios, and the recommended practices provided by Microsoft.

Please keep in mind that the guidance relies on general recommended practices (I intentionally avoid using the phrase best practices, because the best is very hard to determine and agree on). The word general means that the practice I recommend should be used in most of the respective scenarios, but there will always be edge cases when the recommended practice is simply not the best solution. Therefore, you should always evaluate whether the general recommended practice makes sense in your specific use case.

Choosing the Analytical Engine

All Microsoft Fabric roads lead to OneLake! You’ve already learned that mantra. However, which road is right for you? Should you pick a lakehouse, a warehouse, or maybe an eventhouse for storing your data in OneLake? There are numerous factors to take into account, so let’s explore three that I consider most relevant: data volume, supported data formats, and supported programming languages.

Data volume

Let’s first talk about the data volume. Although there is no single number that differentiates small and big data, and you’ll probably find dozens of opinions in the various resources on the web or books, we often refer to big data as an extremely large and diverse collection of data that continuously grows over time. However, for the sake of setting boundaries when making decisions, we will consider big data anything above 5 TB of stored data, and 100+ GB of ingested data per day.

When choosing the optimal analytical engine in Microsoft Fabric, there is no such thing as the minimum amount of data suitable for a warehouse versus a lakehouse, or vice versa. In theory, you can store only a few records of data in any of the analytical stores in Fabric, but you’ll start reaping the benefits of Massively Parallel Processing (MPP) architectures, which are implemented in both Fabric lakehouse and Fabric warehouse, with large amounts of data only.

Explaining the details about MPP is beyond the scope of this article, but you can find more details here.

To wrap up – data volume shouldn’t be a determining factor when deciding which analytical engine to use, as each lakehouse, warehouse, and eventhouse, scales to support petabytes of data if necessary.

Supported data formats

When choosing a storage option in OneLake, you’ll need to first consider the data format. The table you see depicts supported data formats in the three Fabric analytical storages.

As you can see, both lakehouse and eventhouse support all data formats. So, the fair question would be: when should I use a lakehouse over an eventhouse?

The short answer is: whenever you need to handle any type of streaming, or event-based data, choose an eventhouse. A few examples of event-based data are: telemetry and log data, time series, data collected from IoT devices, etc.

Of course, the long answer to the question about Lakehouse vs. Eventhouse is more nuanced and depends on numerous factors, such as:

• Using data downstream – For example, if you are using Direct Lake mode for Power BI semantic models, data must be available in Delta format, which is not the case in the eventhouse (unless you enable the Eventhouse OneLake Availability feature)

• Optimizing storage cost – Generally speaking, storage is more expensive for eventhouse than for lakehouse, since eventhouses use OneLake Cache Storage to provide the fastest query response times at an additional cost, as described here. A point worth mentioning is that when you set the caching policy to move the data to cold storage in the eventhouse, it’s billed the same as regular OneLake storage. Hence, only keeping the data in the hot cache (which improves the query performance) will cost you more

• Processing streaming data – Eventhouse is optimized for processing streaming data (indexing and partitioning happen automatically), whereas processing streaming data using a lakehouse will force the creation of multiple small parquet files that must be vacuumed and optimized later, which can lead to increased capacity unit consumption. Explaining the vacuum and optimize operations is out of the scope of this article, but you can read more about it here

Supported programming languages

Let’s now examine how supported programming languages might affect the storage choice. This is undoubtedly one of the key factors when making a decision, not only about the analytical engine, but also when designing the entire Fabric architecture. The reason is obvious – let’s imagine that the entire data team in your organization consists of hard-core T-SQL developers. Would you force them to learn another programming language (Python, for example)? Or would you rather choose a Fabric component that plays to their strength?

The following table shows supported programming languages for both read and write operations.

Let’s stop for a moment and examine the importance of the selection you make based on this information. Imagine that you’re implementing a medallion design pattern in Microsoft Fabric. In accordance with the recommended practice of building a Star schema dimensional model in the final layer (gold/curated), you’d need to apply various data transformations to implement business rules and logic. Hence, if your data engineers or analytics engineers are feeling comfortable writing T-SQL, you should probably choose a warehouse for this layer, since T-SQL can’t be used for inserting, updating, or deleting the data in the lakehouse and/or eventhouse.

On the flip side, if the majority of the data team is proficient with Python or any language that can be used to manipulate the data using the Spark engine, you’d probably go all the way with the lakehouse. Although, in full honesty, the road between the T-SQL and Spark SQL is not that long, in case you plan to leverage SQL skills while using the lakehouse.

OneLake Interoperability

Now, things become more complex…“What if I choose the lakehouse for my silver layer, and the warehouse for the gold layer? Can I combine the data from both the lakehouse and the warehouse?” This is a very common question asked by data professionals considering or already using Microsoft Fabric. Hence, in this table, you’ll find an overview of the interoperability between various analytical engines in Fabric.

Scenario-based Decision Guide

Based on all of the aforementioned criteria and typical analytical requirements, I identified a few common scenarios you might face when deciding which Fabric component to pick for the particular use case. The table below illustrates the level of suitability of each analytical engine for the scenario in the scope. A 5-star rating means that the particular engine is a good fit for the specific use case. A 3-star rating means that the required scenario may be accomplished by using the particular analytical engine, but with some limitations or considerations. Finally, a 1-star rating means that I don’t recommend using the particular engine for that specific use case.

Read more