Recently, a colleague experimented by handing a complex user request verbatim to Databricks Genie “One” (the data consumer-facing, data-aware “AI Coworker” in Databricks). The only additional context they gave Genie One was to use the silver and gold schemas, and bronze only if really needed.

The request was for information on the “typical” residential urban low-voltage (LV) network. Histograms over customer counts, length, Transformer types, Conductor types, solar and battery installs, etc.

A Primer

To understand the analysis, you need a quick primer. An Electrical Distribution Network is divided into Feeders, long strings of connected poles and wires that originate at a Zone Substation and extend kms out into the suburbs.

Feeders are divided into Segments (protected by devices that isolate each segment in the event of a fault). A segment that starts at a Distribution Substation (a Site) or LV Transformer (an Asset) is an LV Segment, and it stores information such as premise count and length.

In our silver canonical asset model, we model segments like this:

• Asset. All assets in the electrical network (poles, wires, transformers).

• Site. A location into which an asset is installed. A Distribution Substation Site contains several Assets, including an LV Transformer. Over the years, multiple transformers may be replaced at the same Distribution Site.

• Segment. The subset of Assets on a Feeder protected by a protection device.

• Segment-Asset and Segment-Site. The many-to-many relationships between a Segment and the Assets and Sites on it.

• protection_asset_fk. A pointer from Segment to the Asset that is the protection device for that Segment.

• protection_site_fk. A pointer from Segment to the Site that is the protection device for that Segment.

Take note that there are four paths you can take to get from Segment to Asset, two of which are mutually exclusive. An LV Segment has a protection_asset_fk relationship if its protection device is an Asset (an LV Transformer), or a protection_site_fk relationship if its protection device is a Site (a Distribution Substation).

Request 1 - Transformer Rating

The requester asked for “transformer ratings (to identify the most common size)”. Here is the query Genie generated:

SELECT
  asset.rating AS transformer_rating_kva,
  asset.phases,
  COUNT(*) AS count,
  ROUND(COUNT(*) * 100.0 / SUM(COUNT(*)) OVER(), 1) AS percentage
FROM slv_asset.asset AS asset
JOIN slv_asset.segment AS segment
  ON asset.site_fk = segment.protection_site_fk
WHERE segment.segment_type = 'low voltage'
  AND asset.asset_type = 'Transformer'
  AND asset.asset_status = 'Active'
  AND asset.rating IS NOT NULL
  AND segment.segment_feeder_name IN (
    SELECT df.asset_label
    FROM slv_asset.feeder df
    WHERE df.category = 'Urban'
  )
GROUP BY asset.rating, asset.phases
ORDER BY count DESC

As you can see, Genie used tables from the silver asset canonical model to respond to this question:

• slv_asset.feeder - Feeder classification (Urban/Rural)

• slv_asset.segment - LV segment lengths, premise counts

• slv_asset.asset - Transformer ratings, conductor materials

What it got right

• The requester asked for “urban” feeders. There is a category column on the feeder table that was correctly used to filter by category = 'Urban'. Nice!

• It correctly found the rating attribute on the asset table, filtered assets where asset_type = 'Transformer', and, surprisingly, also selected only 'Active' Transformer assets, which is desirable.

• It correctly filtered to low-voltage segments only (segment.segment_type = 'low voltage').

• It generated a percentage, since our requestor is looking for a “typical” residential urban LV network.

Most impressive of all, it worked out that the primary table it needed was segment, even though the requestor never mentioned segments. They asked about the “residential urban low voltage (LV) network,” and the AI correlated that description with low-voltage segments. We put that down to the extensive table and column descriptions we have on those tables.

What it got wrong

It under-reported Transformers. Genie didn’t recognize that sometimes the protection device on a low-voltage Segment is an LV Transformer asset directly, via protection_asset_fk, rather than via its parent Distribution Substation Site. Those Transformers were excluded from the analysis. There is even a flag on Segment called protection_device_is_site that it could have reasoned over, but the context information for it is unclear, so perhaps that was a step too far. It was only 5% of segments, but it could easily have been 30% and produced a materially wrong number.

It short-circuited the join between Segment and Asset. It reasoned that, to reach LV Transformer assets, it needed to find Distribution Substation protection Sites and then obtain the LV Transformer assets at those Sites. An analyst would join segment.protection_site_fk = site.site_sk, then join asset where asset.site_fk = site.site_sk. Genie short-circuited this and went fk to fk, bypassing Site entirely (asset.site_fk = segment.protection_site_fk). I can’t decide whether this is brilliant or dangerous. Same result in effect, but much harder to comprehend.

It joined Segment to Feeder by name. Despite a foreign key being defined between segment and feeder, it decided to use the feeder Name to join. I have an issue with this one. Joins should always use foreign keys where available, and this could cause issues if the segment and feeder are not aligned on the feeder name. I had assumed we’d missed defining the foreign key relationship, but no, it is defined on the segment table. A very poor choice by the AI. More on how that might happen below.

It didn’t run the query it showed us. It reported the query above, but produced two output tables, one for rating and one for phases, so it can’t have used this query exactly. There would have been two separate queries: one aggregating by rating and the other aggregating by phases only.

Request 2 - Solar/Battery Penetration

In this second example, the user was asking for:

1. DER (Distributed energy resources) penetration

2. Mean number of customers with solar installed

3. Mean number of customers with battery systems installed

Genie One produced this query:

WITH urban_substations AS (
  SELECT DISTINCT sd.protection_asset_label AS substation_label
  FROM slv_asset.segment sd
  WHERE sd.segment_type = 'low voltage'
    AND sd.segment_feeder_name IN (
      SELECT df.asset_label
      FROM slv_asset.feeder df
      WHERE df.category = 'Urban'
    )
)
SELECT 
  n.substation AS substation_label,
  COUNT(DISTINCT n.NMI) AS total_customers,
  COUNT(DISTINCT CASE WHEN der.gen_type IS NOT NULL AND der.gen_type != '' THEN n.NMI END) AS solar_count,
  COUNT(DISTINCT CASE WHEN der.storage_type IS NOT NULL AND der.storage_type != '' THEN n.NMI END) AS battery_count,
  ROUND(COUNT(DISTINCT CASE WHEN der.gen_type IS NOT NULL AND der.gen_type != '' THEN n.NMI END) * 100.0 
    / NULLIF(COUNT(DISTINCT n.NMI), 0), 1) AS solar_penetration_pct,
  ROUND(COUNT(DISTINCT CASE WHEN der.storage_type IS NOT NULL AND der.storage_type != '' THEN n.NMI END) * 100.0 
    / NULLIF(COUNT(DISTINCT n.NMI), 0), 1) AS battery_penetration_pct
FROM gld_location.dim_nmi n
INNER JOIN urban_substations us 
  ON us.substation_label = n.substation
LEFT JOIN dev.gld_itg_{saas}.der_register der
  ON der.nmi = n.NMI
WHERE n.NMI_Status = 'Active'
GROUP BY n.substation
ORDER BY total_customers DESC

As you can see above, it used the following additional tables over Query 1:

• gld_itg_{saas}.der_register - Solar and battery installations.

• gld_location.dim_nmi - NMI-to-substation mapping (NMI = National Metering Identifier, the unique ID for an electricity connection point)

What it got right

• It discovered tables that could help it answer the question. Whether it should have used those tables is another question.

• Its calculations appear correct.

What it got wrong

• It used an integration schema table. gld_itg_{saas}.der_register is a schema we created for a SaaS vendor’s integration. Its purpose is purely for integration and not analytics. Now, we did tell it to use the gold schemas, and an integration schema carries a gold prefix, so that’s partly on us. The problem is that nothing in the catalog marked that schema as unfit for analytics, while several trusted, analytics-focused gold schemas hold the same solar and battery information.

• It mixed production and development data. It picked the DEV version of gld_itg_{saas}.der_registerpossibly because it wasn’t deployed to production yet.

• It made dubious connections between tables. It joined a gold analytics dimension, gld_location.dim_nmi, directly to a silver table via a label, and then directly to an integration schema? A human analyst would question their choices at this point.

• It compensated for dubious joins by an overreliance on DISTINCT. The use of DISTINCT in an analytics query is a code smell, serving as a workaround for a modeling issue. That’s exactly what is happening here.

So how does Genie One choose its tables?

To answer that, you need to understand the Genie Ontology, the automated context Genie One relies on. To quote this Databricks article: “Genie Ontology automatically extracts snippets of knowledge from tables, queries, dashboards, pipelines, and connected apps, and organizes that knowledge into a living graph of how a company works and what the data inside actually means.”

What troubles me here is the reliance on past queries. There’s no guarantee those queries were ever valid or effective in the first place, and I suspect this is the root cause of the questionable choices Genie made.

Lessons and takeaways

1. It’s a good start, and an analyst’s knowledge is still extremely important. Genie did a good job discovering the right tables in the right schema, mostly, and how to filter them. An analyst could iterate over this output, with or without Genie. But it would take someone who understands the model in greater detail than the AI does and has better judgment to reach the right outcome.

2. Without six person-months of prior effort, the AI had a 0% chance. All that work to shape the data into the segmentation model is what made this possible at all. The source is spread over multiple source systems, in formats and shapes incompatible with this kind of analysis. The architecture, analysis, design, modeling, and engineering work are still necessary to achieve this result.

3. Modeling still matters, especially Gold dimensional modeling. There were four paths from Segment to Asset. Genie chose one, short-circuited the join, and under-reported transformers. In a dimensional model, every dimension has a direct relationship to its fact. Hence, the model has far less reasoning to do when joining data across tables, and the outcome is far more deterministic. If we had an “asset lifecycle” fact with one row per asset, directly related to the feeder, segment, site, and asset dimensions, the query would become a simple filter on segment.segment_type = 'low voltage', feeder.category = 'urban' and asset.type = 'Transformer', with metrics counted over the filtered fact.

4. Context matters even more. The descriptions of columns and tables in our Unity Catalog are what enabled Genie to get even close here. It found segment for a request that never mentioned segments. But it also picked a DEV integration schema because nothing told it not to. Context and ontology matter more than ever, and so does context about what not to use.

5. Past queries as context are a problem. Genie draws on query history, which captures what people did rather than what the model means, including every shortcut taken in every one-off and prototype analysis. That is my best explanation for the segment-to-feeder join by name: even though the foreign key was defined and available, the AI still favored a pattern that a human had used before.

6. Over eagerness. Genie just seemed overeager, as AI does, to find a way to solve the problem. A human might pause on the issues of cross-schema joins and push back. They might search further for other solutions, or suggest a model change to answer the question properly. In other words, apply some judgment.

7. I see the emergence of a new breed of Data Analytics Agents. New data analytics agents with specific jobs like fact-checkers, validators, and quality scorers that form part of an AI data analytics workflow.

8. And finally, a new emphasis on modeling over reporting outcomes. Until now, Data teams have struggled to secure business buy-in to put in the necessary effort to model a data warehouse/lakehouse properly. The Businesses’ emphasis on reporting and project outcomes over all else has led to poisoned models and data swamps. That status quo cannot stand in the face of data analytics AI agents and their need for context. There is now a direct correlation between data model quality and business outcomes.

Note: We used Genie “One” for this experiment, which is a Databricks consumer-facing, data-aware “AI Coworker” in Databricks. Databricks has since introduced “Genie Agents,” where, for a given domain (max 30 tables), you can provide curated “context” for analytics. I think curated context, combined with excellent Gold-layer dimensional modeling and semantic models, will greatly enhance the success of AI reasoning for a request like this. Hmm… the next experiment. Stay tuned!

An Analytics agent, given a natural language question, can reason over your Data Warehouse data model(s), generate code (SQL), and produce answers to those questions. An analytics agent is different from a coding agent. Coding is open-ended and rewarded for creativity. An analytics agent is constrained, and there is generally only one correct answer.

Anthropic describes their Analytics Agent solution as follows:

The Problem

There are three problems they encountered in their initial efforts to use an analytics agent to answer users’ questions:

1. Which Entity? From the 1000s of entities in your data warehouse across various layers, an analytics agent could reason that it can use 10-100s of them to answer the question. Which one does it use?

2. Agent Knowledge staleness: The agent’s knowledge of the data model failed to keep pace with evolving changes to the model; its reasoning went stale.

3. Retrieval Failure. Even if the data model can answer the question, given the vastness of the “search space” (i.e., thousands of entities at multiple layers), does the AI agent find what it’s looking for?

The Solution

Anthropic proposes (uses) the following solution, which they describe as an analytics stack:

1. Bottom Layer - Data Foundation: A data foundation and “single source of truth” to shrink the “search space” of plausible entities that answer any given question. The data warehouse schema, transformations, and the metadata that describes it. They specifically mention “dimensional modeling”. Data Foundation is primarily aimed at the ambiguity problem. “The fix is fewer, more heavily governed logical models: curate a small set of canonical, single source-of-truth datasets.” “The goal is that when an agent searches for a concept, it finds a single governed answer.” Treat Metadata as a first-class product and colocate it with your codebase. Coding agents work because the codebase is legible; do the same for your Analytics Agent.

2. Source of Truth: If data foundations are the data warehouse itself, sources of truth are the reference surfaces the agent consults to navigate it. This is your semantic layer that defines metrics and further reduces ambiguity. “If a question maps cleanly to a defined metric, the agent calls a function and gets one number, the same number every other surface in the company produces.” When a matching metric is not found, lineage allows the agent to reason about which upstream model to use. They also pipe in a company knowledge graph consisting of indexed docs, roadmaps, decision logs, and organizational structure, so the agent can resolve ambient references and ask better-clarifying questions.

3. Skills: Skills define the agent’s procedural knowledge: which datasets to consult, in what order. I.e., “try the semantic layer, but if that doesn’t work, consult these 30 reference files for this domain”. The reference files describe the tables in the domain, explicit routing triggers (if question about X, then use Y), and gotchas. These skill files live in the same repo as the code, and they use a code hook to ensure they are updated when the code changes (addressing the staleness issue). “Without skills, Claude’s ability to answer analytics questions accurately didn’t exceed 21% on our evals. Adding skills gets these numbers consistently above 95% in aggregate.”

4. Top Layer - Validation: Anthropic discusses offline validation, in which a known answer is paired with a question, and the agent is evaluated on its ability to generate that answer given the question. They have elaborate skill validation infrastructure, even using an adversarial review skill to “aggressively challenge all underlying assumptions on a potential final answer, " which increased accuracy by 6%.”

Interestingly, things that didn’t work:

• Using an LLM to auto-generate the semantic layer metric definitions from raw tables and query logs.

• Giving the agent access to thousands of prior queries. This moved accuracy by less than a percentage point.

My Takeaway

My simple takeaway is that modeling still matters, especially Dimensional and Semantic modeling, and context (metadata, data descriptions, metrics, lineage, etc.) matters even more! This is nothing new; data practitioners have been banging on about this for years. It’s just that we have had Human “agents” who can smooth over gaps and sort through ambiguity to arrive at the answers the business needs. With AI Analytics agents, that ambiguity is no longer acceptable. Data Teams need to redouble efforts in the modeling and context areas (hooray!). If your job is primarily building analytics dashboards, etc., your job may evolve to work down the stack, defining semantic models, context, and skills for AI agents to use.

I also believe that AI in the data space won’t lead to major job losses. The pent-up demand for analytics within the organization is massive and ever-evolving, and the data team currently fails to meet it.

Anthropic Article - How Anthropic enables self-service data analytics with Claude | Claude by Anthropic

• What is a Data Warehouse and Why do you need one?

• Modern Data Warehouse Architecture.

• An Introduction to Dimensional Modeling, then dig further into the Star Schema topic.

• Now you are ready to understand Slowly Changing Dimensions and Fact tables.

• A Semantic Layer is essential to your overall architecture.

• Last but certainly not least are the ETL and Staging topics.

And as a bonus:

• Some articles on Project approach, method and management.

Traditionally, the Kimball methodology for data warehousing has advocated using integer surrogate keys to uniquely identify dimension records. This approach has been widely adopted due to its storage, indexing, and query performance efficiency. However, with the advent of Data Vault modeling, the use of hash keys as surrogate identifiers became popular.

The rationale behind hash keys in Data Vault is that they provide deterministic identifiers that can be generated on the fly based on business keys, eliminating the need for lookups when inserting records. This concept has since influenced broader data warehouse architecture, leading some to use hash keys as surrogate keys in traditional dimension tables.

However, using hash keys in dimensional models introduces significant drawbacks that outweigh any perceived benefits. This article explores why hash surrogate keys should be avoided in a data warehouse and makes the case for sticking with integer surrogate keys.

The Case AGAINST Hash Surrogate Keys in a Data Warehouse

1. Hash Keys Do Not Eliminate Lookups for Type 2 Dimensions

One of the primary arguments for using hash keys is that they allow fact tables to reference dimension records without requiring lookups during fact table loads. However, this is only true if all dimension attributes are Type 1 (i.e., they do not track historical changes).

In reality, many dimensions contain Type 2 attributes. This means a lookup is required during a Fact load to find the correct historical version of a dimension member before assigning a foreign key in the fact table. Since lookups are still necessary for Type 2 dimensions, the supposed efficiency gain of using hash keys is diminished in practical data warehouse implementations.

2. Hash Keys Are Larger and Increase Storage and I/O Overhead

A key consideration in any data warehouse design is the size of foreign keys in fact tables. Fact tables tend to have many rows and are frequently accessed. The standard integer data type in most relational databases is 4 bytes, whereas a SHA-1 hash key is 20 bytes. SHA-2 can be even larger, depending on the bit length used (e.g., SHA-256 is 32 bytes, and SHA-512 is 64 bytes).

This difference in key size has serious implications:

• Fact tables with many foreign keys experience significant storage bloat when using hash keys.

• Increased I/O due to larger row sizes results in slower query performance.

• Indexes and joins become less efficient because hash keys are more complex to compare than integers.

3. Hash Keys Reduce Query Performance

Indexes are crucial in optimizing query performance, and integer-based indexes perform significantly better than hash-based indexes. Hash values are inherently more random and do not cluster well in typical B-tree indexes used by relational databases. As a result, joins between fact and dimension tables using hash keys are often slower than joins using integer surrogate keys.

PostgreSQL users, for example, have noted this issue in practice, as discussed in this Stack Overflow post.

4. Increased Computational Overhead

Using hash keys requires additional computation at multiple stages of the ETL process:

• When inserting a new dimension record, the hash must be generated.

• When inserting a fact record, the hash must be recalculated to determine the correct dimension reference.

This added processing burden can slow ETL jobs, especially for large data loads. In contrast, integer surrogate keys are simple, auto-incrementing values that require no computation.

The Case FOR Integer Surrogate Keys

Given the drawbacks of hash keys, integer surrogate keys remain the best practice for data warehouse dimensions. Here’s why:

1. Reduced Fact Table Storage and I/O

Fact tables often contain millions or even billions of rows. Using 4-byte integers instead of 20+ byte hash keys reduces storage requirements and improves query performance by minimizing I/O. If the number of unique dimension records is expected (very unlikely) to exceed the 4.2 billion limit of a standard int, then a bigint (8 bytes) can be used, which still offers significantly better performance than hash keys.

2. Better Performance in Joins and Queries

Integer surrogate keys are optimized for B-tree indexing, leading to faster joins between fact and dimension tables. Since relational databases are designed to efficiently manage integer-based primary keys, using them ensures better query execution plans and overall system performance.

3. Flexibility with Different Integer Data Types

Integer-based surrogate keys offer flexibility in choosing the optimal data type:

• tinyint (1 byte): Stores up to 255 values.

• smallint (2 bytes): Stores up to 32,767 values.

• int (4 bytes): Stores up to 4.2 billion values.

• bigint (8 bytes): Stores up to 9 quintillion values.

Using the smallest practical integer type minimizes storage requirements while maintaining efficient join performance. Moreover, different integer types can be joined without conversion in most relational databases, so mixing smallint, int, and bigint across tables does not create significant performance issues.

The Evidence!

Recently, a colleague of mine built an example Fact table with just three Dimensions. He built two versions, one with bigint surrogate keys and the other with hash SHA2 512 surrogate keys. This test would have been better with integer instead of bigint and SHA1, but this is still representative of the results.

• Fact table - 673M rows

• Asset Dimension 10M rows

• Rate Type Dimension 3 rows

• Failure Dimension 4 rows

Here are the results:

The Fact Table Integer version was 1/3 the size. The Fact table will gain the most advantage because it has multiple Dimension foreign keys.

The Integer version was 42% faster.

The query performance of the integer version was up to 4 times faster.

Addressing Fact Table Lookups for Dimensions

One concern raised against integer surrogate keys is the need for lookups when inserting fact records. However, this is only necessary for Type 2 dimensions. For dimensions that are guaranteed to contain only Type 1 attributes, the need for lookups can be eliminated through strategic design choices. Follow this decision tree to determine what type of surrogate key to use.

Although this method does introduce some variability and, therefore, complexity, it is well worth the time it saves in ETL load processes. I recommend a comment on each Dimension’s surrogate key explaining how the surrogate key is formed.

1. Using Smart Keys

Certain dimensions, such as the Date dimension, can use a smart key that encodes meaningful information. For example, a date key might be stored as an integer in YYYYMMDD format (e.g., 20250124). This eliminates the need for a lookup while keeping keys small and efficient.

2. Integer Natural Keys

If a dimension’s primary key is an integer and never requires Type 2 changes, it can be used directly as the foreign key in the fact table. Should the need for Type 2 tracking arise later, a new surrogate key system can be implemented while preserving historical data integrity. For example, the surrogate key can be set using a formula like max(surrogate_key) + row_num.

3. Character-Based Natural Keys (When Justified)

Although Kimball generally recommends against using character natural keys, there are cases where it can make sense—particularly if the key is short and unique (e.g., product codes, state abbreviations). A prefix, such as K-, can be used to distinguish keys from other attributes. If composite keys are required and their combined length exceeds 20 bytes, a hash may be justified, but this should be an exception rather than the norm.

Conclusion

While hash surrogate keys have a place in Data Vault modeling, they are not well-suited for traditional data warehouse dimension tables. The disadvantages—larger storage requirements, poorer indexing performance, increased computational overhead, and slower queries—far outweigh any perceived benefits.

For best performance and scalability in a dimensional model, integer surrogate keys remain the superior choice. Fact tables benefit from reduced storage and I/O, joins are optimized, and overall query performance improves. By thoughtfully designing dimension keys and employing lookup strategies only where necessary, you can build a highly efficient and maintainable data warehouse.

Key Takeaways:

• Use integer surrogate keys whenever possible.

• Select the smallest practical integer type to minimize storage.

• Use smart keys for dimensions like Date where feasible.

• Consider integer natural keys if Type 2 changes will never be needed.

• Avoid hash keys unless dealing with exceptionally long composite natural keys.

By following these principles, your data warehouse will remain performant, scalable, and easy to manage.

Before you build a data warehouse, you must define some standards for how you and your team will design and structure your data warehouse system. A Data Warehouse Architecture describes how data is sourced, stored, processed, and accessed. The architecture can vary depending on the specific needs and scale of the organization, but generally, it will follow a layered approach.

There have been various Data Warehouse Architecture articulated by well-known experts in the field. These architectures have included:

• Kimball Dimensional Data Warehouse Architecture

• Inmon Data Warehouse Architecture

• Data Lakehouse Architecture

• Data Vault Architecture

What all these architectures have in common is they are layered, include a dimensional presentation layer and exhibit the characteristics outlined below:

1. Integrated. A data warehouse takes a copy of the data (on a regular basis) from the enterprise’s application systems. It integrates that data into “one place”, simplifying data access for reporting purposes. A second form of integration is to match and merge data from multiple systems into single subject-oriented entities.

2. Subject-oriented, not source system-oriented. A data warehouse reorganises the source data into business subjects/domains, making it easier for users to understand and consume.

3. Historical/Time Variant. A Data Warehouse records the history of how data changes. This is important for accurate reporting, auditing and efficient data management.

4. Non-Volatile. Data is loaded in periodic “batches”. The data doesn’t change from moment to moment; rather, it’s stable between load periods. This means reporting and analysis can be conducted without the data changing underneath you.

Here, you will discover the latest thinking in modern data warehouse design. It represents a blend of previous architectures, selectively incorporating the best elements from established designs.

Modern Data Warehouse Architecture

The modern Data Warehouse architecture has evolved to meet the growing demands of big data and data science, leveraging new technologies to do so. It selectively integrates the most effective elements of traditional architectures, resulting in a mature, comprehensive platform that can address a wide range of use cases. The key factors driving this evolution include:

• New “big data” technologies like Hadoop and Spark (e.g. Databricks).

• A desire to unify data warehousing and business intelligence with new requirements like machine learning, data science, and reverse ETL.

• The advent of Cloud data platforms (like Databricks, Snowflake and Fabric).

• Many years of experience of leading practitioners.

The modern data warehouse architecture consists of 6 Layers, including the Visualization layer. The diagram below summarizes each layer and its purpose and contrasts it with the Data Lakehouse architecture.

This may seem unnecessarily complex, but the layered approach simplifies the architecture technically and conceptually, with each layer serving its role and purpose. The design principles behind this layered approach are well-known software engineering principles that have matured as software design and delivery have matured. They include:

• DRY (Don’t Repeat Yourself): Avoid redundancy by ensuring that every piece of logic is represented only once in the codebase.

• Separation of Concerns: Divide into components, each addressing a separate “concern” or doing a different “job” to simplify manageability and maintainability. The layered approach is an example of the separation of concerns.

• Patterns: Data engineers who work with data warehouses or lakehouses for an extended period will begin to identify recurring code patterns. They often repeat the same code over and over for multiple entities. Adopting a layered approach with a “separation of concerns” can help isolate these patterns, allowing for standardization and, eventually, automation.

• Portability: Data platforms change over time, and new platforms are arising at an ever-increasing pace. You don’t want to be locked into any one vendor. At Dimodelo, we emphasize defining architectures and patterns that can be implemented with any data platform, ETL tool, or transformation language. The tools can change, but the architecture, standards, and patterns remain the same.

A key aspect of the modern data warehouse is how data flows through each layer. The process of “moving” the data is known as ETL, an acronym for Extract, Transform and Load. The diagram below is an overview of the various processes that occur to load each of the layers. The layered approach allows for the separation of concerns at each layer. This, in turn, allows us to identify, implement and even automate common patterns and standards at each layer.

You’ll notice that the modern data warehouse architecture shows a relationship to Data Science/Machine Learning and Data Integration/Reverse ETL. I address these aspects in the “Extending the Data Warehouse Architecture” Section.

Layers of the Data Warehouse Architecture

Landing Layer

The sole function of the landing layer is to “land” or store the data retrieved from source systems, which is ready for further processing. In today’s world, this data can be in various formats, such as structured (tabular), semi-structured (e.g., XML, JSON), and unstructured (e.g., emails, documents). The Landing layer must be capable of capturing all of it. The Landing layer is the equivalent of the underlying data lake of a Data Lakehouse medallion architecture.

Typically, the Landing layer is implemented as a file system (such as HDFS or Data Lake) where an ETL Extraction tool deposits files representing data extracted from source systems. The ETL tool will read structured or semi-structured data in its raw format and write it in a standardised format (e.g., parquet, proprietary, CSV, etc.). This standardised format helps standardize and automate the downstream Staging layer ETL patterns.

The Landing layer data can be transient. That is, once a Persistent Staging Layer has captured the data, the data can be deleted. However, in some cases, especially for unstructured data, there may be good reason to keep the data in the Landing layer, typically for machine learning or data science purposes.

Landing layers are often loaded with dedicated Extraction tools like Airbyte, Fivetran or Qlik. These tools specialize in connecting to source systems, extracting data incrementally where possible, and landing it in the Landing layer. They typically excel at this job but don’t work well (if at all) at transforming the data.

The data schema in the Landing layer model will match the source systems schema.

Typically, the landing layer is only accessible to data engineers responsible for creating and maintaining it. The exception may be data scientists working with unstructured data.

Extract patterns tend to fall into one of the below:

• Full Extract. All the data in the source entity is extracted.

• Incremental Extract. Only the changed data is extracted from the source. This method is much faster than a Full extract. However, the source entity must support it. For the pattern to work, you need to identify a column or multiple columns that the source system uses to track change. They would usually be a modified date or sequential identifier. The downside of this pattern is that it doesn’t detect hard row deletes from the source. That makes it necessary to run a period Full Extract as well.

• Change Tracking. The change tracking pattern uses the change tracking mechanism of the source system(e.g., change tracking in Microsoft SQL Server) to identify changes in the source entity and extract only changed data. The Change Tracking pattern has an advantage over the Incremental pattern in that it recognizes hard deletes and is, therefore, a superior pattern if available.

• File Pattern. Extracting files is quite simple—it’s effectively a copy-and-paste. However, files can be very unreliable and variable in nature. There may be many files in the source that represent a single entity. The files might represent Full, Incremental, Change tracking or Date Range extracts. Files are probably the most unreliable with respect to their schema stability. From an extract perspective, file sources are reasonably simple but quite complex from an ingestion standpoint.

• Date Range Pattern. The date range pattern is used to extract a subset of the source data based on a date range. The end of the date range is the “current” date, and the start range is the number of days before the current date. The date range ‘window’ moves forward a day as the current batch effective date changes. For example, you might have a very large transaction table that doesn’t support either incremental or change tracking. Instead of doing a Full extract every day, you might only extract the last 31 days of transaction data each day to capture any changes that have occurred in the last month.

(Persistent) Staging Layer

Traditionally, the Staging layer served as a temporary area where raw data was collected, cleaned, and prepared before being transformed and loaded into the Presentation layer. In the modern data warehouse architecture, the Landing layer now assumes the role of data “collection”, while the Staging layer has evolved into a “persistent” Staging layer that permanently records the source data and its change history.

The Staging layer is the equivalent of the Bronze layer of the Data Lakehouse medallion architecture.

The Persistent Staging layer serves as a reliable foundation of the data warehouse. If all else fails (literally), the higher data warehouse layers can be rebuilt from this foundation. Indeed, as long as the Persistent Staging layer stays consistent, you can confidently change, modify and reload higher layers with full history without losing data. This wasn’t possible with older data architectures where the Presentation layer was the first persistent layer in the architecture. The concept of a Persistent Staging layer evolved because of problems with reloading and changing the Presentation layer, which resulted in historical data loss.

The schema of persistent staging data is tabular based, in either a database table format or file-based formats that lend themselves to tabular formats, such as Parquet with Delta table or Apache Iceberg.

The schema of the persistent staging layer closely mirrors the source system schema, except for some additional data management columns used for handling history and ETL processes. While the format might change, such as converting from JSON to tabular, resulting in some flattening, the column names and structure remain the same. There should be no data transformation.

Modern data platforms like Databricks, Snowflake, and Microsoft Fabric can directly read files from a Landing layer. The code required to write data and maintain history in a target persistent staging table can be implemented on your preferred data platform.

Typically, the ingestion pattern for Persistent Staging entities can be standardised.

• Identify the delta change data set, I.e., only the data that has changed since the last execution. This depends on the extract pattern used to extract the data from the source. Identifying the delta change set is easy for Incremental and Change Tracking patterns. A full comparison with the target persistent staging entity is required for Full extracts. A full comparison with the target Persistent Staging entity over the date range is required for date range extracts. For file extracts, it’s dependent on the pattern the file represents.

• Once identified, the delta change set can be applied to the target persistent staging entity in a straightforward and standardized way.

Modelled Layer

The “modelled” layer is also known as the “Core” or “Middle” or “Reconciled” or “Transformed” layer. We chose the word “Modelled” because it seems the most descriptive to us.

The Modelled layer marks the beginning of the data’s transition into a subject-oriented format. This layer is where the complex process of transformation takes place. Here, data from various systems is commonly consolidated and transformed into entities within a subject area. Data from multiple systems is matched and integrated to form a whole-of-enterprise view of key data entities. It serves as the central enterprise business model on which all enterprise reporting is built, not directly but as the foundation. The Modelled layer is the equivalent of the Silver layer in the Data Lakehouse medallion architecture.

The important thing is that the data model is subject-oriented. “Subject-oriented” refers to the organization of data around the key subjects and entities of the enterprise rather than around the source system representation (schema) of entities. Data is grouped by major subjects with entities relevant to that subject area, such as customers, products, sales, or orders. Each subject area is designed to provide a comprehensive view of the data related to that area.

Subject-oriented data modelling ensures that sometimes complex business logic is captured in one place, adhering to the DRY principle. This makes data consistent and standardized across the enterprise, reducing redundancy and discrepancies. Organizing data around subject areas fosters a common understanding across the enterprise and makes accessing data related to specific areas of interest easier for developers.

Remodelling your data from the source schema to your own subject area model decouples higher layers (i.e., presentation, semantic, and reporting) from source systems. If a source system is replaced (as it often is), it just requires “plugging in” the new source system to the existing Modelled layer schema. No reports, presentation, or semantic layer code must change (in theory).

The Modelled layer data model can take a few forms. Different teams and data architects have their preferences. The options include:

• 3rd Normal Form. 3rd normal form is highly normalized data usually used in the database of operational systems. It’s often overkill for the Modelled layer of a data warehouse, but it does provide norms and structures for developers to follow. Bill Inmon’s corporate information factory is an example of this approach.

• Data Vault. A Data Vault is designed for the modelled layer of a data warehouse. However, practitioners often find it difficult to understand and apply consistently. The criticism of data vault is its “explanation tax” where you “spend a lot of time explaining the nuances to stakeholders, both technical and non-technical”. I personally am not a fan. It’s a lot of work for little return.

• Pragmatic. A pragmatic approach simply treats the Modelled layer as a stepping stone to the presentation layer. It doesn’t follow any particular rule set or modelling technique. It’s usually highly denormalized. It can work for smaller teams where the primary outcome is business intelligence, reporting, and analytics. It requires the least effort but can become unruly after some time.

• Hybrid. A hybrid approach is a cross between the 3rd Normal Form and the Pragmatic. It still identifies key subject-oriented entities and their relationships but is pragmatic in allowing denormalization to support the presentation layer.

At Dimodelo, we favour a hybrid approach. The modelled layer must trade off effort and utility. We favour a third Normal Form schema, with denormalized tables wherever possible. Modelling is an art, not a science, so it’s difficult to define an exact description of how this is achieved. In later posts, I’ll develop some examples.

Modern data platforms such as Databricks, Snowflake, and Microsoft Fabric provide procedural languages like SQL and Python that you can use to implement the entities in the Modelled layer.

Presentation Layer

The presentation layer is the final subject-oriented representation of the data designed specifically to support business intelligence use cases such as reporting, ad hoc analysis, and dashboards. It is modelled using a strict Star Schema, which is comprised of Dimensions and Facts and their relationships. Data in the Modelled layer is read and transformed into the Dimensions and Facts of the Star Schema. The Presentation layer is the equivalent of the Gold Layer in the Data Lakehouse medallion architecture.

The Star schema is used because of these characteristics:

• The Star Schema is an easy-to-understand and navigate data model that supports multiple analytic use cases.

• Because of its simplified joins, a Star Schema supports high-performance aggregated queries. It performs well over many different analytics use cases.

• Star schemas are broadly accepted and used across the industry.

• The Star schema is the schema expected by Semantic layer tools. The semantic layer can consume Star Schemas as-is and apply various compression and pre-aggregation techniques to further enhance aggregated analytical query performance.

Because of the nature of the Star schema, it is easy to implement standardized patterns to load Facts and Dimensions. Check out our Slowly Changing Dimension article for an example.

Modern data platforms such as Databricks, Snowflake, and Microsoft Fabric provide procedural languages like SQL and Python that can be used to implement the entities in the Presentation layer.

Semantic Layer

The semantic layer is typically exposed to end users for business intelligence, ad-hoc analysis, reporting, and dashboard use cases. End users interact with the semantic layer via visualization tools like PowerBI or Tableau.

A semantic layer is implemented in a technology that facilitates high-performance aggregated queries over large amounts of data.

The semantic layer’s data model mirrors the presentation layer’s Star Schema. It further augments the data model with calculated measures and hierarchies that other technologies cannot create.

There are several styles of Semantic layers, including materialized, virtual or hybrid. There are also various tools and vendors. PowerBI implements a semantic layer. Other materialized/hybrid semantic layers include Kyligence, GoodData, Apache Druid and Apache Pinot. Some virtual/hybrid semantic layers include Cube, Malloy, LookML, Metriql, AtScale, MetricFlow, Metlo, and Denodo.

The Ralph Kimball approach to Data Warehouse architecture focuses on creating smaller data marts optimized for specific business areas with conformed dimensions unifying the data marts. Practically speaking, the value of a data mart is to present a subset of the data warehouse that only includes data relevant to a business area or user group. This can be achieved by developing multiple semantic models. The added advantage is the ability to apply security at the semantic model level.

Read a deep dive into semantic layers for further information.

Visualization Layer

The visualization layer of the data warehouse refers to the set of tools and technologies used to present and visualize the data stored within the presentation/semantic layer of the data warehouse. You deliver meaningful insights through various forms of visual representation, enabling users to understand trends, patterns, and relationships in the data. Components of the visualization layer typically include:

• Dashboards: Interactive, real-time interfaces displaying key metrics and performance indicators through charts, graphs, and gauges.

• Reports: Predefined and ad-hoc reports that present data in a structured format, often with tables, charts, and summaries.

• Data Visualization Tools: Software that allows users to self-serve, creating custom dashboards and reports.

• Ad hoc Analysis tools facilitate multidimensional analysis, often providing pivot tables and drill-down capabilities that enable users to explore data and generate insights.

• Business Intelligence Portal: Providing online access to reports and dashboards, organised in relevant folders, groups and departments. User and developers can publish their reports to the Portal, making them available for others to use. The portal provides safeguards like access control.

Examples of popular tools used in the visualization layer include Tableau, Power BI, Looker, and QlikView. These tools provide user-friendly interfaces that enable business users, analysts, and decision-makers to interact with and interpret the data effectively.

A Visualization layer won’t typically store data. It simply queries and presents the data from the presentation/semantic layers.

ETL/Data Integration

“ETL,” is an acronym for Extract, Transform, Load. It describes the process of extracting data out of one data store transforming it, and loading it into another.

Initially, data warehousing comprised only two layers: Staging and presentation. At the time, ETL referred to the process of extracting data from its origin, transforming it to fit the star schema, and then loading it into the Presentation Layer. Although the term ETL is still used today, it’s difficult to neatly apply the concept to the multiple layers of modern data warehouse architecture. In reality, every layer has some aspect of extract, transform and load. Nonetheless, the ETL terminology endures, and we continue to use it when describing the movement of data through the various layers.

Some people characterize ETL in modern data warehouses as data extracted from source systems, transformed into a Modelled layer, and then loaded into the Presentation layer. However, this description does not really depict the reality of the ETL system.

Instead, in the modern data warehouse architecture, I like to think of each Layer having its own mini ETL process:

• Landing Layer. Data is extracted from the source and loaded into the Landing data store.

• Staging Layer. Data is extracted (or, more precisely, queried) from the Landing layer, lightly transformed into tabular format (with history), and loaded into the Staging layer datastore.

• Modelled Layer. Data is extracted (or, more precisely, queried) from the Staging Layer, transformed into a subject-orientated and loaded into the Modelled layer data store.

• Presentation Layer. Data is extracted (or, more precisely, queried) from the Modelled Layer, transformed into a star schema, and loaded into the Modelled Layer data store.

• Semantic Layer. Data is simply loaded into a Materialized Semantic Layer. Data is not loaded into a Virtual Semantic Layer.

I think this description more accurately reflects the reality.

In the following sections, I’ll discuss Extract, Transform and Load in more detail.

Extract

As I’ve discussed, every layer of the modern data warehouse architecture has its own version of “extract”, but in this section, I want to focus on the process of extracting data from a source system and loading it into a Landing layer data store.

A core part of your technical Data Warehouse architecture will be the tool you use to implement data extraction from source systems. Generally, these tools specialize in extraction and exist in your architecture purely to fulfil that role (i.e. separation of concerns). All these tools should have these features:

• Ability to connect to a broad array of source systems and databases (usually in the hundreds).

• The capability to support either batch, data streaming, or both methods of extracting data. Batch is the most common method for Data Warehouses. It involves scheduled extraction of multiple rows of data (i.e., batches) in bulk. Batch extraction is reasonably straightforward and easily understood. Streaming involves continuous real-time or near real-time “streaming” of individual source system transactions. Streaming is more difficult to implement and maintain but may be necessary to support real-time analytics and reverse ETL use cases.

• Support a variety of extract patterns like Full, Incremental, Range and Change Tracking.

• Ability to load data into various target datastores (e.g. AWS S3, Azure Storage) in a standardised format (e.g. parquet, CSV, JSON).

• Support Scheduling, restart and recovery.

• Support workflow monitoring.

There are many dedicated Extraction tools. Some examples are Airbyte, Singer.io, Kafka, Fivetran or Qlik.

Transform

In the early days of Data Warehouses, there was only one “transformation” step from the Staging layer to the Presentation Layer. Today, there are multiple transforms, from Landing to Staging (a very light transformation), Staging to the Modelled Layer, and Modelled Layer to Persistent Layer.

Transformation is the code that is written and executed to transform the data from one layer’s schema/format to another. It typically includes a combination of:

• Matching entities across various source systems

• Merging data from multiple entities across multiple source systems into a single entity

• Joining data across entities

• Cleansing data, identifying and correcting errors, inconsistencies, and inaccuracies.

• Summarizing data at higher grains

• Denormalizing data into larger, consolidated tables, introducing redundancy to support simplified high-performance queries.

• Remodeling data into a subject-oriented schema.

Transformation is typically built within your core data platforms, such as Databricks, Snowflake, or Microsoft Fabric. These platforms provide procedural languages like SQL and Python that you can use to implement transformation logic. At Dimodelo, we recommend tools like “dbt,” which are platform-agnostic, provide data engineering automation, significantly reduce development time, and improve consistency.

Load

I confess that the L( i.e., Load)in ETL has often confused me. Isn’t it just a given that after you transform data, you load it into a data store? What’s the big deal? Well, it turns out that’s pretty much all there is to it!

Each layer in a modern data warehouse possesses its own “Load,” which results from a transformation process. Typically, the target is a single entity within a layer, usually a table in the Staging, Modelled and Presentation layers. Within the Landing layer, an entity is often implemented as a folder that receives all the extracted files corresponding to an entity from the source.

Loading a materialized Semantic layer is generally straightforward. The tool used to implement the Semantic layer will be able to load data, either incrementally or in full, directly from the Presentation Layer.

Data Warehouse Architecture Components

If you are going to implement a data warehouse, there are several technical/software components you need to consider:

• Landing Data Store: A temporary storage area where raw data is initially landed before processing; examples include Amazon S3 and Azure Data Lake.

• Data Extraction Tool: Software that pulls data from various source systems into the data warehouse. Some examples are Airbyte, Singer.io, Kafka, Fivetran or Qlik.

• Data Platform: The foundational infrastructure that processes and stores data within the data warehouse, often including databases and processing frameworks. The compute and storage within the Data Platform are often managed separately and can scale independently. Examples include Snowflake, Databricks, Google BigQuery, and Microsoft Fabric.

• Semantic Layer Tool: A tool that facilitates high-performance aggregated queries over large amounts of data; examples include AtScale and PowerBI.

• Business Intelligence Tool: Software that allows users to visualize and analyze data through reports and dashboards; examples include Tableau, Power BI, and Looker.

• Development:

  • Code Repository: A version-controlled storage space for source code and development artifacts, enabling collaborative development and change tracking; examples include GitHub, GitLab, Azure DevOps and Bitbucket.

  • Agile Management Tool: Software that supports Agile project management methodologies, facilitating sprint planning, tracking, and collaboration; examples include Jira, Trello, and Azure DevOps.

  • Data Warehouse Automation Tool: A tool that automates the transformation of data within the warehouse, enabling modular and maintainable data workflows; dbt (Data Build Tool) is an example.

• Scheduling and Monitoring Tool: Software that manages and monitors data workflows, ensuring that tasks are executed at the right time and in the right order and provides user monitoring of progress. Examples include Apache Airflow and dbt.

• Network, Security, and Access Control: Mechanisms and tools to ensure secure data transmission, protect data from unauthorized access, and manage user permissions; examples include AWS IAM, Azure Active Directory, and Okta.

• Reference Data Management: The process of managing data lists for analytics classification or mapping purposes. It can be as simple as storing .csv files or as complex as master data management systems like Ataccama.

• Data Quality Management and Reporting: Tools and processes to ensure that data meets quality standards, with capabilities for profiling, cleansing, and reporting.

Extending the Data Warehouse Architecture

Data science and Machine Learning

The number one complaint Data Scientists make is how much time they need to spend sourcing and transforming data to prepare it for input into their data science models. With the modern data warehouse architecture, it’s possible to unburden Data Scientists from the chore of sourcing data and instead allow them to concentrate on building insightful models.

Having data from multiple source systems consolidated in one location, such as a data platform, and ensuring it is current is beneficial for data scientists. Ideally, data science models should utilize the “Modelled” layer, where data integration and standard business rules have been applied, enhancing consistency with business operations. However, as the Data Warehouse may often be under development, it is permissible to construct data science models from the Persistent Staging layer. When dealing with unstructured data, which commonly resides only in the Landing layer, it is also acceptable for data science models to derive their data from this layer.

Modern data platforms like Databricks, Snowflake and Fabric are integrated with machine learning frameworks, allowing data science models to be trained, deployed, and monitored directly within the data platform. By extending the data platform with capabilities such as machine learning, automated feature selection, and seamless integration with visualization tools, organizations can empower data scientists to experiment and iterate rapidly. This alignment between data warehousing and data science ensures that models are built on the most up-to-date data and that insights can be operationalized quickly, driving more informed decision-making across the enterprise.

Data Integration/Reverse ETL

Traditionally, Data Integration teams responsible for the system interoperability of operational systems built completely separate data infrastructure and integration workflows to the data warehousing team. In recent years, people have recognized that there is a lot of cross-over in the work data integration and data warehousing teams do. This includes sourcing and transforming data in very similar ways. There is an acknowledgement that the Bronze and Silver layers of a modern data warehouse could be used as the source and transform processes for Data integration.

The modern data warehouse can consolidate the efforts of the data integration and warehousing teams into a centralized data “hub”. More specifically, the modern data warehouse can be used to source and transform data, which is then used as both a source of the presentation layer of a data warehouse and of operational system interoperability through data integration.

This does pose some challenges, as the timeliness requirements of data integration can be near real-time, making the landing and staging processes more difficult to engineer. In these cases, an ETL tool with streaming capabilities would be required.

The buzzword for this kind of data integration is “Reverse ETL”, which is the process of moving data from a data warehouse back into operational systems like CRM, marketing, and sales platforms. Reverse ETL includes the concept of taking the enriched data from the Presentation Layer or Semantic Layer of the data warehouse and pushing it back into the tools where end-users can act on it.

Real-time analytics

Real-time analytics sources, transforms, and analyzes data as soon as it is generated, enabling immediate insights and actions. Unlike traditional analytics, which often rely on batch processing, real-time analytics operates continuously, providing up-to-the-moment insights. This capability is crucial in scenarios where timely decision-making is essential, such as fraud detection, recommendation systems, or IoT device monitoring.

The need for real-time analytics has a significant impact on the architecture of modern data warehouses. Traditional data warehouses, designed primarily for batch processing, are not inherently equipped for the low-latency requirements of real-time data processing. To accommodate real-time analytics, modern data warehouse architectures often integrate technologies like streaming data tools that enable the ingestion, processing, and querying of data in real-time (e.g., Apache Kafka, Amazon Kinesis) and in-memory processing engines (e.g., Apache Flink, Apache Spark).

The name “Star” schema refers to the star-like pattern that emerges when you present the Fact table and its Dimensions in an entity-relationship diagram.

The image above is an example of a Star Schema. It depicts an imaginary video streaming company, GetFlix, and a “Shows Viewed” Fact table. The “Shows Viewed” fact table is at the centre of the star schema and surrounded by its Dimensions, which form the points of the “star.”

Benefits of Star Schemas

A Star Schema focuses on delivering simplicity and query performance for the end user. The benefits include:

• It is easy for users to understand and navigate. Users can easily find and use the data they need for reporting and ad hoc analysis.

• Because of its simplified joins, a Star Schema supports high-performance aggregated queries. It performs well over many different analytics use cases.

• Broadly accepted and used across the industry.

• The source schema expected by Semantic layer tools. These tools can consume Star Schemas as is and apply various compression and pre-aggregation techniques to further enhance aggregated analytical query performance.

• The source schema expected by visualization tools like PowerBI and Tableau. They support these tools’ drag-and-drop nature.

• Eliminates issues present in “third normal form” (3NF) schemas, where multiple alternate join paths can produce different or incorrect query results from the same data. For example, in a Star schema, there is only one way to join “Shows Viewed” to Customer, whereas in a 3NF schema, there may be multiple ways to join from “Shows Viewed” to Customer.

The 3 essential building blocks of a Star Schema

A Star Schema consists of two types of tables (Facts and Dimensions) and their relationships. Thus, it’s simplicity.

The following is a brief introduction to these concepts.

Facts tables

A Fact table represents a business process or an event in a business process. It contains the measures/numbers you want to analyse. For example, the “GetFlix” “Shows Viewed” fact table depicted in the star schema example below contains one row for every time a customer views an Episode of a Show. “Customer views an Episode of a Show” is the business event we are measuring.

Read more about Fact Tables.

Dimensions

A Dimension represents a business entity, which is the people, places, and things that come together to perform business activities. A Dimension contains the attributes (i.e. fields) of the business entity, e.g., Customer Gender, Age, Category, etc. The attributes are used to filter and group fact data when performing data warehousing queries. Attributes values should be descriptive values rather than codes (e.g. “Male” instead of “M”). The reason is that attribute values become column headings in a report. A column heading of “M” is not very informative.

This demonstrates one of the key differences between a Star Schema and a 3NF schema. A Dimension contains descriptive attributes and a lot of data redundancy. It doesn’t follow the conventional ideas of 3rd normal form (3NF) modelling. It is like this on purpose. That is often a hurdle that DBAs and application developers, coming from a traditional database background, need to overcome before they can fully embrace Dimensional modelling concepts.

Read more about Dimensions.

Star Schema Relationships

A Star Schema is a simplified model suitable for reporting and analytics with easy relationship navigation for end users. The image below depicts the relationship between a Fact and a Dimension.

The rules for relationships are very simple. The direction of a relationship is always from a Fact to a Dimension. That is, the Fact contains the foreign key to the Dimension. The cardinality is always 1 to many. A Fact is related to only 1 Dimension row, but a Dimension row can be related to many Fact rows. There are many-to-many joins, but this is resolved through an intermediary “Group” Dimension and “Bridge” fact with two 1-to-many relationships. Read more about dimensional many-to-many relationships.

The best practice is to always have surrogate keys on Dimensions. Therefore, the foreign key on the Fact table is related to the surrogate key of the dimension.

Star Schema vs Snowflake Schema

The Star and Snowflake schemas are two types of Dimensional modelling techniques used in Data Warehousing to organize and structure data for efficient querying and analysis.

Below is the model for a Sales Fact Star Schema vs. a Snowflake Schema for the same Fact.

In the Star schema example, the Product Dimension contains an attribute for the “Brand” of the Product, whereas the Snowflake schema has a separate Brand Dimension with a “snowflake” relationship to the Product Dimension. This is the crucial difference between the Star Schema and the Snowflake Schema. In the Snowflake schema, attributes are normalised into separate Dimensions, whereas they are Denormalised in the Star Schema. The same data exists in the models; just where it is stored is different.

The only advantage of Snowflake Schema is less storage, and in the modern data warehouse, storage is no longer an issue, especially considering Dimensions are usually relatively small compared to Facts. The Star Schema is easier to design and maintain, is easier to understand and navigate, and offers faster performance.

Our recommendation is never to use the Snowflake Schema.

Read more about Star schema vs Snowflake schema.

Star Schemas and Conformed Dimensions

So far, we have discussed using a Star Schema. However, a Star Schema only describes the model for a single Fact table. An Enterprise data warehouse should model many different business processes/events, each with its own Fact table and related Dimensions.

The question is, how do we prevent duplication of the Dimensions used by Facts and facilitate cross-business process analysis?

The answer is “Conformed” Dimensions. Essentially, a conformed Dimension is a Dimension shared by many Facts. Typical conformed Dimensions are key business entities like Employees, Customers, Products, Cost Centres, Dates, and Times. Conformed Dimensions are the “links” between multiple Star Schema models.

When two Fact tables share a Dimension, you can produce a report with measures side-by-side from both Fact tables, categorized or filtered by attributes of their conformed (i.e. shared) Dimensions. In this way, dimensional modelling with conformed Dimensions supports enterprise-wide cross-business process analysis.

Another way of representing conformed dimensions is through a data warehouse matrix.

How to build a Star Schema

Below is the image of the physical Fact table and its Dimensions, implemented in a relational database. The key thing to note is that each Dimension has a surrogate key, and the Fact table has a foreign key to each of those surrogate keys. You may be wondering why there are no physical relationships depicted. This is because, in this case, each table is implemented with a ColumnStore index, and no primary keys are defined. ColumnStore indexes are ideal for analytical style aggregated queries (basically an index on every attribute) and ETL bulk loads. Also, generally speaking, Primary Keys and Clustered indexes will slow down your ETL Load. The foreign to primary key relationships still exist but are not physically implemented.

The full details of the above table schema can be found in the Dimension and Fact articles.

Why you should use Star Schema in Visualization tools like PowerBI

The Star Schema design is crucial for creating Power BI semantic models that are both high-performing and user-friendly.

Each visual in a Power BI report generates a query sent to the Power BI semantic model. These queries filter, group, and summarize the data within the model. An effectively designed model offers tables for filtering and grouping and tables for summarization. This aligns seamlessly with the star schema’s Dimensions and Facts. Dimensions support the filtering and grouping function, while Fact tables support summarization.

From a performance perspective, it has been proven that Star schemas are more performant than “flattened” “one big table” (OBT) tables when used in PowerBI.

• The Star schema is much faster for retrieving values for columns in slicers. This is important for useability.

• The Star Schema model is much smaller. Dimensional models are far more compressible. They consume less memory.

• The Star Schema is faster for more complex queries, and when the flatted model is faster, it’s not by much. Performance is even worse for 3NF models.

• Furthermore, it’s been demonstrated that “flattened” tables can produce inaccurate numbers.

It’s likely that Tableau and other visualization tools experience the same results.

The conclusion here is that Star Schemas are the best schema to use in semantic models that support visualization tools.

Star Schema vs 3rd Normal Form 3NF

The star schema and third normal form (3NF) schema are both database design methodologies, but they serve different purposes and have distinct structures.

3NF is intended for use in operational (OLTP) systems where data integrity and minimization of redundancy are critical. It supports write-heavy operations, ensuring data consistency and efficiency in updates.

Tables are organized to eliminate redundancy and ensure that each piece of data is stored only once. Tables are normalized, which involves dividing data into multiple related tables.

3NF does not perform well for aggregated analytical queries due to the need to involve multiple tables with many joins. Tables are optimized for data integrity and efficient updates rather than read performance.

Overview

Below is the model for a Sales Fact Star Schema vs. a Snowflake Schema for the same Fact.

In the Star Schema example, the Product Dimension contains an attribute for the “Brand” of the Product, whereas the Snowflake schema has a separate Brand Dimension with a “snowflake” relationship to the Product Dimension. This is the crucial difference between the Star Schema and the Snowflake Schema. In the Snowflake schema, attributes are normalised into separate Dimensions, whereas they are denormalised in the Star Schema. The same data exists in the models; just where it is stored is different.

How do Star Schemas differ from Snowflake Dimensions?

Star Schema

1. Structure:

   • Central fact table surrounded by dimension tables.

   • Dimension tables are typically denormalized, containing all necessary attributes.

2. Relationships:

   • Only direct one-to-many relationships between the Facts and Dimensions.

3. Simplicity:

   • Easier for Users to understand and navigate when producing reports.

   • Queries are faster due to fewer joins needed between tables.

4. Normalization:

   • More redundancy because dimension tables are not normalized, which can lead to larger storage requirements.

Read a detailed description of the Star Schema.

Snowflake Schema

1. Structure:

   • Central fact table surrounded by normalized dimension tables.

   • Dimension tables are further normalized into multiple related Dimensions, resembling a snowflake.

2. Relationships:

   • More complex relationships due to the normalization of Dimension tables.

   • Dimension tables can have relationships with other Dimension tables.

3. Simplicity:

   • More difficult to design and maintain.

   • More difficult for Users to understand and navigate.

   • Potentially slower query performance due to the need for more joins.

4. Normalization:

   • Dimension tables are normalized, reducing redundancy and improving data integrity.

   • Queries may involve more joins due to the normalization.

Comparison

• Design and Maintenance: Star schema is simpler and easier to design and maintain, while the snowflake schema is more complex but can offer better data integrity.

• Simplicity: Users find the Star Schema easier to understand and navigate. The Snowflake Cchema requires the User to navigate and use multiple joins to get data from a Fact.

• Query Performance: Star schema typically offers faster query performance because it requires fewer joins, while the snowflake schema might be slower due to the need for additional joins.

• Storage Requirements: Star schema usually requires more storage space due to denormalization, whereas the snowflake schema can be more efficient in terms of storage because of normalization.

Recommendation

Our recommendation is never to use the Snowflake Schema. The only advantage is less storage, and in the modern data warehouse, storage is no longer an issue, especially considering Dimensions are usually relatively small compared to Facts. The Star schema is easier to design and maintain, easier to understand and navigate and offers faster performance.

In the above example, an insurance company receives medical claims. The medical claims are the business event represented by the Fact table. Each medical claim can have more than one associated diagnosis. These groups of associated diagnoses are modelled in a “Group” dimension. Individual diagnoses are associated with the “Group” Dimension via a “Bridge” fact. The “Bridge” fact makes the association between the “Group” and individual diagnosis.

In this scenario, the most challenging thing to build is the “Group”. It doesn’t exist in your source and must be derived. To do this, you need to select all combinations of diagnoses that exist in medical claims. The source of the target fact will usually be the source of your “Group” as well. For example, Let’s say you had these groups of diagnoses that are used on claims.

• A,B

• C,D

• A,D

• B,E

Then, there would be one row in the Group Dimension for each Group. In fact, the concatenation of diagnosis keys is an excellent candidate to use as the business key of the Dimension, i.e. DiagnosisGroup_Id. You may also want to include a concatenation of the diagnosis names as an attribute if you want to analyze the impact of certain diagnosis combinations. For example, by choosing the “C,D” “Group” Dimension member, an analyst can quickly get the claims measures for claims with both C and D, but only C & D diagnoses.

The Bridge is simply another Fact, usually a Fact-less Fact, that just records the existence of individual diagnoses within the group. However, it is possible to put measures on this Bridge. For example, a ratio of the contribution of each individual Diagnosis in the Group to the overall value of the claim. A measure on a bridge is usually some ratio between the individuals. Another example is the ratio of ownership of joint buyers on a Sale.

A Group isn’t always derived, as shown in the example below:

In the example above, a Bank Account has joint Account Holders (Customers). There is no need to derive the Group as it exists as an entity in your source.

Another example might be if you had a Fact that represented the process of a Customer viewing a show on a streaming service. A show is related to many Actors, but the Group is readily available in the source system as a Cast entity that already groups the Actors. In this case, the “Cast” entity can be used as the source of the “Group” Dimension.

Why are Many to Many relationships modelled this way? One reason is that OLAP Cubes and other semantic layer software recognize this format and faithfully produce the correct aggregations when analyzing the Fact by the Individual Dimension. For example, if you want to sum the Medical Claim amounts for a given Diagnosis, you would select the Diagnosis from the Diagnosis Dimension, drag in your Claim Amount measure, and the semantic software would navigate the Bridge and Group relationship to produce the Total Claim Amount for that one Diagnosis.

If you need an introduction to Dimension Tables in general, read our Dimension Table – Introduction article.

What is a Slowly Changing Dimension?

The term “Slowly Changing Dimension” (SCD for short) acknowledges that Dimension Members change, but only slowly, over time. However, most Dimensions are, in fact, slowly changing, so why is the concept of “Slowly Changing Dimensions” so important in Data Warehousing?

The answer is that there are several different “Types” of “Slowly Changing Dimensions,” and each one has a different impact on what data users see in reports and analyses. Each SCD Type represents a different change-handling strategy used to manage changes to Dimension data. Whether users want to report using historical or latest values and how they want to view history dictates the SCD Type you choose to handle changes to Dimension attributes. Therefore, it’s very important you understand the different types of Slowly Changing Dimensions and their impact when designing a Data Warehouse.

Where did the concept of “Slowly Changing Dimensions” come from? Ralph Kimball originally identified the concept to address the requirement of a Data Warehouse to accurately support historical reporting. He defined the 6 now industry-standard change-handling strategies or “Types”. The Types include Types 1, 2, and 3, along with a couple of hybrid techniques, Types 4 and 6.

Although we talk about Slowly Changing “Dimensions”, what we actually implement is Slowly Changing “Attributes”. The SCD change-handling strategies are implemented per attribute. That means different attributes within one Dimension can have different “Slowly Changing Dimension” Types. Developers often think of a whole Dimension as being Type 1, 2, etc. That is a mistake. A Dimension can have Type 1, 2, 3 etc attributes at the same time. The ETL patterns we describe in this post show how to handle multiple SCD Types in one Dimension.

So, to sum up, the term “Slowly Changing Dimensions” is kind of confusing. Why is it important to call out the slowly changing nature of Dimensions if almost all Dimensions are slowly changing? The reason is the different ways of handling slowly changing “attributes” and the impact they have on reporting.

The following sections discuss the SCD Types in detail and describe how to write ETL code to implement the SCD Types.

Types of Slowly Changing Dimensions

People generally agree there are two common SCD Types, 1 and 2, and some less-used SCD types, 3, 4 and 6. The list of Slowly changing dimension types follows:

• SCD Type 1 – Don’t keep history, i.e. overwrite.

• SCD Type 2 – Keep a version of the dimension member for every change.

• SCD Type 3 – Keep the latest and prior versions (but not every version).

• SCD Type 4 – Current and History Tables.

• SCD Type 6 – A hybrid combination of Type 1, 2 and 3.

We use the following example to describe each of the Slowly Changing Dimension Types.

The star schema below depicts a typical Sales scenario. Employees sell Products to Customers. The Sales organization is divided into Divisions. Each Sale is a business event and, therefore, can be modelled as a “Sales” Fact. The diagram below depicts the Sales Fact and its related Dimensions. The Dimensions provide the Who (Customer, Employee), the where (Division) and the when (Calendar, Time of Day) context for the Sales Fact.

Slowly Changing Dimension Type 1 – Overwrite

The Slowly Changing Dimension (SCD) Type 1 change-handling strategy is used when it doesn’t make sense to keep an attribute’s history or when users aren’t interested in it.

When an attribute’s value changes, the SCD Type 1 code simply overwrites the existing value with the new value. Essentially, it records no history of the change.

Our example has an Employee Dimension associated with a Sales fact. In the Employee Dimension, an Employee Surname is a good example of a Type 1 SCD Attribute. If an Employee’s Surname changes due to marriage, you want to overwrite the Surname with the new value.

Since there is only one version of each Employee in the Dimension, all historical Sales Fact records remain associated with the single Employee record, and all future Sales Fact records are associated with the same Employee record.

Now imagine a report that shows Sales by Employee Name. All Sales are reported against the Employees new name. A desirable outcome.

If you kept multiple versions of the Employee (e.g., the SCD Type 2 strategy) and looked at the Sales by Employee Name report, the Employee would show up twice, once with the new name and once with the old name. This is an undesirable outcome.

SCD Type 1 is the most used strategy for handling changes, often serving as developers’ go-to method by default (for better or worse). It’s also the easiest to implement. Hmmm…

Slowly Changing Dimension Type 2 – Add a new Version

You use the Slowly Changing Dimension Type 2 strategy to ensure that reports present a historically accurate data view. The SCD Type 2 change strategy is designed to keep a history of changes to Type 2 attributes. When a Type 2 attribute value changes, the ETL code pattern I describe later creates a new row. The new row represents a new version of the given Dimension Member. Therefore, there can be more than 1 row for each Dimension Member. In order for reports to present history accurately, The Dimension load pattern must work in tandem with associated Fact table‘s load Pattern. I discuss that further later.

Take the SCD Type 2 example below, which shows a Sales Division Dimension. The example treats the Region attribute as a Type 2 attribute. In the example, The Sales dept decided to move the “Richmond” Division from the “North East” Region to the “East Coast” Region on September the 1st, 2024. As the Region attribute is an SCD Type 2 attribute, this change will result in the creation of a new version row for the “Richmond” Division. After September the 1st, 2024, there will be 2 rows in the Dimension for the same “Richmond” Division Dimension Member, one with the old Region and one with the new Region.

Note that the system has assigned the new record a new surrogate key (45). Also, note that the system has tagged the new record as the “latest” version and set the “Row Effective Date” to the date of the change. The system has also set the “Row End Date” to the date of the change for the superseded version row ( division skey = 1). The How to Create a Slowly Changing Dimension section provides a detailed discussion of this pattern.

As I mentioned earlier the SCD Type 2 pattern needs to work in tandem with the Fact load pattern to accurately present history. To determine which version of the Dimension member to associate with any given Fact, the Fact table load pattern should use a combination of the business key and effective dates. Generally, a Fact will contain some transaction date. The Fact finds the correct Dimension Member version using the business key, where the Fact transaction date is between the Dimensions Effective and End dates.

Using this logic, all existing Sales Facts prior to September 1st, 2024, remain associated with the original version of the Richmond Division, and all Sales Facts after that date are associated with the new version. If you were to look at a Sales by Region report, Sales prior to September 1st, 2024 appear in the “North Coast” Region, and Sales after in the “East Coast” Region.

Slowly Changing Dimension Type 3 – Latest and Prior Version

With the Slowly Changing Dimension Type 3 change-handling strategy, you only capture the latest and prior values instead of capturing every version of the attribute value, as you do with SCD Type 2. There are some rare circumstances where this is desirable. Take the previous Type 2 SCD example of a Division shifting Regions. What if the business wanted to see today’s Sales attributed to the new Region but also “as if” they had remained in the old Region, just to compare the performance of the old Region to the new one? The SCD Type 2 strategy does not accommodate this kind of request as it never associates new sales with the old Region. A pattern that associates a Fact with both the Old and New Region is necessary, thus the need for the SCD Type 3 pattern.

SCD type 3 satisfies this comparative kind of reporting requirement. In the SCD Type 3 strategy, you don’t create a new row for a new version of the Dimension member. Instead, the Dimension contains two columns. One for the current value and one for the prior value of the same attribute.

For example:

Revisiting our Richmond Division example, the Richmond Division was in the “North” East Region before the change. The Dimension only contained a single column for the Region attribute:

To accommodate the Type 3 change, you add a “prior Region” column to the table. When the Richmond Division moves to the ‘East Coast’ Region, the system transfers the ‘North East’ value to the ‘prior Region’ column and records ‘East Coast’ as the new value in the Region column. See the result below.

Now, the single version of the Dimension member is associated with all Sales Facts, both new and old. Analysts have the option to select the current Region, Prior Region, or both in their reports. This selection enables them to compare sales between the current and prior Regional organizations.

Note that we don’t recommend implementing the Type 3 ETL as a separate pattern. Instead, we recommend emulating the Type 3 pattern. Read more about our recommended approach to Slowly Changing Dimensions and implementing SCD Type 3.

Slowly Changing Dimension (SCD) Type 4 – Add a history table

Slowly Changing Dimension Type 4 is similar to SCD Type 2, except it stores the historical versions of Dimension Members in a separate table.

The Dimension is divided into two separate current and history tables. The current table contains only the latest version of each Dimension Member, while the history table contains both the historical and current versions. Both tables share the same surrogate keys.

SCD Type 4 proves useful when users want to analyze one Fact table that accurately reflects history and another Fact table where history is irrelevant. In such a case, the historical Dimension table is associated with the ‘historical’ Fact table, and the ‘current’ Fact table uses the current Dimension table.

SCD Type 4 can also be used to report both a historical and current perspective on the same Fact. In this instance, the Fact is joined to both the Historical and Current Dimensions with 2 separate foreign keys. The outcome is similar to the SCD Type 6 approach below, except there is both a Historical and Current Dimension instead of one Dimension with Historical and Current Attributes.

I don’t recommend Type 4. The ETL to manage 2 tables is more difficult, and there are issues in reload scenarios, etc. Instead, I recommend adding a ‘latest’ management column to your Type 2 dimensions and using that column when loading Fact tables to associate them with either the historical or current version. See the Implementing SCD Type 4 below.

Slowly Changing Dimension (SCD) Type 6 Hybrid – Latest vs Historical Versions

The Slowly Changing Dimension Type 6 approach is useful for enabling accurate history reporting while also supporting the ability to report historical data “as if” it is attributed to the “current” or “prior” versions of history.

Type 6 combines Types 1, 2 and 3 (1+2+3 = 6). It combines the historical versioning described in Type 2 SCD with the Current vs Prior version described in Type 3. To achieve the Type 6 pattern, you add a Type 3 “prior” and Type 1 “current” attribute to the Dimension. The original attribute contains the Type 2 version of history, and the system updates the “current” attribute using a Type 1 SCD change strategy and updates the “prior” attribute using a Type 3 change-handling strategy.

For example, the following images show the Richmond Division, originally belonging to the “Northeast” Region. When the Division moves to the “Mid Coast”, the system creates a new version of the Richmond Division Dimension Member. The original “Region” attribute tracks the Type 2 history of the Region attribute, while the “Current Region” tracks the “Type 1” latest value of the Region attribute. The “Prior Region ” attribute tracks the Type 3 prior value of the Region attribute for each version. Again, the process repeats when the Division moves to the “South Coast” region.

You can use the “Region” attribute to report on sales based on the accurate history of the Division’s Region. The “Current Region” attribute allows you to report on sales based on the latest Region, while the “Prior Region” attribute lets you report on sales based on the prior Regional organization.

Type 6 is powerful and provides analytical options, but users rarely employ it, mainly due to the complexity of data and schema management. Imagine having a Current and Prior attribute for every attribute. You should apply Type 6 selectively, although determining which attributes the business wants to analyze in this way can be challenging up-front. Again, our recommended approach suggests emulating Type 6 slowly changing dimensions.

How to Create a Slowly Changing Dimension

Slowly Changing Dimensions Recommended Approach

Our guiding principles are:

1. KISS (Keep it Simple Stupid).

2. Less is More. Implement as few ETL patterns as possible.

3. Be consistent. Don’t do “special cases” or skip implementation details for “simple” Dimensions.

Our recommended approach is as follows:

1. Only implement Type 1 and Type 2 SCD in your Dimensions.

2. Stick consistently to a single pattern for all your Dimensions, regardless of whether they contain only Type 1 attributes, only Type 2 attributes, or both. Consistency makes your code simpler to maintain.

3. Less is more. SCD Type 3, 4, and 6 are so rare that it’s not worth maintaining a different ETL code pattern to load them. Instead, in the rare cases where they are required, you emulate these patterns. For more detail, see the sections on creating Types 3, 4, and 6.

Slowly Changing Dimension Example

I’ve covered the basics of slowly changing dimensions. To get a better understanding of how to implement the ETL associated with each SCD Type, I’ll use the following example:

The Star Schema below depicts a typical Sales scenario. Employees sell Products to Customers. The Sales organization is divided into Divisions. Each Sale is a business event and, therefore, can be modelled as a “Sales” Fact. The diagram below depicts the Sales Fact and its related Dimensions. The Dimensions provide the who (Customer, Employee), the where (Division) and the when (Calendar, Time of Day) context for the Sales Fact.

Our example will focus on the Customer Dimension. The following sections outline the modelling and ETL patterns you need to create and manage this Dimension.

A Universal Data Model for Dimensions

As discussed previously, you should only implement Type 1 and 2 Slowly Changing Dimension change strategies in your Dimensions (and emulate Type 3,4 and 6). Sticking to our principles, we want to define a consistent data model for all our Dimensions regardless of whether they contain only Type 1 attributes, only Type 2 attributes, or both. You never know when a requirement might emerge to change an attribute from Type 1 to Type 2 or vice versa.

As discussed in “Introduction to Dimensions“, I showed the following example of the Customer Dimension:

The “Introduction to Dimensions” article describes the various types of columns in a Dimension. For the purposes of this exercise, we are mostly interested in the “row management columns.” The Dimension ETL patterns use these columns to manage appropriate Inserts and Updates, speed up ETL execution, and facilitate change logging and auditing.

The Row Management Columns are as follows:

• row_is_latest. Indicates the row version is the latest row version, 0 = row is a superceded version.

• row_is_deleted. Indicates that the source system has deleted a row with the same business key.

• row_effective_date. The start date and time (to the microsecond) that the row version becomes effective. For the first row version, row_effective_date is “low date” (i.e. 0001-01-01…). Fact table ETL patterns require Dimensions to have effective dates to associate Facts with the correct version of Dimension Members accurately.

• row_end_date. The end date and time (to the microsecond) that the row version becomes no longer effective. This is always the effective date of the next row version in the series of row versions. Therefore, row versions’ effective date and end dates are contiguous. For the last row version, row_end_date is “high date” (9999-12-31….). The BETWEEN SQL clause can’t be used to join across row effective and end dates. Instead, use a clause similar to this: X >= Row_Effective_Date and X < Row_End_Date, where X is the transaction date of the Fact.

• row_inserted_batch_id. The system creates a new batch ID for every batch execution and sets the row_inserted_batch_id to the ID of the batch that inserted the row.

• row_updated_batch_id. The system sets the row_upaated_batch_id to the last batch Id that updated the row.

• row_scd_type_1_hash. A hash of all SCD Type 1 attribute values. Used by the ETL to compare incoming rows to detect any Type 1 changes.

• row_scd_type_2_hash. A hash of all SCD Type 2 attribute values. Used by the ETL to compare incoming rows to detect any Type 2 changes.

• row_supercedes_skey Optional. Records the surrogate key of the row that this row has superseded due to a Type 2 change. Used to speed up ETL execution and provides a convenient method of viewing and auditing the history for any given Dimension Member. Since Dimensions are generally relatively small, there is no harm in including this column.

• row_last_tran_code Optional. A useful indicator that helps debugging and audit exercises. Since Dimensions are generally relatively small, there is no harm in including this column.

This seems like a lot of columns, but each has its purpose. Dimensions are generally small, so the increase in storage is negligible. I, therefore, would argue to keep all columns. However, there are options to reduce the number of columns:

• The row_supercedes_skey and row_last_tran_code are optional. Depending on how you manage ETL runs, the batch_id columns may not be needed.

• The row_is_latest and row_is_deleted are bit data types, meaning they occupy one bit of a shared byte on the record. This leaves room for 6 additional bit fields that don’t occupy any additional space. You could potentially use 2 bits for the row_last_tran_code, but all you save is one byte per row.

• The largest columns are row_scd_type_1_hash and row_scd_type_2_hash, at 20 bytes each. You could recalculate them each time the ETL runs instead of storing them. However, this means additional coding and processing overhead. I opt to store them to make the ETL simpler and faster.

The Type 1 and 2 SCD Dimension Code Pattern

The following code implements the Type 1 and 2 SCD ETL code pattern for the example Customer Dimension. The Customer Dimension contains a mix of SCD Type 1 and 2 attributes:

• The Type 1 attributes include name, language and employer.

• The Type 2 attributes include language, type, income_level, marital_status and credit_profile.

The SQL code, originally written for a PostgreSQL database, can be easily modified to run on various platforms such as Snowflake, Databricks, Microsoft Fabric, Google Big Query, or AWS Redshift. The great thing about SQL is that it is ubiquitous across platforms, although each platform has slightly different SQL dialects.

SQL uses a series of chained common table expressions (CTE). CTEs are temporary named result data sets that only exist during the execution of a single query. They act like virtual tables that can be referenced later in other CTEs or the main query.

The advantages of this approach are:

1. CTEs (Common Table Expressions) generate temporary results, such as the delta result set below, which the system evaluates only once. However, these results can serve multiple subsequent queries, thereby enhancing performance.

2. PostgreSQL (and some other SQL dialects) can include INSERT, UPDATE and DELETE statements in CTEs.

3. PostgreSQL also has a feature where INSERTs, UPDATEs, and DELETEs can output a result set of updated rows. This can improve performance, as the result of one update may serve as the input for another, such as the type_2_supercede CTE shown in the following code.

4. You can use this single, self-contained script as a pattern for all your Dimensions.

Below is the code consisting of the following 5 chained CTEs and a final query:

1. Source CTE. Composes the source data.

2. Delta CTE. Determines the delta change set.

3. Inserts CTE. Insert the New and Type 2 change records. Returns a list of surrogate keys of superceded type 2 rows.

4. Type_2_supercede CTE. This CTE updates rows superseded by new Type 2 versions.

5. Type_1_update CTE. Applies Type 1 changes.

6. Soft delete final query. A “soft” delete of any rows no longer present in the source.

The SQL is explained further following the script.

/*************************************************************************
DIMODELO.COM

Type 1 and 2 Dimension ETL pattern

**************************************************************************/

CREATE OR REPLACE PROCEDURE dim_customer_type_1_and_2_update(batch integer)
LANGUAGE SQL
AS $$

-- compose the "source"
WITH "source" AS (
    SELECT customer_id, "name", "language", "type", employer, income_level, marital_status, credit_profile,
    digest("type" || ',' || income_level || ',' || marital_status || ',' || credit_profile ,'sha1') as row_scd_type_2_hash,
    digest(name || ',' || language || ',' || employer ,'sha1') as row_scd_type_1_hash
    FROM stg.customers
),

-- derive delta change set
delta AS (
    SELECT "source".*, dim.skey as dim_skey,
    CASE    WHEN (dim.customer_id = "source".customer_id) AND dim.row_scd_type_2_hash != "source".row_scd_type_2_hash THEN '2' -- type 2 change
            WHEN (dim.customer_id = "source".customer_id) AND dim.row_scd_type_1_hash !=  "source".row_scd_type_1_hash THEN '1' -- type 1 change
            WHEN dim.customer_id is NULL THEN 'N' --new
            WHEN "source".customer_id is NULL THEN 'D' --deleted
            ELSE 'unknown' END AS tran_type
    FROM "source"
    FULL OUTER JOIN whs.dim_customer dim
    ON dim.customer_id = "source".customer_id AND dim.row_is_latest = CAST(1 AS BIT)
    WHERE (dim.customer_id = "source".customer_id) AND dim.row_scd_type_2_hash !=  "source".row_scd_type_2_hash -- type 2 changes condition
    OR (dim.customer_id = "source".customer_id) AND dim.row_scd_type_1_hash !=  "source".row_scd_type_1_hash -- type 1 changes condition
    OR dim.customer_id is NULL -- new records condition
    OR (dim.row_is_latest = CAST(1 AS BIT) AND "source".customer_id is NULL) -- deleted records condition
),

-- insert new and type 2 records
inserts as (
    INSERT INTO whs.dim_customer  (customer_id, "name", "language", "type", employer, income_level, marital_status, 
          credit_profile, row_is_latest, row_is_deleted, row_effective_date, row_end_date,
          row_inserted_batch_id, row_updated_batch_id, row_scd_type_1_hash, row_scd_type_2_hash,row_supercedes_skey,
          row_last_tran_code)
    SELECT  customer_id, "name", "language", "type", employer, income_level, marital_status,
            credit_profile, CAST(1 AS bit), CAST(0 AS BIT), 
            CASE WHEN dim_skey IS NOT NULL THEN CURRENT_TIMESTAMP(6) ELSE '0001-01-01' END,
            '31-12-9999',
            batch, NULL, row_scd_type_1_hash,row_scd_type_2_hash, dim_skey,
            tran_type
    FROM delta
    WHERE delta.tran_type in ('2', 'N')
    RETURNING row_supercedes_skey
),

-- update superceded records (by type 2 changes)
type_2_supercede as (
UPDATE whs.dim_customer AS dim
SET row_end_date = CURRENT_TIMESTAMP(6),
    row_is_latest = CAST(0 AS BIT),
    row_updated_batch_id = batch,
    row_last_tran_code = 'S'
FROM inserts
WHERE dim.skey = inserts.row_supercedes_skey
),

-- update records with type 1 changes
type_1_update as (
UPDATE whs.dim_customer AS dim
SET     "name" = delta."name",
        "language" = delta."language",
        employer = delta.employer,
        row_updated_batch_id = batch, 
        row_scd_type_1_hash = delta.row_scd_type_1_hash,
        row_last_tran_code = tran_type
FROM delta
WHERE delta.tran_type = '1'
AND dim.customer_id = delta.customer_id
)

-- finally soft deletes
UPDATE whs.dim_customer AS dim
SET     row_updated_batch_id = batch,
        row_end_date = CURRENT_TIMESTAMP(6),
        row_is_deleted = CAST(1 AS bit),
        row_last_tran_code = tran_type
FROM delta
WHERE dim.skey = delta.dim_skey
AND delta.tran_type = 'D'
;

$$;

Type 1 and 2 ETL Pattern Explained

The following diagram and dot points break down and describe the code in detail:

• Staging Sources. This pattern assumes that we have loaded the source data into accessible staging tables, following the standard data warehouse architecture. The staging source is either a periodic full copy of source tables/data or a persistent copy of staging data that replicates the source.

• Compose Source (source CTE): The Data source of a Dimension could be a complex join across a set of source staging tables. This step composes the source from its source staging tables, selects values for all the Dimension attributes, and calculates the row_scd_type_1_hash and row_scd_type_2_hash Hashes for the source rows. The code concatenates and hashes Type 1 identified columns into the row_scd_type_1_hash, and similarly, it concatenates and hashes Type 2 identified columns into the row_scd_type_2_hash. The Hashes use the SHA-1 (SHA1 – Wikipedia) hashing algorithm. The SHA-1 algorithm remains suitable for “comparison” purposes, although it is no longer deemed secure for encryption. It generates a relatively compact hash value that is adequately free of collisions and enjoys broad support across databases. MD5 is not advised due to its high rate of collisions, while SHA-256 and higher algorithms yield hash values that are 50% larger.

• “source”. The temporary result of the “Compose Source” step.

• Derive delta change set. In this step, the process derives a data set of the records that have changed in the source when compared to the Dimension.

  • First, the source is joined to the Dimension on the business key of the Dimension (in this case, customer_id)using a FULL OUTER JOIN. A Full Outer Join is necessary to identify deletes. FULL OUTER JOINs can be expensive, but Dimensions are generally small, so performance is rarely an issue. If a full outer join wasn’t used, you need a second separate query. Weighing these two facts, it’s better to do a single Full Outer Join.

  • The WHERE clause selects rows where a matching row (based on the business key) exists in both the source and Dimension, and the Source and Dimension Hashes are different, indicating a Type 1 or 2 change. It also identifies new records (i.e., in the Source but not in the Dimension) and deleted records (i.e., in the Dimension but not in the Source).

  • The query derives a tran_type column to indicate the detected change type. The query sets the tran_type to “1” for a Type 1 change, to “2” for a Type 2 change, to “N” for a new record, or to “D” if the record is deleted.

  • The query SELECTs everything in the source table plus the surrogate key of the matching Dimension row (if there is one). The “Supercede Old Type 2 Records” step uses the surrogate key later.

• “delta”. The “Derive Changes” step outputs the temporary delta change data set. Subsequent Update statements use the “delta” data set multiple times. Creating a “delta” dataset has the advantage of a one-time initialization, repeated utilization, and operation with a significantly smaller dataset. This smaller dataset can be easily filtered by transaction type in subsequent updates, thus improving performance.

• Insert New and Type 2 Records. The “Insert New Records” inserts new Dimension Members that don’t already exist, along with new versions of any Dimension Members subject to an SCD Type 2 change. I.e. delta record with tran_type equal to 2 or N. The RETURNING clause returns a set of rows that contain the surrogate keys of the rows that were superseded by new Insertions.

• Updated Superceded Records. Updates the previous “latest” version of the Dimension Member, which the new version has superseded, to set its row_effective_end_date and row_is_latest flag. The previous “Insert New Record” step outputs a list of surrogate keys of superseded rows. This Update statement updates every row in that list, setting its row_effective_end_date and row_is_latest flag.

• Update Records with Type 1 Changes. The Type 1 Update statement updates any record when it identifies a type 1 change (tran_type = 1). It updates all records in the Dimension that match any business key of rows in “delta” where a tran_type =1. You update all versions of the matching Dimension Member (based on the business key).

• Soft Deletes. Finally, the query “soft” deletes any records in the Dimension corresponding to deleted rows in the source. A soft delete sets the row_is_deleted flag and row_end_date of the Dimension row rather than deleting the row from the Dimension. You only “soft” delete data because historical Facts in Fact tables may still reference the deleted Dimension member.

Creating a SCD Type 3 Dimension

As discussed, the SCD Type 3 change-handling strategy only captures the latest and prior value of an attribute in separate columns.

In our recommended approach we suggested only implementing Type 1 and 2 SCD ETL patterns and to emulate SCD Type 3. Developers use the Type 3 Slowly Changing Dimension (SCD) pattern very rarely, and investing in a specific pattern is not worthwhile when simple emulation is possible. To emulate Type 3 SCD, first the attribute must be a Type 2 attribute. If it’s not, then change it, then create a view similar to this example:

In our example we want to show the latest and prior values of the credit_profile attribute of the Customer Dimension. Use the SQL lag windows function. The query is as simple as this:

CREATE OR REPLACE VIEW whs.dim_customer_type_3
AS
SELECT * FROM
(
    SELECT *,
    LAG(credit_profile,1,NULL) OVER(PARTITION BY customer_id ORDER BY skey) 
    AS prior_credit_profile
FROM whs.dim_customer) S
WHERE row_is_latest = CAST(1 AS BIT);

The output looks like this:

Implementing a Type 4 SCD change-handing strategy

SCD Type 4 is similar to SCD Type 2, except you keep the historical versions of Dimension Members in a separate table. You divide the Dimension into a current table and a history table.

In our recommended approach, we suggested only implementing Type 1 and 2 SCD and instead emulating SCD Type 4. Developers use the Type 4 Slowly Changing Dimension (SCD) pattern very rarely, and investing in a specific pattern is not worthwhile when simple emulation is possible. In fact, we recommend never using Type 4. However, to emulate Type 4 SCD, create a view representing the “current” table, similar to this example:

CREATE OR REPLACE VIEW whs.dim_customer_current
AS
SELECT *
FROM whs.dim_customer
WHERE row_is_latest = CAST(1 AS BIT);

The existing table represents the historical table.

Creating a SCD Type 6 Dimension

The Type 6 approach is beneficial for reporting historical data accurately and simultaneously allowing the data to be presented as if it were attributed to the current version of history.

Type 6 is a combination of types 1, 2, and 3 (1+2+3 = 6). It combines the historical versioning described in Type 2 SCD with the Current vs. Prior version described in Type 3.

In our recommended approach, we suggested only implementing Type 1 and 2 SCD and just emulating SCD Type 6. Developers use the Type 6 Slowly Changing Dimension (SCD) pattern very rarely, and investing in a specific pattern is not worthwhile when simple emulation is possible. To emulate Type 6 SCD, first, the attribute must be a Type 2 attribute. If it’s not, then change it, then create a view similar to this example:

In our example, we want to show the current, prior and historical values of the credit_profile attribute of the Customer Dimension. You can achieve this with an SQL LAG Window function and a JOIN to the “latest” version of each Dimension Member. The query is as simple as this:

CREATE OR REPLACE VIEW whs.dim_customer_type_6
AS
SELECT prior.*, cur.current_credit_profile,
LAG(credit_profile,1,NULL) OVER(PARTITION BY customer_id ORDER BY skey) 
AS prior_credit_profile
FROM whs.dim_customer "prior"
JOIN (
    SELECT customer_id AS current_customer_id, 
    credit_profile AS current_credit_profile
    FROM whs.dim_customer
    WHERE row_is_latest = CAST(1 AS BIT)    
) cur
ON prior.customer_id = cur.current_customer_id

The output looks like this:

What is a Dimension Table?

Conceptually, Dimension tables are database tables modelled to represent the business entities involved in business processes and events. The business processes are, in turn, modelled as Fact tables, and the Fact and Dimension tables are related in a “Star Schema” to provide an easy-to-navigate and query Data Warehouse database.

Where Fact tables represent the “numbers” you want to analyze, Dimension Tables provide the context by which you analyze those numbers. The Who, What, Where, When and Why.

For example, imagine a typical Sales scenario. Employees sell Products to Customers. The Sales organization is divided into Divisions. Each Sale is a business event and, therefore, can be modelled as a “Sales” Fact. The diagram below depicts the Sales Fact and its related Dimensions. The Dimensions provide the Who (Customer, Employee), the where (Division) and the when (Calendar, Time of Day) context for the Sales Fact.

Dimension tables physically contain the descriptive attributes of the business entities. For example, a Customer Dimension would contain attributes like Customer Name, Age, and Gender. The attributes filter and group the Fact table data in reports and analysis. See the next section for more details.

How to Design a Dimension Table

A Dimension contains Members. There is a Member for every business/natural key of the Dimension. For example, in a Customer dimension, there is one member for each individual Customer. However, because a Dimension may keep the history of change for each member, there may be multiple rows in the Dimension table for each member, one for each version. To handle this, Kimball introduced the concept of a surrogate key.

Dimension Surrogate Key vs Business Key

Because a Dimension can keep multiple rows for different versions of any Member, the business key of a Member can be duplicated over multiple rows. Therefore, the business key can’t be used as the unique primary key. Instead, each row is assigned a unique surrogate key by the data warehouse. The surrogate key becomes the primary/unique key of the Dimension. Usually, a surrogate key is a sequentially assigned integer. In recent years generating a hash on the business key and effective date combination has become popular. There are cases where you may want to use a “smart” surrogate key. I.e. a code for the surrogate key. A typical example is the Calendar Dimension, with a smart key like “YYYYMMDD” used as the surrogate key for each row (representing a day) in the Calendar. Read more about Smart Keys.

Note that a surrogate key is only strictly needed for Dimension Tables, which contain type 2 (version history) attributes. However, I recommend assigning a surrogate key to every Dimension for uniformity and future-proofing.

Dimension Table Attributes

Attributes are the columns of Dimensions that contain descriptive information for each Member. E.g. Customer Name, Customer Income Level, Marital Status etc.

It’s important to understand that attribute values (not the attribute column names) become row and column headers in pivot tables/charts. For example, a pivot table showing sales by gender in the past 12 months would look something like this:

Where “female” and “male” are the values stored in the Gender attribute column of the Customer Dimension.

A mistake I often see is developers using codes or keys for values in attributes. So, for example, in the database, “male” and “female” are represented as the codes M and F. If the M and F codes are carried through to the Dimension Table, then the pivot table report looks like this:

Which is far less useful. Users looking at this report may have no idea what M and F stand for. The codes M and F should be converted to their meaningful descriptive values.

Some rules for Dimension attributes:

• Operational codes (e.g. M for Male, F for Female) should be converted in Dimensions to their descriptive format (i.e. Male/Female). 

• True/False, Yes/No values should be converted to descriptions like “Is Contractor” or “Is Not Contractor.”

Some red flags to watch out for:

• Dimension attributes should never be dates. Leave that to slowly changing dimension techniques, Calendar dimensions and various fact types.

• Numerical values, especially contiguous values, should not be used as Attributes. See the next section.

Discrete vs Contiguous Dimension Attribute Values

Dimension Table attributes should be discrete values, not contiguous values. What does this mean? Discrete values are drawn from a small set of distinct values, whereas continuous values can be any value (up to an infinite number) within a generally large range of possible values. A common example of a contiguous value that is often mistakenly used as a Dimension attribute is Product Price. For any given range of products a company sells, the product price could be any price between a range of say $0.01 and $10000.00. Its unlikely that any two products have the same price, which makes using price as a row or column header in a report not very useful. You will get a column or row for every price. See the example below. There is very little analytical value to a report like this:

To deal with this problem, a developer can turn the contiguous values into discrete values by defining groups of contiguous values. For example, a business might be interested in its sales of low-priced (<=$99), medium-priced (>$99 to <$200), and high-priced (>=$200) Products. By defining an attribute with values for each price range, the sales amount for each range can be aggregated and displayed. This is far more useful. See the image below:

Row Management Columns

A Dimension will also contain a set of “management” columns. Management columns are generally used by the ETL (extract, transform, load) process to help load data into the Dimension Table and to set properties of each row like latest flag, effective dates, etc.

Slowly Changing Dimensions (SCD)

Dimensions are often referred to as ‘Slowly Changing Dimensions’. The term recognizes that Dimension Members are relatively static but change (albeit slowly) over time.

This is important because users are often interested in the impact to their reports of these changes to Dimensions over time. It’s necessary to define a number of change-handling strategies for Dimension Table attributes. Ralph Kimball defined these change-handling strategies as Slowly changing Dimension (SCD) types 1, 2, 3 and 4:

• SCD Type 1 – Don’t keep history, i.e. overwrite.

• SCD Type 2 – Keep a version of the Dimension Member for every change.

• SCD Type 3 – Keep the latest version and the prior version (but not every version).

• SCD Type 4 – A hybrid combination of Type 2 and 3.

The most commonly used SCD strategies are Types 1 and 2. SCD Types 3 and 4 are rarely used.

An important nuance to note is that, although we talk about Slowly Changing “Dimensions”, what we are actually implementing is Slowly Changing Attributes. You implement the change-handling strategy per Attribute. That means different attributes within one Dimension can have different “slowly changing Dimension” types. There are various techniques to achieve this. The short answer is to use one hash management column for columns of type 1 and a second hash for columns of type 2.

Read a more in-depth description of Slowly Changing Dimensions, including example code you can download and test for yourself.

Slowly Changing Dimension Type 1 – Overwrite

The slowly changing Dimension (SCD) type 1 change-handling strategy is applied; if a change occurs to an attribute, the existing attribute value is overwritten with the new value. Essentially, no history of the change is recorded.

A good candidate for an SCD Type 1 Attribute is something like Employee Name. If an employee’s name changes due to marriage, you don’t want two versions of the employee, one before marriage and one after. If you did and were looking at sales by employee, the employee would show up twice, once with the new name and once with the old. Instead, you want to overwrite the old name with the new name. In this scenario, the sales report by Employee would only show the Employee with the new name. Effectively all historical sales for that employee are now associated with the new name.

This highlights unanticipated consequences with SCD Type 1, and why you might want to adopt a different strategy (i.e. SCD Type 2). Let’s say you have a Sales Division Dimension, and each Sales Division has a parent Region. If the Sales Division is moved to a new Region, and the Region attribute of the Sales Division is treated as SCD Type 1, and overwritten with the new Region, then all the historical sales facts that were associated with the old Region would suddenly appear in Regional reports as if they had occurred in the new Region. Usually, an undesirable outcome. This is where SCD Type 2 comes into play.

Slowly Changing Dimension Type 2 – Add a new Version

In the SCD type 2 scenario, a new version of the Dimension member row is written when an SCD Type 2 Attribute changes. History is preserved. Existing Facts remain associated with the old version of the Dimension Member, and new Fact data is associated with the new version of the Dimension Member. For Example, let’s say you have a Sales Division Dimension, and each Sales Division has a parent Region attribute. A decision is made to move the Sales Division into a new Region. If the Region attribute of the Sales Division is treated as SCD type 2, then a new row is written to the Dimension Table for the new version of the Sales Division Dimension Member. There are now 2 rows in the Dimension for the same Division, one with the old Region, and one with the new Region.

All the existing Sales facts associated with the old version remain associated with that version and would still appear in Regional reports as if they belong to the old Region. This is a desirable outcome because this region was responsible for each Sale at the time it was made. Only new Sales that occur after the change are associated with the Sales Division with the new Region, thus preserving history.

Slowly Changing Dimension Type 3 – Latest and Prior Version

SCD Type 3 change-handling strategy captures the latest and prior value of a Dimension attribute. It doesn’t capture every version. There are some rare circumstances where this is desirable. Take the previous example of a Division moving Regions. What if the business wanted to see today’s sales “as if” they were in the old region just to compare how they would have performed under the old sales organization? The SCD Type 2 strategy doesn’t accommodate this kind of request. This is because new sales are never associated with the old region. For a period of time, they may want to track sales in terms of the old regional divide vs the new regional divide.

Type three SCD satisfies this kind of reporting requirement. In the SCD Type 3 strategy, you don’t create a new row for the version of the Dimension member. Instead, a new column is added to capture the prior value.

For example:

Before the change the Richmond Division is in the “North” East Region:

A “prior region” column is added to the table to accommodate the SCD type 3 change, and then the Richmond Division is moved to the “East Coast” Region. The result is as follows:

Now all Sales Facts, new and old, are associated with the single version of the Dimension member, but analysts can choose either the current Region or Prior Region (or both) in their reports to compare current to prior Regional organization sales.

Dimension Table Denormalization vs Normalization

Dimensions often represent hierarchical relationships in the business. For example, take a Division->Region->Country hierarchy. In an operational system, these would normally be modelled as separate (normalized) tables with relationships between them. However, Dimensions tend to be denormalized. That is, instead of separate Dimension tables, we would have a single Division Dimension. The Division Dimension has a Region and Country attribute to model the relationship between a Division and its Region/Country. The hierarchical descriptive information is stored redundantly, but this design results in improved ease of use and query performance. Dimensions tend to have relatively few rows compared to Fact tables and a large number of columns. The trade-off with storage space is insignificant.

The exception would be if there were Facts in the Data Warehouse that had a grain at the Region or Country levels; then it’s necessary to have separate Dimensions for those levels so those Facts can be attached at that level.

Snowflake Dimensions

Snowflake dimensions describe hierarchical relationships in a Dimension table that have been “normalized.” Instead of a single Dimension Table with attributes representing relationships, secondary Dimension tables are created and connected to a base Dimension by an attribute key.

My advice (and Ralph Kimball’s) is to avoid them. They simply make the Data Warehouse harder to navigate, and nothing can be modelled in a Snowflake schema that can’t be modelled in a Star Schema.

Read Snowflake vs Star Schema for more information

The Two Types of Dimension Hierarchies

Natural Hierarchies

Many dimensions contain natural relationships between attributes that form hierarchies. In our previous example, in the Division Dimension, there is a hierarchy from Country->Region->Division. This is known as a “natural” hierarchy because it is a relationship between attributes with a fixed depth or number of levels.

Hierarchies can be very useful for analysis, allowing drill-down and drill-up analysis in BI tools. BI tools can be configured to show these hierarchies as nested attributes making it easy to visualize and navigate the hierarchy.

It’s common for Dimensions to have multiple hierarchies. For example, a Calendar Dimension can have a Day>Month>Year… a Day>Month>Fiscal Year… a Day>Month>Quarter>Year and a Day>Fortnight>Year hierarchies. All are valid Hierarchies, and all may be used by different people for different purposes.

Some Natural Hierarchy rules:

• Hierarchies in a Dimension always end with the root Attribute at the Grain of the Dimension. E.g. In the Calendar Dimension, the Day Attribute.

• It’s important that each attribute value belongs to only one of its parent Attribute values. Otherwise, you have a broken Hierarchy.

Parent-Child Hierarchies

A Parent-Child Hierarchy is a hierarchy of Dimension members where child members “point to” their parent members. Parent-child hierarchies are also known as Ragged or variable-depth Hierarchies because each “branch” of the Parent-Child “tree” can have a different number of levels.

Employees or Positions in an Organisation Chart are good examples of parent-child hierarchies. Each object points to its parent object, i.e., each Position in an Organisation Chart points to its parent Position.

These kinds of relationships are known as “recursive” relationships. Recursive relationships have traditionally been difficult for SQL and BI tools to navigate. Thankfully, today, most SQL dialects and good BI tools can natively query these relationships or emulate natural hierarchies based on the parent-child hierarchy.

Some Parent-Child Hierarchy rules:

• A Parent can have multiple children, but a child only has one parent.

• The child contains an attribute that contains the identifier of its parent.

The Importance of Conformed Dimensions

The Kimball method of Data Warehouse design describes a Data Warehouse Bus Matrix Architecture and the concept of “conformed” dimensions. Put simply, the concept of “conformed” Dimensions states that Facts should share common or “conformed” Dimensions to enable cross Datamart/Enterprise/Business process analysis.

The aim is to design your data warehouse so that Facts share as many Dimensions as possible. A data warehouse bus matrix is a good way to document the design of your Data Warehouse and maximize conformed Dimensions.

A Data Warehouse matrix documents the relationship between all planned Facts and Dimension Tables. By completing a matrix, you can see the overall high-level design of your Data Warehouse on a single sheet. In the image below, where there is a “1” at the intersection between Facts (the rows in blue) and Dimensions (the columns in green), there is a relationship between the Fact and the Dimension.

Conformed Dimensions are powerful. They allow cross-fact analysis. If two Facts share a Dimension, then, when a member is selected in that Dimension Table in a BI tool, it filters both Facts. For example, let’s say you have sales and accounting fact tables that are connected to the same calendar dimensions. You can select a Calendar period (e.g. Month), and select measures from both Fact tables (e.g. Sales Qty, and Revenue) in the same pivot table and they are both filtered by the Calendar Period member. See below:

This also allows you to define calculated measures that use measures from more than one Fact table. For example, the “Avg Sale Amount” above which is Revenue/Sales Qty.

Role Play and Junk Dimensions

Role Play Dimension

A Fact can be associated with a Dimension in a different role more than once. These secondary associations are known as Role Play Dimensions. For example, a Task fact can be associated with the Calendar Dimension a number of times: once for a Scheduled date, once for a Started date, once for a Completed date and so on. Each of these would appear as separate Dimensions, i.e. the Started Calendar Dimension, the Scheduled Calendar Dimension and the Completed Calendar Dimension. This structure allows you to use the Scheduled Calendar, for example, to get the count of Tasks scheduled to start in a month. You could use both the Schedule and Started Calendars to get a count of Tasks that were both Scheduled and Started in a month. If you are directly querying your relational Data Warehouse, it’s useful to create a view for each role-play of the calendar. If you are using OLAP, it is not necessary.

Junk Dimensions

Transactions typically produce a set of miscellaneous, low-cardinality flags and indicators associated directly with the transaction. E.g. Sale Type, Sale Status etc. Our first instinct is to create a Sale Dimension to “house” these flags and indicators. However, a Sale Dimension would have one member for every Sale. That would mean the Dimension Table potentially has millions, hundreds of millions of rows, or even billions of rows. This would result in very poor performance.

Another remedy is to create a separate Dimension for each flag or indicator. However, this leads to a proliferation of Dimensions.

Rather than making separate dimensions for each flag, the best remedy is to combine them in a single “Junk” or degenerate dimension. Junk dimensions contain either the full Cartesian product of all attributes’ possible values or a combination of just the values that actually occur in combination within the source data.

Using the cartesian product is generally less resource-intensive. To generate a list of only existing combinations means scanning the whole source transaction table.

The more attributes you have in the Dimension, the bigger it gets. It’s important that the attributes are low cardinality. If you have 10 * 345 * 81 * 3 * 21 attribute values, you have 17M+ members. So you need to be careful what you include. In this case, perhaps the attribute with 345 values can be separated into its own dimension.

The business key of Junk Dimension is the combination of all attributes. When you do a lookup from the Fact to the Dimension to associate the correct member of the Junk Dimension, you need to combine all the attributes as the Lookup value. One tactic is to create a single business key column with a concatenation of the attribute values or a hash of the attribute values.

A Fact doesn’t exist in isolation. It is related to a set of Dimensions. The Dimension Tables represent the business entities involved in the business process or event—the Who, What, Where, When, and Why of the business event. Dimensions provide context about the Fact.

Below is a diagram depicting an example “Job” Fact and its related dimensions – Customer, Calendar (i.e. Date), Time of Day, Employee and Address. It models a business event where Employees are sent on “Jobs” to mow lawns. Each Job is a business event and, therefore, can be modelled as a “Job” Fact.

The Job Fact table contains the measures related to the Job. For example:

• The $ cost of the job (e.g. $200.00)

• The duration of the job (e.g. 1.5 Hrs)

• The sq meters of the job (e.g. 200 sqm)

Measures, also known as metrics, are numerical values that can be aggregated; meaning they can be summed, averaged, or subjected to other mathematical operations. However, some measures are only semi-aggregable or non-aggregable. You can read further about measure additivity in the “three types of measures”.

A Fact table also contains a primary key (in this case, a “Job Id”) and the foreign keys to its dimensions. You can learn more in the schema of a fact table section.

A fact table works with its related Dimensions to support ad hoc analysis, reporting, and dashboards. For example, perhaps you want to know the average cost per sq meter for Pensioners. First, you select all jobs where the associated Customer has a Pensioner Status = true. Then, you apply a formula over the result set of fact rows to calculate the average cost per square meter—in this case, “sum(cost of job)/sum(sq meters).”

Why do you need Fact tables?

Fact tables are one element of a simple but powerful data modelling technique called dimensional modelling.  Dimensional modeling focuses on delivering simplicity for the end user. It allows users to easily understand and navigate the data available for reporting and ad hoc analysis. A dimensional model supports high-performance aggregated queries and supports many different analytics use cases.

The 3 Types of Fact Tables

There are 3 types of fact tables.

1. Transaction Fact table

2. Accumulating Snapshot Fact table

3. Periodic Snapshot Fact Table

The 3 types are derived from the 3 types of business events:

• Discrete business event (transaction fact)

• Evolving business event (accumulating snapshot fact)

• Recurring or Periodic business event (periodic snapshot fact)

Each type of fact event/process is modelled differently.

Transaction Fact Tables (Discrete Business Events)

A transaction fact table is used to model discrete business events. Discrete events are “point in time” events, usually of short duration. They are typically atomic-level transactional events.

 Examples include:

• The customer buys a product.

• A marketing email is sent.

• A meter reading.

• A visit to a website.

Discrete events are completed at the moment they occur, or shortly after. Discrete events are generally associated with a single date and time.

Discrete Events are represented as Transaction Fact tables in the Data Warehouse.

A transaction Fact represents a discrete business event. It is often identified as a single record within a source database. It could be part of a larger business process, but this one event within the process generates its own record in the source database.

Accumulating Snapshot Fact Table (Evolving Business Events)

Evolving events are long-running business processes. They are often a series of related discrete events.

Examples include:

• Purchasing: Requisition, Purchase order, Vendor Invoice, Delivery, Payment.

• Home Care Visit: Scheduled, Started, Finished, Confirmed, Billed.

• Mowing Job. Booked, Scheduled, Started, Completed.

Evolving events are typically associated with multiple dates and times, one per discrete event in the series of related discrete events.

Evolving Events are modelled as accumulating snapshot Fact tables.

An accumulating snapshot fact captures multiple business process events within the one fact table. Usually, the events represent an entity proceeding through a series of known statuses. For example, a work item could go from proposed, to approved, to in progress, and complete. Each event has its own timestamp. Accumulating snapshot fact measures usually include the duration it takes to move between each event of the process and other measures of each event. Usually, each event of the process involves updating a single existing record in the source system. The history of the fact is captured by updating the timestamp on which the fact changes to each status within a single row.

Periodic Snapshot Fact Table (Recurring Business Events)

Recurring Events are period measurements that occur at predictable intervals, such as daily, weekly, monthly, etc.

Examples include:

• Balance Sheet. Financial Account Balances.

• Nightly Balance for Bank Accounts.

Recurring events are typically used to sample and summarize discrete events, especially where cumulative measures are required. Balances are stored because deriving the balance from the start of time by summing all discrete events for an account is expensive.

Recurring Events are modeled as periodic snapshot Fact tables.

A periodic snapshot fact captures the aggregate or balance of a business process or event for a given period. Common examples are monthly financial account balances, monthly bank account balances, etc. Periodic Snapshot fact tables are usually built from the data contained in a transaction fact table. They start with an opening balance (from the previous period) tally up the transactions for the current period and produce a closing balance. However, periodic snapshot fact tables may also represent aggregations (SUM, AVG, etc) of a period. For example, at the end of each day, the rolling 12-month sum of Asset failure minutes. The fact is a historical snapshot at a point in time.

Factless Fact Table

Okay, Okay. I know I said there were 3 types, but this is the exception to the rule! A factless table is designed to simply acknowledge the existence of a relationship between 2 or more dimensions. There is no business event.

A good example can be found in companies that manage electrical assets. The poles and wires that deliver electricity to houses. Imagine you wanted to count the number of assets of a particular type that exist within a region. An Asset factless Fact table with relationships to an Asset and Region dimensions would serve this purpose.

Why is it called a factless fact table? An “eventless” fact table would be a better description since no event needs to occur to record the existence of the relationship. Another description might be a “measureless” fact table since factless fact tables are assumed to have no measures. But even that is a misnomer. A factless fact table does, in fact, have a single measure: a “count” measure. I think the confusion arises because it’s not necessary to explicitly put a count measure column on a fact table to be able to count rows. As a rule, I always explicitly add a count measure column, with a value of 1 in every row. It makes it clearer why the factless fact table exists, and makes it easy to create the count measure in a semantic layer.

Is a factless fact table a “design smell”? Not exactly, but it could indicate you haven’t considered the nature of the “exists” relationship. Especially if you throw in Calendar/Time as a dimension. When does the relationship exist? When did the relationship start to exist, and when did it cease? This thinking can often lead you to build an accumulating snapshot fact table. In our Asset example, consider the lifecycle of the Asset. It’s planned, constructed, enters operation, and retired. All of these events occur at a point in time. If you created an accumulating snapshot fact that matched the business process, you can still answer the original question, “Count the number of assets of a particular type that exist within a region”, but it also adds the ability to enhance that question to “Count the number of operational assets of a particular type that existed within a region at the end of FY 26“, which is probably a more accurate request, that better matches the users intent.

Facts vs Dimension table

Facts and dimensions serve different purposes but work in concert to support reporting and analytics.

As discussed previously, a Fact table represents a business process or event. It contains the measures/numbers you want to analyze, whereas Dimensions are a business entity related to the business event the Fact represents. A business entity is the people, places, and things that perform a business process or event.

The Fact only contains the measures (e.g., $, quantity, count, duration, etc.) related to the business event and foreign keys to the related dimensions. The Dimensions contain the context/qualitative attributes of the related business entities. For example, a Customer dimension might have attributes like Name, Gender, Pensioner Status, Has a Dog, etc.

Facts and Dimensions exist together in a Star Schema. When analysts use the star schema to answer a business question, they filter the facts by selected dimensional attributes. E.g. “Select Facts related to Customers where HasADog = ‘true'”, then aggregates a measure to produce a value E.g. “Select AVG(duration) From Facts related to Customers where HasADog = ‘true'”

The schema of a fact table

A Fact table contains four types of columns:

1. Measures. Measures are the “numbers” or metrics by which you “measure” your business process/event. Typically, a measure is an amount, count, $ value, duration, area, length, ratio, etc. A measure is always numeric. A fact table can have multiple measures.

2. Foreign Keys to Dimensions. Facts contain a foreign key to each Dimension associated with the Fact. Typically, the relationship is with the surrogate key of the dimension. These foreign keys are the “link” that allows Fact tables to be filtered by the attributes of the Dimensions.

3. Primary Key. The primary key is the unique key used by the source of the Fact to identify a unique instance of the business event. E.g. job_id in the source Job management system. A primary key can be a composite key. A primary key is important. The ETL process uses it to determine if it needs to insert or update a row. There should be only one row in the fact per primary key. See the “Fact table grain” for more details.

4. Management Columns. Management columns are used by the ETL process and occasionally at query time. They provide metadata about the row. Examples include effective dates, deleted indicators, latest indicators, and batch load IDs.

A Fact is designed to be “narrow and long” versus dimensions which are typically “short and wide”. A fact table could contain billions of rows (i.e. long). It’s desirable to store as few columns as possible (i.e. narrow) to aid performance. The contextual information (which is generally substantial) is offloaded to the “short and wide” dimensions. Dimensions typically contain fewer rows (they are not generally transactional in nature) but contain many columns representing all the attributes of the business entity they represent.

Typically, end users only see the measures via a semantic layer. The other columns are hidden. They are only required by the ETL process or the underlying semantic model and query engine.

The 3 types of measures on a Fact table

Additivity is a crucial concept when discussing measures. In general, analytical queries retrieve thousands or millions of rows. Once retrieved the measures are aggregated in some way. The default aggregation of most semantic and BI tools is to add, or in other words, sum the values.

• Additive. A measure is additive if it can be summed across any combination of dimensions to produce a sensible result. For example, imagine a sales fact with a sales_amount measure. A user can use any combination of dimension attributes to filter the fact rows and still sum the sales amount to get total sales.

• Semi-additive. Semiadditive measures can be added only along some of the dimensions. Usually semiadditive measures can’t be added across the calendar/date dimension. This is especially true for period snapshot fact tables. For example imagine a Monthly Inventory period snapshot fact with an end_of_month_product_balance measure. For a given month, you can add the balances of different products to get a total balance for a Product Category (an attribute of a Product Dimension). However, you can’t add the end_of_month_product_balance product balance for January + February + March (Month attribute from the Calendar/Date Dimension) to produce an end_of_quarter_balance. This doesn’t make sense. Instead you need to use a different aggregation (e.g. Count, MAX, MIN or AVG) over the Calendar/Date Dimension after summing across all other dimensions.

• Non-additive. Non-additive measures can’t be added at all. Nonadditive measures must use a different aggregation (e.g. Count, MAX, MIN, but not AVG) over all dimensions. Measures that are percentages and ratios, such as gross margin, are always nonadditive. Percentages and ratios cannot be averaged either. Instead of storing nonadditive measures in the fact table, it is better to store the numerator and denominator. The ratio can be calculated in the semantic layer or reporting tool later. When calculating the ratio, ensure you calculate the “ratio of the sums”, not the “sum of the ratios”.

A good example is unit_price. unit_price is the ratio sale_amt/units_sold. See the image below of analysis of 12 sales of a given product.

Summing unit price across any of the dimensions results in a nonsense number. Therefore it is nonadditive. Averaging unit_price results in an incorrect value because you are “summing the ratios” instead of the “ratio of the sums”. In order to analyze the average unit_price for a product, you do the “ratio of the sums”, which is sum( sales_amt)/sum(units_sold). The MIN or MAX unit_price is a valid aggregation.

A fact can contain a mix of additive, semi-additive and non-additive measures. For example, imagine the inventory fact discussed. It could have the additive measure total_units_sold, the semi-additive measure end_of_month_product_balance, and the non-additive measure avg_unit_price.

Understanding Fact Table grain

The “grain” of a fact table describes the lowest level of detail available in the fact table. It provides the answer to the question, “How do you describe a single row in the fact table?” The atomic level of a fact table aligns with its primary key.

In the case of a transaction fact table. the preference is to store data at the most atomic level captured by a business process. Atomic data is the most detailed information available.

Examples of the grain of transaction facts are:

• An individual line item on a retail sales receipt. The primary key is sales_id and line item number.

• An individual product produced. The primary key is a product serial number.

• An individual mowing job is done. The primary key is the job_id.

Period snapshot facts have a grain that includes a snapshot business process and a calendar interval. For example:

• A daily snapshot of the inventory levels for each product in a warehouse. The primary key is product and day.

• A monthly balance for each bank account. The primary key is the account and month.

All measures on a Fact table share the same grain. If a measure exists at a different grain, you need another fact.

There is a misconception that the grain of a fact aligns with the intersection of all its dimensions. This isn’t true. It’s typical for a fact to exist that doesn’t yet have all of its possible dimensions. Instead, it is the atomic level of detail of the underlying business process.

There is also a misconception that dimensional models should only contain summarized data. This mistake invites the criticism that the dimensional model needs to anticipate the business questions at various grains. Fact tables with grains that represent a higher aggregation of the most atomic data are limited to fewer and less detailed dimensions. Less granular facts inevitably run into issues when users try to drill down into details.

In contrast, an atomic fact offers the utmost analytical flexibility. It can be filtered and aggregated in every conceivable manner, across detailed dimensions, at all levels of granularity. Atomic data within a dimensional model can accommodate a variety of unforeseen use cases and ad-hoc analyses.

And don’t forget, you can have more than 1 fact table! It’s typical for a data warehouse to have more than one type of fact for the same business process. Typically, you would couple a transaction or Accumulating Snapshot fact with a periodic snapshot fact.

For example:

• An inventory transaction fact. I.e. Add to inventory and take from inventory transactions.

• A monthly periodic snapshot of inventory balances.

Why have different types of facts for the same business process? Because each one can answer different questions.

Don’t try to mix grains to fit into a single fact. An example where people make this mistake is a sales fact. Certain measures exist at the line item level, e.g. num_of_units and $ amount. Other measures only exist at the Sale level, like discount_given, and sales_tax. You could try apportioning the discount to the line item level, but a better and easier approach is having 2 facts. A fact at the sale grain and a fact at the sale_line_item grain. You use conformed dimensions to allow analysis across these two facts, but that’s a story for a different article.

Dimensional modelling is broadly accepted across the industry. It was first popularized by Ralph Kimball. Ralph Kimball’s data warehouse methodology remains relevant, even today, with the advent of Data Lakes and Data Lakehouses. In the Kimball methodology, a dimensional model is delivered using a modelling technique known as the “star” schema.

The 3 essential building blocks of Dimensional Data Modeling

A dimensional data model consists of 2 types of tables (Facts and Dimensions) and their relationships. Thus it’s simplicity. Facts and Dimensions are related together and arranged in a “star” schema. The image below is an example star schema. It depicts an imaginary video streaming company, GetFlix, and a “Shows Viewed” fact table. The “Shows Viewed” fact table is at the centre of the Star Schema and surrounded by its Dimensions.

The following is a brief introduction to these concepts.

Facts tables

A Fact table represents a business process or an event within a business process. It contains the measures/numbers you want to analyse. For example, the “GetFlix” “Shows Viewed” fact table depicted in the star schema example above contains one row for every time a customer views an Episode of a Show. “Customer views an Episode of a Show” is the business event we are measuring.

Dimensions

A Dimension represents a business entity, which is the people, places, and things that come together to perform business activities. A Dimension contains the attributes (i.e. fields) of the business entity, e.g., Customer Gender, Age, Category, etc. The attributes are used to filter and group fact data when performing data warehousing queries.

Star Schema

A Star Schema refers to how Facts and Dimensions are related in a Data Warehouse. A Star Schema is organized around a central fact table that is related to its Dimension tables using foreign keys in the fact table. The name “star” schema refers to the star-like pattern that emerges when you present the Fact table and its Dimensions in an entity-relationship diagram. See the example above.

How to Build a Dimensional Data Model

Using the 3 essential dimensional data modeling building blocks discussed above a data warehouse modeler can build an entire data warehouse data model.

Very briefly, the process is:

1. Identify the business process.

2. Identify the grain (i.e. unique key) of the fact or fact(s) of the business process.

3. Identify the Who, What, When, Where and Why are involved in the business process. The Dimensions of your star schema.

4. Identify the sources of data. Write the code to extract the data into the data warehouse.

5. Write code to transform the data into a form suitable to be loaded into the presentation layer of the data warehouse.

6. Write code to load the data into the presentation layer of the data warehouse modeled as a star schema.

Obviously, there is a lot more to it than that. Data warehouse data modeling and architecture is a specialized field that benefits from training and years of experience. This is just a quick overview.

Star Schema vs Enterprise Data Warehouse Data Model

So far, we have discussed using a star schema. However, a star schema only describes a single Fact table. An Enterprise data warehouse should model many different business processes/events, each with its own Fact table and related Dimensions joined in a star schema.

The question is, how do we prevent duplication of the Dimensions used by Facts and facilitate cross-business process analysis?

The answer is “Conformed” Dimensions. Essentially, a conformed Dimension is a Dimension shared by many facts. Typical conformed Dimensions are key business entities like Employees, Customers, Products, Cost Centres, Dates, and Times.

When two Fact tables share a Dimension, you can produce a report with measures side-by-side from both Fact tables, categorized or filtered by attributes of their conformed (i.e. shared) Dimensions. In this way, dimensional modeling with conformed Dimensions supports enterprise-wide cross-business process analysis.

Another way of representing conformed dimensions is through a data warehouse matrix.

Typically, on a BI project, if you ask a business user, ‘what do you want’ you will get one of 2 responses.

1. I don’t know.

2. I want everything.

Which are, effectively, the same thing…

As much as it pains IT people to hear these responses, if you think about it, they are actually the right answer. From day to day, a business user, especially any analyst in a business, doesn’t know how they will want to view and analyse their data. They are responding to changing business environment on a day to day basis. The BI team’s job is to deliver the capability to analyse data in a variety of ways to these business users.

Gathering requirements for a Data Warehouse project is different to Operational systems. In Operational systems, you can start with a blank sheet of paper, and build exactly what the user wants. On a Data Warehouse project, you are highly constrained by what data your source systems produce. Therefore letting an end user go wild with all kinds of esoteric requirements can lead to horrible disappointment.

There is no silver bullet. Like on most projects, you have to work with what’s in front of you. Organisations undertaking BI projects all start with very different levels of vision, understanding and expertise. Each situation calls for a unique approach. However, one thing we find is true is that the requirements gathering process is as much an education process as it is a requirements gathering exercise. Most business users are unaware of the possibilities of BI, and will often describe their requirements in terms of what they have had before (e.g. A list all sales for June that I can import into Excel…)

Building a Data Warehouse is mostly about building capability, rather than delivering specific report outcomes. It’s a mistake to take a ‘Business Intelligence’ requirement (i.e. this Report, that report… etc) and build a data Warehouse just to satisfy the reporting requirement. Tomorrow, a new requirement might arise, which would fundamentally change the Data Warehouse (Usually the detail level, known as the grain, of a Fact table).

In the next sections, we outline 3 different approaches to gathering business requirements for a data warehouse.

Ontology

The mantra for Data Warehouse design is “Model Reality”. In previous lessons, we have discussed how Facts represent Business Events/Processes and Dimensions represent business entities.

The natural way to understand the requirements of the Data Warehouse is to simply describe the Business Entities and Processes.

To do this we recommend writing an Ontology. An Ontology is a big word for a simple concept:

• Ontology: “a set of concepts and categories in a subject area that shows their properties and the relations between them.”

Keeping things simple, we recommend creating a document with the following format:

Business Process

Below is an example of a Business Process description in an Ontology document:

• Definition: Provide a business definition of the business process.

• Business Events. If the business process is evolving, provide a list of business events.

• Measured By. List how the process is measured. Obviously, these become the measures in your Fact.

• Related To. List the business entities that represent the who, what, where, when and why of the business process. Give a cardinality where it makes sense.

• Business Domain: The business domain this entity belongs too. It’s an indication of what data mart the entity will belong too. If you want to keep conformed dimensions in a Master Data Mart, then, set this value to master.

• Type of: If this entity is a subtype of another entity, name the other entity here.

• Subtypes: The list of subtypes of this entity.

A business process will become a Fact in the Data Warehouse.

Business Entities

Below is an example of a Business Entity description in an Ontology document:

• Definition: Provide a business definition of the entity. Keep this definition relatively concise. Be careful not to wander into describing other entities and processes in the business.

• Attributes: Include a list of attributes of the entity. Provide example values where possible.

• Synonyms: Where it’s possible to identify the same entity in a source system, name the same entity as a synonym. This can help understanding, especially if you have multiple systems providing the same role.

• Business Domain: The business domain this entity belongs too. It’s an indication of what data mart the entity will belong too. If you want to keep conformed dimensions in a Master Data Mart, then, set this value to master.

• Type of: If this entity is a subtype of another entity, name the other entity here.

• Subtypes: The list of subtypes of this entity.

A business entity will become a Dimension in the Data Warehouse.

Model Storming

A very similar approach, with much more detail about how you arrive at these definitions is described by Lawrence Corr in his book Agile Data Warehouse Design: Collaborative Dimensional Modeling, from Whiteboard to Star Schema. It is a step-by-step guide for capturing data warehousing/business intelligence (DW/BI) requirements and turning them into high-performance dimensional models in the most direct way: by model-storming (data modelling + brainstorming) with BI stakeholders. It describes BEAM✲, an agile approach to dimensional modelling, for improving communication between data warehouse designers, BI stakeholders and the whole DW/BI development team.

Data Driven Design

Bottom Up “Data Driven Design” is also an exercise in “Modelling Reality”.

Data Driven Design refers to a design technique that focuses on examining the data and database schema of Source Systems to understand the Business Processes and Entities of the Business. The fact is that the Business process and entities exist in the source systems, in detail.

A forensic examination of the source systems can reveal the business processes and entities.

Data Warehouse projects have certain characteristics that make them suitable for Data Driven Design. The key characteristic is that Data Warehouse projects are highly constrained. They are constrained by the data contained in the source systems of the Data Warehouse, and, from a requirements perspective, a Data Warehouse is constrained to modelling existing business processes (other than perhaps reporting and management processes).

Unlike an operational system project where development is about new or changing business processes, it is not realistic to get requirements about, for instance, what attributes of a Customer that need to be captured, and then model into a Data Warehouse Data Model. First… the underlying Source System may or may not support the attributes and secondly it is easier to go directly to the existing Source System and discover them.

Data Driven Design will shortcut the requirements process. Clearly existing Business Process will be manifest in one or more Source Systems, and can be ‘discovered’. Thus a Data Driven Design approach can be taken, using existing data to derive a design for the Data Warehouse. Data Driven Design doesn’t mean ignoring business requirements all together. To the contrary the business needs to be brought along on the journey, but only limited and specific input and prioritization is required from the business in the early iterations. In later iterations the business will be better equipped to properly articulate their requirement.

Overview of Data-Driven Design for Data Warehousing

Below is an outline of Data-Driven Design and Agile Methodology applied to Data Warehousing.

First Iteration

The first iteration is a little different. It is as much a business education process, as it is a development exercise. The four steps of the process are described below. They will occur in roughly this order, but it’s likely that some steps will occur simultaneously and repetitively as required.

1. Discover the Business Processes that are included in the project scope. Data Driven Design doesn’t mean ignoring business requirements altogether. To the contrary, the business needs to be consulted constantly and brought along on the journey. A detailed description of the Business Process is not required. Just a simple accurate story card for each Business Process and Business Event, identifying the key entities involved (The Who, What, Where, When and Why) and how the Business Events are measured. For example… an Ontology! The identified Business Processes need to be prioritized in a business value order. This order needs to be weighed against the technical complexity of each Business Process to decide which Business Processes will be implemented in the first iteration of the project. Try to choose a relatively high value and simple Business Processes for the first iteration.

2. Discover the Source Systems for the selected iteration 1 Business Processes and get access to their database/files etc. Having real-life data is important, and can sometimes be a challenge.

3. Discover within the Source System database(s) the manifestation of each of the Business Process. This will require communication back and forth with the business representatives. Think of yourself as a data detective. A forensic data detective, CSI style! Quite often Source Database tables and columns are well named and easy to understand. In many cases, the Database schema is undecipherable. Many packaged ERP suffer from this. Every Holmes needs a Watson. Collaboration with a data-focused Subject Matter Expert (SME) or experienced Source System programmer is essential. A data focused SME will have experience with the underlying data, perhaps learned from extracting data from the Source System Database for reports. This step will identify the tables, columns, data and grain of data involved in each Business Process. It should also discover various cases and examples of the Business Process within the data. Save these cases if possible, they become useful for testing later. Physically, this step will involve a lot of data profiling and querying of the Database. Other resources you will find useful are Source System user manuals, technical manuals, a data dictionary or existing reports that extract data.

4. Develop a first cut Data Warehouse Data Model and ETL based on what has been discovered. The Data Model will contain only those tables required for the first iteration but must conform to good Data Warehouse design principles, so that the model can be easily expanded in the future. It is absolutely essential that a next generation Data Warehousing tool like Dimodelo Data Warehouse Studio is used to develop the Data Warehouse and ETL. The Data Warehouse and ETL is going to go through many iterations, much change, plenty of regression testing and several releases. Short iterations are just not feasible without a tool like Dimodelo Data Warehouse Studio.

The idea of the first iteration, and to a lesser extent, other early iterations, is to give the business a first look at the Data Warehouse. This helps to get them thinking dimensionally and improve their ability to articulate requirements. Gathering requirements is traditionally a major issue in Data Warehouse projects. The first iteration is an education process, helping the business understand the capabilities of BI. We recommend you demonstrate standard reports, dashboards, scorecards and ad-hoc analytics. Only deploy the first iteration to a sandpit environment. Somewhere the users can ‘play’. Keep expectations of data accuracy low, it’s unlikely you will nail the data 100%. It’s possible you may even need to start from scratch again, but that’s not an issue with Dimodelo Data Warehouse Studio or other similar tools.

Subsequent Iterations

Rinse and Repeat. Subsequent iterations follow the same process as the first iteration but the focus of subsequent iterations is to improve and refine the existing solution based on feedback from previous iterations. Only introduce new functionality for new Business Processes when resources allow.

Subsequent iterations introduce additional Agile disciplines, like Test Driven Development, Continuous Integration, Source Control and Release Management, each of which could fill several blog posts on their own. It does suffice to say, automation is the key.

It’s likely that subsequent iterations become release candidates for release into production. Again discipline is required. Releasing a Data Warehouse into production is not an easy task, especially if there is an existing version already in production. At this point, it is also important to focus on non-functional requirements like security, disaster recovery etc. Usually, the planning for non-functional requirements would have already occurred, and there is nothing wrong with that. However, where there is a fairly standard architectural scenario, there is an argument that it may not be wise to invest time and money into the architectural tasks until a viable Data Warehouse and BI solution that meets the business needs has been proved.

.. Now… if only we could get the PMO, the Op Ex and Cap Ex budgets, and the business to align with the uncertainty of an Agile project… A whole other blog post again.

The Criticism of Data Driven Design

In the past Data Driven design has been used on Data Warehouse projects to the exclusion of all other techniques, and without using an Agile project methodology or Agile Data Warehouse development tools like Dimodelo Data Warehouse Studio. User involvement was avoided or minimized (which suited IT). It was mistakenly thought that Data Warehouse development meant simply re-modelling multiple source systems into a single Data Warehouse model. Only after the system was built, using inefficient, error-prone, time consuming and inflexible manual coding practices were the users engaged properly. Unfortunately, more often than not, the results miss the mark and failed to answer the business questions. With an inflexible ETL framework that takes months to make any significant change, the project team found themselves in trouble, and potentially out of budget.

The real problem was the time the development team took to develop the solution and the inflexible nature of the resulting system. Shorten the development cycles, use a Data Warehouse Automation tool, call the result a prototype, and a paradigm shift occurs. Iterate multiple times to improve and refine the application to the business requirements.

Why use the Agile Methodology for Data Warehousing Projects?

Many IT projects fail. Depending on who you believe, its anywhere between 50% and 80% (See references below). Shocking! I believe that you need to take these figures with a grain of salt. It depends on your definition of failure (See discussion on failure below).

It has also been shown that small projects are more successful than large projects and those projects using the Agile methodology are also more successful (See references). Given that small agile projects are the most successful and that Gartner states that over 70% of BI projects are not successful, it follows that applying an Agile project methodology to a Data Warehouse project is a sensible strategy. Data Driven Design is a perfect companion for Agile.

Failure Rate References:

• Gartner 70-80% of BI Projects Fail and Report

• Standish Chaos 2009

• Standish Chaos Original

Small projects are more successful than large projects references:

• Gartner Survey Shows Why Projects Fail

• Failure of large projects – why it is better to be small particularly in IT
  Agile project methodology is more successful references:

• Ambysoft Project Success Rate Survey

• Agile Business Intelligence Would Be A Good Idea

• Agile succeeds three times more often than Waterfall

Definition of Project Failure

The definition of project failure is usually a project that has a combination of the following issues: significantly over budget, significantly over time, doesn’t deliver full functionality, or is cancelled outright. However, I think it’s more subjective than that, and depends on project context. If a Home building project went over budget 20% or time by 20%, but at the end of the project, the house was everything the client wished for… is this a failure? The answer – depends… For a more in-depth understanding of the subjective contextual nature of project failure, you can read The rise and fall of Standish Chaos Report figures.

Testing your Design

This is an essential step that will improve your design.

How do you test your design? No first cut design stands up to the appearance of a report! Designing a data warehouse to cater only for existing reports, spreadsheets and analysis is a mistake. However, you should test your design against those existing reports etc.

Can you produce those reports from your data warehouse? Inevitably the answer will be no. You will discover new calculated measures, hierarchies and different aggregations needed to respond to these requirements. You may discover you need new periodic snapshot tables or entirely different facts from different domains.

Conclusion

In my experience, the Ontology method is the most successful way of defining Data Warehouse requirements.

If you want to take this exercise even further I would recommend the Model-Storming approach. Indeed, I would highly recommend at least reading the Laurence Corr’s book. It will improve how you write an Ontology.

The Ontology exercise will most likely be informed by a Data Driven Design approach. Because your source systems are the manifestation of your business process and entities, it is likely you will discover the processes and entities you need to document through Data-Driven analysis.

Once the ontology is complete you then use it to define your Data Warehouse Matrix. This then becomes your high-level design document, project scope and key project planning and communication tool.

I highly recommend you undertake this style of requirements gathering before starting the build of your data warehouse. I see many implementations dive straight into the design of facts and dimensions. The requirements gathering exercise is the missing link between the business and the design.

I’ve seen many experienced data warehouse developers encounter an ontology document for the first time and thank God for its existence! Literally.

It need not be a long exercise. A 3-day workshop with experienced data analysts, business analysts and/or SMEs is all you need to gather the information for a reasonably complex domain, plus perhaps 2-3 days to document. This document will keep delivering value, especially as the makeup of the team changes. It may be the only definitive definition of the business processes in existence.

A data warehouse should exhibit the following four characteristics:

1. Integrated. A data warehouse takes a copy of the data (on a regular basis) from the enterprise’s application systems. It integrates that data into “one place”, simplifying data access for reporting purposes. A Data warehouse doesn’t capture or create new data itself.

2. Subject-oriented, not source system-oriented. A data warehouse reorganises the source data into business subjects/domains, making it easier for users to understand and consume.

3. Historical/Time Variant. A Data Warehouse records the history of how data changes. This is important for accurate reporting, auditing and efficient data management.

4. Non-Volatile. Data is loaded in periodic “batches”. The data doesn’t change from moment to moment; rather, it’s stable between load periods. This means reporting and analysis can be conducted without the data changing underneath you.

At the heart of a data warehouse is the ability to organize data so end users, report developers, data analysts and data scientists can easily consume it. What do I mean by easily consumed? First, they have a single source of trusted, modelled, easy-to-understand data. Second, users can access that source through BI tools that make it easy to create new reports and analyses or interact directly with the data via code (SQL, R, Python, etc).

You need solid data architecture and modelling techniques to achieve the ideal of “easily consumable data”. The most commonly used architecture and modelling technique is the “Kimball” data warehouse methodology. The Kimball methodology describes:

1. A layered data architecture typically comprised of Staging, Transformation, Presentation and Semantic layers.

2. A Dimensional data modelling technique that models data in a simple, easy-to-understand “Star” schema.

• Data from multiple systems is landed in the staging layer. It implements the first “integration” characteristic described above, i.e., simply having all your data “in one place.” A persistent staging layer is a common variant that also captures the history of change, implementing the third “historical” characteristic.

• The transformation layer is where data begins its journey into a subject-orientated format (the second characteristic). Transformation can be complex, and the transformation layer is where this complexity is implemented. It also supports the reuse of the logic by the presentation layer. In this layer, data from multiple systems is often matched and merged into subject area entities.

• The presentation layer is the final subject-orientated representation of the data (the second subject-orientated characteristic). It is modelled using a strict “star” schema. The Star Schema is an easy-to-understand data model that is easy to navigate and supports multiple analytic use cases. The Star schema is comprised of Dimensions and Facts and their relationships.

• Typically, the semantic layer is the layer exposed to end users. End users interact with the semantic layer for data analysis and visualization via BI tools like PowerBI or Tableau. A semantic layer is implemented in a technology that facilitates high-performance aggregated queries over large amounts of data. The semantic layer’s data model mirrors the presentation layer’s Star schema. It further augments the data model with calculated measures that can’t be created in other technologies. More advanced users like data scientists and advanced data analysts may be granted access to the presentation of transformation layers as needed.

• Extract, Transform, Load (ETL). ETL is not a layer but rather the logic that moves data between layers. The ETL code extracts data from the source, transforms it into a Star schema, and loads it into the presentation layer. ETL is usually executed periodically (daily, weekly, four hourly, etc.) and loads batches of data that have changed in that period. It implements the “non-volatile” characteristic.

An alternate architecture growing in popularity is the data lakehouse. A data lakehouse is a combination of the data lake and data warehouse architectures. The data lakehouse describes Bronze, Silver and Gold layers that correspond to the Staging, Transform and Presentation layers of the Kimball Data Warehouse architecture. This is a vendor lead (Databricks) reimagining of the data warehouse implemented on different technologies. Data warehouses have traditionally been developed using database technology, whereas the data lakehouse is born out of the open source big data, file system based (HDFS), and Spark ETL technologies that Databricks have commercialized.

Why you need a Data Warehouse

The universal truths of data in every organisation I’ve worked in are:

1. Data is messy!

2. Someone, somewhere, somehow needs to deal with the mess.

3. The best place to deal with the mess is in a centralized data warehouse rather than over hundreds of disparate reports.

The best way to understand the benefits of a data warehouse is to imagine your life as a report developer without one.

1. First, your data is spread across multiple systems, most of which you cannot access.

2. Second, the data quality is poor. It’s incomplete, inaccurate, contradictory and not up to date.

3. Third, you need to mix, match, and merge that data with data from other systems. There may not be an obvious or easy way to do that.

4. Fourth, the volume of data you need for your report is massive, and the source systems cannot or will not cope with your query.

5. Fifth, the source system doesn’t keep a history of data change, so the reports unexpectedly change for end users.

6. Lastly, you use some logic in your report that doesn’t match the logic of someone else’s report, so end users are using and comparing results that don’t match across reports.

Imagine a bunch of developers all creating reports and analyses and solving the same data problems over and over again. This would very quickly result in a chaotic and unmaintainable mess.

The situation above is what inspired the development of the data warehouse concept.

The benefits of a data warehouse include:

• Integrating data from multiple sources into a single data store provides easy access to data across business domains, such as financial, HR, operations, and sales.

• Ability to model data to make it easy to consume. By modelling data in an integrated subject-orientated, easy-to-navigate star schema, data becomes much easier for end users to work with and consume. This leads to the next point – end-user productivity.

• End-User Productivity. In any organization, a subset of people spend part or all of their day producing information in one form or another. Typically, they spend much of their time wrangling messy data. One of the benefits of a data warehouse is end-user productivity. All that data manipulation is already done, and the users can concentrate on analysing and responding to information rather than producing it.

• A single version of the truth. A data warehouse lets you implement the logic for metrics once and then have all reports and analyses use the data warehouse as their source. That way, there is just a “single version of the truth”. Business users work with agreed definitions for KPIs, metrics and measures.

• A Data Warehouse is a sustainable solution that copes well with underlying change. The Transform/Presentation layer has its own business-focused model that doesn’t use source system schema or language. This delivers a level of abstraction between data sources and reports. This means that even if the data source changes (which happens constantly!), the business-focused model can remain the same, and reports that depend on that model can remain untouched. The new source does need to be “plugged into” the business-focused model, which can be done with little impact on reports.

• Remove load from operational systems. Operational systems are generally tuned to manage hundreds, if not thousands, of small individual transactions. Analytics can require querying millions (even billions) of rows. For example, a total annual sales metric for a large e-commerce platform requires summing across all sales rows in a year. A single analytical query can cause major performance issues for an operational system. Separating reporting and analysis load to a data warehouse removes adverse impacts on operational systems.

• Analyze the past as it existed in the past. For example, imagine a salesperson (let’s call him Gary) works in region A. A regional sales report rolls up the sales figures for Jane into the total for region A. Imagine Jane moving from Region A to Region B. In the report, Jane’s past sales suddenly moved from Region A to Region B, which was undesirable. A Data Warehouse has modelling methods to prevent this issue. In a data warehouse, historical sales would remain associated with Region A, and only new sales would be attributed to Region B.

• Eliminate Personnel Risk. Business logic is often locked up in spreadsheets or visualizations (Power BI, Tableau, Qlik, etc.) created and managed by individuals. There is a risk that if that individual leaves, no one else is able to manage these sometimes business-critical spreadsheets or reports. Centralizing your business logic (measures, KPIs, etc.) managed by a data warehouse team reduces this risk.

• Data augmentation for reporting purposes. A Data Warehouse can model and augment data specifically for reporting purposes. This augmentation isn’t available in a source system. For example, analytics functions like relative period (e.g., MTD, YTD, etc.), periodic and rolling calendars (e.g., Christmas Period, Public holidays, etc.), the definition of acceptable ranges, targets, KPIs, data aggregation or disaggregation, and periodic balances (e.g., end-of-month balances).

• Keep historical data. A data warehouse can capture the history of how data changes. This can assist with accurate data reporting and auditing and improve load performance. A Data Warehouse can keep historical data beyond the normal retention period of operational systems.

To align your Data Warehouse with the strategy, objectives and core processes of the business, you first need to understand the concept of the Organizational “Value Chain” and how this leads to Business modelling and Data Warehouse design. This ultimately feeds back into strategic decision-making.

Understanding the Organization’s Value Chain and conducting Business modelling to define the Data Warehouse design massively enhances the value derived from Reporting and Analysis.

Organization Value Chain

What is an Organizational Value Chain?

Organizational “Value Chains” were first introduced by Michael Porter in his book “Competitive Advantage“.

A value chain is a set of connected activities an organization carries out to create value (i.e., products and services) for its customers. The value created for the customer should be greater than the cost of production, and the result is the margin. A value chain is a systematic way of examining these internal activities and how they interact to enhance customer value, margin and competitive advantage.

Porter divided his activities into the following:

• Primary activities. Primary activities are related directly to the production, distribution and sale, maintenance and support of products or services.

• Support activities. These activities support the primary functions above, E.g. Procurement, Human Resources, IT, Accounting, Legal, Administration, etc.

By understanding the collection of primary and supporting activities, a data warehouse designer can start to form a view of the data warehouse that aligns with the activities and strategic goals of the business.

To learn more about the Value Chain, I recommend watching this video –  Value Chain Analysis EXPLAINED | B2U | Business To You – YouTube.

Business Modelling

“Business Modeling” describes the process of capturing details of the Organizational Value Chain in a format appropriate for Data Warehousing. The actual Data Warehouse “design” work happens in this process.

What is Business modelling? – “The act of describing the properties and relationships of a set of business activities and the business entities involved in the business activities.”

• Business activities: The primary and supporting activities identified in the Organizational Value Chain.

• Business entities: The “who”, “what”, “where”, “when”, and “why” that participate in or are produced by the activities.

The Business Modeler must distinguish between business activities and business entities. This distinction is key to success. In later sections, you discover that business activities become one or more Facts in the data warehouse, and Business Entities become Dimensions.

The Business Model also serves to identify the integration of business activities across business entities, I.e., the business entities used in common across activities. These common entities become the conformed dimensions of the Data Warehouse.

The Business Model also provides a common language to the business. It becomes the basis of a Business Glossary or Data Dictionary.

Use a Data Warehouse Bus Matrix to capture the business model

You can use a Data Warehouse Matrix to capture and communicate the Business Model at a high level.

A “Data Warehouse Bus Matrix” describes the high-level design of a Data Warehouse. At a glance, it shows all the facts and dimensions of a data warehouse and their relationships in a table-like ‘matrix’.

for example:

The Data Warehouse Bus Matrix is a vital tool for documenting the design of the data warehouse, the value chain activities it covers, and their relationships with Business entities. It can quickly communicate the value chain coverage, scope, status and high-level design to stakeholders, developers and end-users.

Grab your free Data Warehouse Bus matrix template ->

“I’ve used this Bus Matrix spreadsheet on all my projects to help design, plan and estimate data warehouses and communicate the scope and progress to stakeholders.”

The feedback loop between the Value Chain, Business Modeling and the Data Warehouse

If executed correctly, there is a circular relationship between aligning your data warehouse with the value chain and the feedback from the data warehouse into value chain strategic decision-making.

1. The Value Chain drives the Business Model.

2. The Business Model defines the Data Warehouse design.

3. The Data Warehouse Matrix captures the Data Warehouse design.

4. Reports and Analysis source their data from the Data Warehouse.

5. Insights and Analysis feedback into strategic decisions about the value chain, increasing customer value, margin and competitive advantage.

The objective is Reports and Analysis that reflect the Value Chain, Business Model and Data Warehouse design, aligning reporting to business goals and delivering data and analysis that feeds directly into strategic decision-making.

Importantly, the common business language permeates at every stage, driven by the Value Chain and Business Model.

In the case of a Data Warehouse, the business value is provided when a Fact is delivered with at least a subset of its associated Dimensions. Decisions about which Facts and Dimensions to prioritize are complex. Below are some of the steps you can use to arrive at these decisions:

1. First, identify high-value dimensions. High-value dimensions are those shared by many Facts. In the example below, the high-value dimensions include Product, Staff and Calendar. By concentrating on high-value Dimensions, iterative delivery has a cumulative effect.

2. Second, identify Fact/Business Processes that utilize the high-value dimensions.

3. Third, rank these Facts by their business value vs development complexity. High business value, low complexity Facts that utilize high-value dimensions are ideal candidates.

Example Prioritization

Take the Data Warehouse shown below in a Data Warehouse Bus matrix.

To decide which Fact to deliver in the first project iteration, we did the following:

1. First, identified the high-value Dimensions as the Calendar, Staff and Product Dimensions. At least 3 facts share them.

2. Identified that the Product and Sales facts use all these high-value dimensions.

3. Determined the build priority based on complexity. The Production and Sales facts are both considered complex. However, the Sales fact has more dimensions; therefore, the Production fact costs less to deliver. Delivering the Production Fact first provides a cumulative development benefit. This benefit is realized when you deliver the Sales Fact in a subsequent iteration. The Sales Fact only requires the delivery of an additional three dimensions. I.e., By delivering the Production fact, you have already partially delivered the Sales fact.

A Data Warehouse Matrix provides a framework for overall delivery that can be decomposed into rational development sprints that conform to an overall design.

Grab your free Data Warehouse Bus matrix template ->

“I’ve used this Bus Matrix spreadsheet on all my projects to help design, plan and estimate data warehouses and communicate the scope and progress to stakeholders.”

Measuring Business Value and Complexity

It’s essential to engage the business in determining each of the Fact’s business value and priority. For better or worse, the business usually derives business value only from the perceived value of a narrow set of reports the Fact supports. Competing interests between different business functions are inevitable and challenging, to say the least. The cumulative benefit of delivering high-value dimensions can help business users understand that delivering on other users’ priorities also moves the needle closer to their requirements.

Designers measure the complexity of Facts in several ways:

• The number of dimensions associated with the Fact.

• The number of source systems and source tables required to load the Fact.

• The data format of the source system.

• The data quality of the source system.

• The accessibility of the source system.

• The complexity of the transformation logic.

• The number of measures in the Fact.

• The type of Fact (accumulating/transaction/periodic).

Designers measure the complexity of Dimensions in the same way with a couple of additions.

• The number of attributes in the Dimension.

• The number of Type 2 attributes on the Dimension.

To help with this prioritization, I use an enhancement to the Data Warehouse Matrix.

In the image below, I have added a measure of complexity to each of the Dimensions and Facts.

This simple enhancement helps in the prioritization process. It also feeds into the estimation process discussed in a later post.

Shown below is a 4-level complexity scale that usually suffices:

Balancing value, complexity and delivery

There is a complex interplay between business value, complexity, and cumulative delivery. Designers need to consider all these factors when planning iterations. Due to the cumulative nature of the Data Warehouse delivery, the complexity of delivering any given fact may change over time. For example, in a given iteration, you may have staged the required source data or delivered ¾ of the related Dimensions, which makes some of the remaining Facts less complex. Also, the business value of any Fact may increase relative to the remaining facts. Designers can take advantage of the cumulative delivery and plan a series of iterations to provide the most value quickly. Essentially, if you deliver “Fact A”, delivering “Fact B” becomes more straightforward.

One of the phenomena experienced when delivering a Data Warehouse is that early Facts and Dimensions appear to take significant effort and are more costly without providing a lot of value. Development tends to accelerate as more of the Data Warehouse is built. It can be a struggle getting over this low-value, high-cost “hump”, with the business becoming despondent and taking shortcuts. Understanding and communicating progress, value and complexity via the Data Warehouse Matrix can help mitigate this issue.

The “Data Warehouse Matrix” comes from the industry-accepted Kimball Data Warehouse methodology. In the Kimball method, practitioners are encouraged to create a “Data Warehouse Bus Matrix” to describe the high-level design of the Data Warehouse.

It should be the first thing a Data Warehouse Architect does to define a Data Warehouse project.

Free Data Warehouse Bus Matrix Excel Template

Grab my free data warehouse bus matrix Excel template, which is shown in this post.

Free data warehouse matrix template ->

“I’ve used this Bus Matrix spreadsheet on all my projects to help design, plan and estimate data warehouses and communicate the scope and progress to stakeholders.”

Why do you need a Data Warehouse Matrix?

A Data Warehouse Matrix is a simple yet powerful communication tool. Its simplicity is its core strength, making it useful for communication that works at every level of the organization, from executives to end-users and developers.

A Data Warehouse matrix serves several purposes:

1. Capture Design: Succinctly capture the high-level design of the enterprise-wide data warehouse.

2. Communication: At a glance, communicate the scope, business process coverage and high-level design of the Data Warehouse to stakeholders, developers and end-users.

3. Estimate: With a few enhancements, Designers can use the Data Warehouse matrix to estimate the time required to build the Data Warehouse.

4. Plan: Assist in planning the delivery of each development iteration of the project whilst ensuring each iteration conforms to an overall design.

5. Progress: Communicate the progress of the project.

Grab your free Data Warehouse Bus matrix template ->

A Data Warehouse Bus Matrix example

Below is an example of a very simple Data Warehouse Matrix.

On the rows of the Matrix are the Facts/Business Processes (e.g. Sales). On the columns are the Dimensions/Business Entities (e.g. Customer). The cells represent the relationships between the Facts and Dimensions. The number represents how many relationships exist between the Fact and Dimension. An empty cell means there is no relationship.

For example, the “Production” fact below is related to (or “dimensioned by”) the Dimensions Product, Staff and Calendar. This means an end-user can analyze “Production” measures by any combination of the Product, Staff, or Calendar Dimensions attributes.

You will also notice that the Production fact shares the Product, Staff and Calendar dimensions with the Sales fact. When Facts share Dimensions, it’s known as “conformed” dimensions. That is, the facts “conform” to the same set of dimensions. The benefit of conformed dimensions is cross-business process analysis.

For example, the Product and Calendar dimensions shared between the Production and Sales facts mean we can analyze Sales vs Production for any given time period broken down by Product. If Facts share a Dimension, measures of those facts can be analyzed together on the same pivot table when slicing by that Dimension.

The Data Warehouse matrix is a very easy way to communicate the cross-business-process analysis capabilities of a data warehouse.

The matrix can also highlight issues with your design. For example, the matrix above shows the Sales Fact has no relationship with the Account Dimension. Users might have requested to see Sales broken down by Account and compare to current Accounts Receivables for that same Account. The Matrix shows this isn’t possible, and there is a need for an additional relationship between Sales and Account.

Grab your free Data Warehouse Bus matrix template ->

Data Warehouse Development Planning and Progress

You can use the matrix to help plan your project. Generally, you won’t or can’t deliver the entire Data Warehouse at one time. Having a matrix allows you to deliver parts of the Data Warehouse while still maintaining an overall plan that integrates new Dimensions and Facts as they are created.

Initially, you will make decisions about which Facts are going to be the most valuable and deliver those first. But you will have to temper that decision based on complexity, which is partially determined by how many Dimensions a Fact table is related to.

While the project progresses, you can use colour to communicate project progress.

Continuing on from our previous example, the project has completed its first iteration and is currently working on the second. In the image below, the Bus matrix has been color-coded to communicate status.

• Green = done,

• Yellow = in-progress

• Blue = not started.

In iteration one, the project delivered the Sales fact with some of its dimensions (shown in green). In the current iteration, the project is delivering the Production Fact with a single Product dimension (shown in yellow). Since we are delivering the Product dimension, we are also augmenting the Sales fact with the Product dimension. In this case, the Production fact was considered a quick win since the majority of its Dimensions were already delivered.

Using colour-coding is a simple but effective way of quickly communicating overall progress.

Grab your free Data Warehouse Bus matrix template ->

What does a Data Warehouse Matrix communicate?

Depending on the audience, a Data Warehouse matrix can communicate different things. The audience can be broken down into:

• Executives/Project Management

• End Users/Business Analysts

• Developers

Executives/Project Management:

• The scope of the data warehouse.

• The coverage across the value chain and business processes.

• The effort required to complete the Data Warehouse. A Data Warehouse Matrix enhanced with automated estimating can communicate effort to Project Managers for duration and cost Estimation.

• Tracking and communicating progress is possible by adding colour coding to the Data Warehouse matrix.

End Users/Business Analysts:

• How end-user analytics requirements are satisfied by the Data Warehouse.

• How end-user requirements fit within a broader enterprise-wide reporting framework.

• How end-users can analyze a given Fact by a given set of Dimension attributes.

• How conformed dimensions enable integrated reporting across business processes.

Developers:

• A master plan for data warehouse delivery, providing context for current iterations.

• Overall high-level design.

• Progress. Tracking and communicating progress is possible by adding colour coding to the Data Warehouse matrix.

Free Data Warehouse Matrix Template

A semantic layer exists to present data to users as a set of related and commonly understood business entities, terms and metrics.

A semantic layer is typically the “top” layer of a data warehouse/lakehouse. It is accessible to end users and report developers, who use it as the source for reports, dashboards, and ad hoc analysis.

A semantic layer is important because it fosters a shared understanding of information across the business. It supports the autonomous development of consistent reports, analysis and dashboards by end users.

A semantic layer is:

1. First, a business glossary. The basis of a semantic layer is a glossary of related commonly understood business entities, terms and metrics, e.g., The concepts of a Customer, Product, Sale and the relationship between Sales and Products, and Sales and Customers.

2. Represented as a “business view” of data. Physically, the “business glossary” exists as a “business view” of enterprise data. The “business view” is implemented in the semantic layer. The semantic layer is typically the highest/final layer of a data warehouse or data lakehouse implementation. The raw enterprise data undergoes a series of transformations to arrive at the semantic layer’s “business view” format. The most common “business view” format is a dimensional model.

3. Accessible to BI developers and end users. The semantic layer is accessible to end users, data analysts and BI developers. It’s the source of data for reports, ad-hoc analysis and dashboards.

4. Supports Autonomy. A semantic layer supports autonomous access and navigation of the data. It supports self-service BI and ad-hoc (drag-drop) analysis.

5. Fosters a shared understanding. A semantic layer fosters a common understanding of enterprise data amongst all business users.

Why do you need a Semantic Layer?

In large and small businesses today, there are challenges in producing consistent reports.

Without a semantic layer, users, data analysts and BI developers inevitably use data from a variety of sources to develop reports, etc. These reports will often contain conflicting definitions and calculations. These contradictory reports generate a lot of confusion. Business users waste time arguing about these differences with little possibility of resolution. Ultimately, without a semantic layer, the quality and trust in the data on which decisions are made is eroded.

In contrast, semantic layers provide a shared understanding of business terminology and a “single source of truth” for reporting etc. Users, data analysts and BI developers use this “single source of truth” as the source of all reports, etc. This promotes a common business language and understanding and fosters trust, quality, and user collaboration. In addition, it encourages reuse and reduces duplication of effort and waste.

Physically, semantic software in a Data Warehouse or Data Lakehouse architecture provides the following benefits:

1. High-performance aggregated queries. Sub-second response time to aggregated queries over billions of rows.

2. Augments and Enhanced Information. The ability to enhance and simplify the information in the underlying Data Warehouse or Data Lakehouse, including:

   • “Cleaning up the model” to Hide tables, columns, and relationships irrelevant to the business.

   • Adding Hierarchies. Hierarchies enable hierarchical reports, drill-downs and more straightforward navigation.

   • Adding reusable context-aware calculated metrics.

   • Renaming tables and columns if necessary (although your data warehouse, if modelled correctly, should already use the correct naming standard).

3. Unified Data. Semantic models can combine data from multiple sources. Indeed, an end user could enhance the data warehouse with their own data source in the semantic layer.

4. Support for BI tools. Many BI tools natively support connecting to and querying semantic layer software.

5. Context-Aware Security. Restrict data access based on tables, rows, columns and formulas.

The different Types of Semantic Layers

The semantic layer tends to fall into four categories:

1. Data Store or fat semantic layer

The Datastore flavour contains a complete additional copy of the Data. The data is stored in a proprietary, highly compressed and optimised format (usually column store). This flavour comes with an internally optimised vector-based query engine. The Datastore flavour tends to perform better than the Virtual flavour because these data stores are structured to support high-performance aggregated queries. The emphasis is on aggregated. The data store is usually transient and re-loaded periodically from the Data Warehouse. Example technologies include Microsoft Azure Analysis Services, PowerBI data sets, Kyligence, GoodData, Apache Druid and Apache Pinot.

• Advantages:

  • High performance for aggregated queries

  • Easier to define security.

  • It contains functional query languages that make it possible to define complex metrics.

• Disadvantages:

  • Increased complexity and cost to load an additional datastore.

2. Virtual or thin semantic layer

The virtual semantic layer doesn’t store a separate copy of the data (although it might cache data for performance). Instead, the virtual semantic layer contains the logic that defines the semantic model. When a query is executed, the virtual semantic layer acts as a proxy and generates and runs an SQL query against the underlying data source(s). Examples of technologies include Cube, Malloy, LookML, Metriql, AtScale, MetricFlow, Metlo, and Denodo.

• Advantages:

  • Avoids data movement to an additional data store.Lower complexity.Lower cost due to less data storage.

• Disadvantages:

  • Tend to be slower than data store-based semantic layers optimised for aggregated queries. Caching and pre-computation can elevate some of this disparity, but not all; frankly, it moves this solution toward the hybrid option anyway.

3. Hybrid semantic layer

Hybrid flavours support both datastore and virtual modes. Developers can generally define which tables within a semantic model are stored vs virtual. This can help with the trade-offs of performance vs complexity/capacity. In some cases, the data may be so large it is beyond the capacity of the semantic platform to store it. It could be argued that virtual caching is one way to achieve a hybrid model. A better example is Power BI datasets that allow a single semantic model to have a mix of stored vs virtual (direct query) tables.

4. Meta Semantic layer

This is a relatively new development in semantic modelling. Effectively, companies like dbt allow developers to define metrics in a platform-agnostic language. The idea is that dbt can generate the code for a given metric for any supported platform. That’s the idea, at least. Think potentially generating a Power BI dataset or another language for a different platform. This provides portability. It’s very early days, and so far, dbt’s implementation is more focused on supporting its own dbt cloud semantic server. However, the original open-source and open-platform idea persists. Time will tell. dbt Semantic Layer | dbt Developer Hub (getdbt.com)

How are semantic layers implemented?

From a physical point of view, a semantic layer is implemented using specialized software. There are several flavours, which we will expand on in this section.

1. Semantic layer implemented within a BI tool

Modern BI reporting and analysis tools like Power BI, Tableau and Qlik allow data analysts to model a semantic layer directly within a dashboard or report. Some tools will enable you to deploy the semantic model independently from a report and have it act as the semantic layer that many dashboards and reports share.

Ultimately, this approach has issues. An undisciplined proliferation of separate semantic models within dashboards and reports defeats the purpose of a shared semantic layer. What is needed is a disciplined development approach that enforces shared data models.

Power BI has the most comprehensive use case as a semantic layer among the most popular BI tools. While it can support a semantic layer coupled with a report, it can also support a scenario as a stand-alone semantic layer. First, a data set can be published independently from a report and shared by many reports. Power BI premium capacity supports large data sets beyond the typical 10GB up to the capacity size. Power BI has an XMLA endpoint and a REST API that allows external applications to make queries. It enables users to create composite models to supplement data, e.g., a budget Excel spreadsheet combined with actuals from a data warehouse. Power BI supports stored, virtual and hybrid query models. People have even used Power BI as the semantic layer and Tableau as the client via XLMA endpoints. It also has a highly capable functional language, DAX, an internal Vertipaq/tabular data store, and a query engine.

In contrast, Tableau’s strength is its visual aesthetics. It does support some semantic modelling concepts in its data model. It also supports high-performance queries with its Hyper query engine. However, it lacks the concept of publishing a data model as a stand-alone semantic layer. It also lacks the rich analytics language like DAX.

Semantic Layer implemented in a Data Warehouse/Data Lakehouse Architecture

The semantic layer is the final layer of a data warehouse/data lakehouse architecture.

The semantic layer sits between the presentation layer (the gold layer in Data Lakehouse) and the reports/analysis/dashboards. End users can view, navigate, and query the semantic model through a BI tool.

The best practice is to model the presentation/gold layer as a dimensional model. A dimensional model star schema lends itself nicely to semantic layer entities. The core of a dimensional model is business processes, which are modelled as facts with their measures/metrics. The business entities involved in business processes are modelled as dimensions with their descriptive attributes.

Most semantic modelling work and transformation occurs in the presentation/gold layers. The semantic layer should map directly to the Facts and Dimensions in that layer.

So, if the semantic layer looks like the presentation layer, why need a separate semantic layer?

As discussed before, the semantic layer software offers:

1. High-performance aggregated queries.

2. Augments and Enhanced Information. Enhances and simplifies the information in the underlying Data Warehouse or Data Lakehouse. This includes:

   • Hiding tables, columns, and relationships irrelevant to the business, including surrogate keys and management columns.

   • Adding Hierarchies. Hierarchies enable hierarchical reports, drill-downs and more straightforward navigation. Adding reusable context-aware calculated metrics.

   • Renaming tables and columns if necessary (although your data warehouse, if modelled correctly, should already use the correct naming standard).

3. Unified Data. Semantic models can combine data from multiple sources. Indeed, an end user could enhance the data warehouse with their own data source at the semantic layer.

4. Supports BI tools. Many BI tools natively support connecting to and querying semantic layer software.

5. Context-Aware Security. Restrict data access based on tables, rows, columns and formulas.

6. Supports ad hoc analysis. Provides an interface that BI tools use to enable ad-hoc, drag-and-drop, pivotable style data analysis. The data store and query engine behind semantic layer software makes this possible.

Universal Semantic Layer

A universal semantic layer is more about a philosophy or methodology. Personally, I advocate for always building a semantic layer on top of a data warehouse/data lakehouse, and, therefore, don’t endorse the “Universal Semantic Layer” concept.

“Universal Semantic Layer” advocates believe building a data warehouse requires too much effort. They advocate skipping the data warehouse build. Instead, you connect your semantic layer directly to source systems or a raw dataset in a common data store. Then, the semantic layer is used to join and transform that data into the semantic model. When a user accesses the semantic layer, the semantic layer software generates and orchestrates the queries across many sources and joins the disparate result sets to produce the final result.

This approach often lends itself to the “virtual” type of semantic layer software, where the semantic layer is just a thin veneer with a semantic model.

An obvious issue with this approach is performance. The underlying data sources are not specialised in aggregated queries. Joining across source systems often leads to inefficient query plans. The queries can take minutes or hours to return results. To counter this issue, vendors introduce pre-processing and caching, but frankly, this can only achieve so much. Some have even taken to embedding an analytics database behind the virtual semantic model, shifting source data to this analytics database. This sounds a lot like a data warehouse… just worse.

Rather than saving the developer’s effort, it simply shifts it from the data warehouse/data lakehouse (where it belongs) to the semantic layer. The semantic layer is not well suited to implementing transformations. It also shifts the processing workload from batch/overnight data warehouse loads, where the data is transformed and prepared by the data warehouse/data lakehouse for the semantic layer, to query time, when the end user has to wait for the transformation to occur while they run their report.

The semantic layer becomes the sole purveyor of the semantic model. The modern data warehouse is designed to support various usage scenarios besides ad-hoc analysis, reports and BI. Today, the modern data warehouse must support Data Science workloads, non-real-time data integration (i.e. reverse ETL),  near-real-time analytics, etc.  These workloads often require fine-grained and/or high-volume data rather than aggregated data. Semantic layer software excels at aggregated queries but not fine-grained high-volume row-based ones. If you have a Data Warehouse/Data Lakehouse, use it to serve these fine-grained, high-volume, row-based queries. Then, use your semantic layer for what it is best for – serving aggregated analytical queries. Horses for courses!

The History and Future of Semantic Layers

Prepare for a rant!

Unfortunately, the concept of a semantic layer got lost in the vendor-driven, VC-backed “data warehouse is dead”, data analyst-centric collective nightmare of the modern data stack that valued agility over all else! In the melee, we lost the consistency of meaning, reuse and sustainable data assets that data warehouses were designed to deliver.

The problem with data warehouses was never the concept; it was the execution. Back in the ’90s and ’00s, business users were very frustrated by how long it took to create what seemed like a fairly simple report. Couple that with the explosion of data that stressed the state-of-the-art databases at the time, and it’s true that something had to change.

Personally, I believe we should have (and could have – with automation) addressed the methodology of delivering data warehouses. Unfortunately, the data warehouse baby got thrown out with the bath water. Instead, we got “Big Data” (Remember Hadoop, etc.) and Self-Service BI. The “Big Data” hype cycle was driven by the very largest organisations (the Facebooks and Googles of the world) and pumped by the consultancy industry onto organisations that didn’t have the same big data issues. Consultancies love this kind of disruption. It puts bums on seats and $ in the bank. “Big Data” was a monumental flop, but out of the ashes emerged HDFS and Spark. Spark is the open-source distributed computing technology underpinning Databricks, Microsoft Fabric and other analytics platforms. Self-service BI was a myth that gave organisations the excuse to favour agility over sustainability. The result was “data swamps” and a reemergence of the “single version of the truth” problem and other issues that data warehouses were designed to fix.

However, now, finally, things are changing. We are seeing the signs of the industry awakening from its coma. The emergence of the Data Lakehouse architecture acknowledges that the Data Warehouse concept is still valid. The Modern data warehouse, I think, is still overly complicated, but now, rather than being purely reporting/dashboard/analysis focussed, we have a unified platform that can serve more usage scenarios like Data science/analysis, real-time analytics and ETL, operational system analytics APIs, embedded BI and reverse ETL.

Even better news: the industry is recovering from its amnesia around semantic layers. In the 90s and 00s, semantic layers were regularly implemented with software like Microsoft Analysis Services, Business Objects Universe, etc. Unfortunately, semantic layers didn’t fit the narrative and were discounted and “forgotten”. Thankfully, the semantic layer is making a renaissance, as it should, as businesses encounter the reemergence of the problems that data warehouses were designed to solve.

There is new (and existing) software that is focused on implementing a Semantic layer. The emerging software sometimes uses new terms (e.g. Headless BI, Metrics Layer) as synonyms for a symantic layer. Headless BI may be slightly different as it implies that the semantic layer is not delivered within or alongside a visualisation platform.

The software can be divided into roughly 3 groups:

1. Stand-alone tools like Cube, AtScale, Denodo, Kyligence.

2. Embedded within a Visualization platform. E.g. GoodData, PowerBI, LookML.

3. Meta or Virtual layers like the dbt offering. In dbt Core/Cloud, it’s possible to define metrics over your models using a metrics language. Dbt is developing a dbt Server in dbt Cloud that allows a visualisation client to connect in real-time, get metadata about available metrics, and then query the dbt server. The dbt server interprets the query using the metric definitions, generates an SQL query, executes against the underlying database, and returns the result. Other similar tools work in a similar way but have other strengths. Examples are Malloy, Metriql, MetricFlow, and Lightdash.

Semantic later software has needed to evolve from the narrow BI focus of the 90s and 00’s to serve a more diverse set of usage scenarios of the modern data warehouse. A semantic layer must now be accessible for more usage scenarios like Data science/analytics, operational embedded BI, reverse ETL, etc.

Thankfully, nearly all of the above tools have responded to this need. Many provide APIs, SDKs for Python, JDBC/ODBC interfaces, Robust metrics language, etc. Semantic layers are now able to play a wider role in promoting a common business language for end users and data professionals across any usage scenario.

The future for semantic layers is looking bright!

ETL stands for Extract, Transform and Load. It refers to the process of extracting data from the Source systems, transforming it into the star schema format and loading it into the relational Data Warehouse. Development of an ETL process is the major cost in delivering a Business Intelligence Solution. Up to 80% of your cost will be in developing the ETL.

ETL is complex. You could write an entire book about ETL, and several people have including Kimball himself, including the 34 Subsystems of ETL. This section will just provide an overview of ETL processes.

On the surface ETL looks simple. It’s merely extracting data from one data source and inserting it into another. But once you delve in to it, it becomes more difficult with lots of use cases that must be managed. For example, how do you handle the following:

• Inserted, Update, Deleted at Source.

• Late arriving Rows.

• Duplicates.

• Persisting data and keeping an accurate history.

• Heterogeneous Source systems and Connectivity.

• Identifying changed data at source.

• Identifying changed data at target.

• Deleted Rows being reinstated.

• Type 1 and 2 SCD. Type 1 Only, Type 2 Only and Type 1 and 2 mixed Dimensions.

• Schema changes of Source and Target entities.

• Data Quality.

• Full,Partial or Incremental sources and joins across each source.

• Restarting failed processes.

• Deployment to Multiple Environments.

• Scheduling and Orchestrating Batches.

And that’s just the start. It can take several months at least to derive effective ETL patterns.

In addition, ETL techniques are constantly changing. With the advent of the cloud, with a limited “pipe” between on-premise data sources and a cloud based data warehouse, and with different data load techniques targeting new technologies (Massive Parallel Processing Databases, Data Lakes, Big Data), the nature of ETL has changed significantly. ETL that worked on on-premise databases, won’t work for the cloud environment.

ETL vs ELT

Extract Transform/load (ETL) is an integration approach that pulls information from remote sources, transforms it into defined formats and styles, then loads it into databases, data sources, or data warehouses.

Extract/load/transform (ELT) similarly extracts data from one or multiple remote sources, but then loads it into the target data warehouse without any other formatting. The transformation of data, in an ELT process, happens within the target database. ELT asks less of remote sources, requiring only their raw and unprepared data.

ELT is gaining popularity because of the exponential growth of high scale processing power with database platforms themselves, like MPP databases, Big Data Clusters etc. ELT also has the advantage of keeping large amounts of historical unprocessed data on hand ready for the day it may be needed for new analysis.

ETL Process

There are as many ways to design ETL as their are designers. The diagram below describes the ETL and Data stores utilized by Dimodelo Data Warehouse Studio when generating a Data Warehouse solution. This borrows heavily from the Kimball methodology, but also incorporates our learning over many data warehouse implementations, the advent of the persistent staging concept, and the advent of Cloud based Data Warehouse solutions.

Extract

The extract process pulls data from a source system, usually on a nightly basis. Source systems can include Databases, Text Files, Excel spread sheets, or any other kind of source data. The data is written to a Landing data store, typically a local or cloud based file system ready for loading into the Staging/Persistent Layer. Conceptually, because of it’s file based nature, the Landing data store could be the raw data layer of a Data Lake. For the purposes of the Data Warehouse, the Landing Data Store is transient, meaning it doesn’t persist the data between ETL batch runs. Several patterns for Extract can be employed:

• Full Extract. All the data in the source entity is extracted. In this case change data detection occurs during Load.

• Incremental Extract. Only the changed data is extracted from the source. In this case, the Load does not need to do change detection. This method is much faster. However, the source entity must support it. For the pattern to work you need to identify a column or multiple columns that the source system uses to track change. They would usually be a modified date or sequential identifier. The down side of this pattern is that it doesn’t detect hard row deletes form the source. That makes it necessary to run a period Full Extract as well.

• Change Tracking. The change tracking pattern, uses the Change tracking mechanism of the source database (e.g. Change Tracking in Microsoft SQL Server) to identify change in the source entity and to extract only changed data. The Change Tracking pattern has an advantage over the incremental pattern, in that, it does recognize hard deletes, and is therefore a superior pattern if available.

• File Pattern. Extracting data from files is different. There may be multiple files that match a file name pattern. The files also need to be archived after data has been extracted from them. A specialized pattern is required.

• Date Range Pattern. The date range pattern is used to extract a subset of the source data based on a date range. The end of the date range is the current batch execution date. The start of the date range is a number of days prior to the current batch effective date. Each time the ETL runs, the current batch effective date, for that batch execution, is set to midnight of the prior day. The date range ‘window’ moves forward a day as the current batch effective date changes. To use this pattern, its important that dependent entities further up the chain are also working with the same date range.

Load

Load is the process of loading data from the landing data store into the staging data store. The staging data store is typically a database co-located with the Data Warehouse database. It could potentially be the same Database. The staging data store is transient, meaning it doesn’t persist the data between ETL batch runs. In the case of a Persistent target entity, the staging entity is very short lived, only existing until it has been persisted in Persistent data store.

It is necessary to stage data for a number of reasons:

• Staging places less load on source systems. Extract procedures are keep as simple as possible. The next steps, load, persist, transformation etc may require complex queries, that you don’t want to run on the source, mission critical, system.

• If you are combining data from more than 1 source system, then you need to stage data from all those systems before you can combine the data in the transformation step.

• Staging gives the Data Warehouse the opportunity to implement its own change data capture and data quality screening across source systems.

• Staging allows more rapid failure recovery, because the data does not need to be Extracted a second time on recovery.

Persist

A Persistent Data Store, store all change history of the source entities. Technically it is a Bi-temporal database. Every row in a Persistent Entity has a Start and End Effective date. Each row is also associated with the an inserted and last updated batch Ids, that are related to a date.

1. Provides the ability to re-load the data warehouse layer with full history if required (due to a change in logic, model or mistakes).

2. Improves ETL efficiency. It becomes possible to do incremental (delta) loads at all levels.

3. Without Persistent staging, it was difficult to do complex transform logic over the data from incremental extracts into staging tables. These staging tables would, on any given day, only have a small subset of rows from the source table/query. It was impossible to join these tables to other tables to get some other representation of the data that needed more than just the subset that was available.

4. Provides trace-ability and audit-ability of data.

An example of the power of persistent staging is the analysis of the rate of change for scheduled Visits. Scheduling visits across a large complex home care service is complex. This can lead to a lot of schedule and route changes.The persistent layer can capture all these changes and can be used to analyse the rate of schedule “churn”. Schedule churn can be very disruptive and ultimately costly to the organisation. In addition, this schedule churn can lead to questions about “Why did my report change”. The persistent staging layer provides evidence that the source data is changing, and not the report logic, or a problem with the report.

The Persistent ETL takes data from the temporary load data store, generates a delta change set if the data has come from a full extract, then applies that change set to the Persistent entity, inserting new versions of rows and end dating existing rows where necessary.

Transformation

Transformation is the process of transforming the data in either the staging or persistent data stores into the star schema format, and loading it into the Data Warehouse. This is where data staged from multiple source systems is combined into a cohesive set of Facts and Dimensions. There are many techniques required to identify change and implement high performance data loading into the data warehouse. Because it is easy to identify the changed data set from persistent entities, Transform ETL that uses the persistent data store as its source is incremental. Incremental ETL is a design technique for improving ETL throughput.

Semantic Layer

The Semantic layer is the layer/data store that BI tools connect too. The semantic Layer in the Microsoft world is generally an OLAP Cube, a Tabular model, or a PowerBI data set. There are “virtual” semantic layers that pass through queries to the data warehouse. Unless the Data Warehouse is utilizing advanced technology like column stores and MPP, then its likely the queries won’t execute as quickly as if they aren’t cached by the semantic layer. The semantic layer is also transient. It is reloaded from the Data Warehouse on regular intervals. The Semantic layer is a high performance aggregated query engine. The emphasis is on aggregated. The mentioned technologies don’t perform as well on non aggregated queries. Non aggregated queries (like lists of transactions) will perform better when run directly against the data warehouse.

Throughput

ETL throughput is important especially with large quantities of data. Throughput is generally addressed in 3 ways:

1. By Design. For example, Incremental extracts are much faster than Full extracts.

2. Through Parallel processing. By splitting the work of a single process over different compute units (e.g. servers) like an MPP database (Azure Data Warehouse).

3. Concurrency. By running multiple processes (i.e. Extract, Load, Persist, Transform) at the same time for different entities. Your Extract is running on-premise using on-premise compute, your load is running on the data lake using poly-base compute, and the Transform is using Azure DW compute.

What follows is a comparison of the Metadata Driven approach to Data Warehouse development vs the traditional approach taken with an ETL tool.

Introduction to Metadata Driven Data Warehousing

A Metadata Driven Data Warehouse (MDW) tool is a class of ETL Tool, designed to improve developer productivity (amongst other things). It’s a specialized tool dedicated to Data Warehouse development that uses metadata and patterns to deliver data warehouses, combining modelling, ETL and Orchestration. It works at a logical design level, rather than a physical code level. It captures the design of both the Data Warehouse schema and ETL. It combines the captured design with a set of generation templates to create/sync the data warehouse schema and generate ETL code. The generation templates are pre-defined and tested to create consistent, best practice, quality, and comprehensive code.

An ETL tool is a general tool that can be applied to different data scenarios like data Warehouse development, Data Integration, System Interoperability, Data Migration. Being generalist makes it flexible, but this flexibility comes at a cost. It requires more architectural input and time consuming manual development from developers. While many ETL tools have GUI front ends, they are essentially the equivalent of programming code for a specific ETL runtime. They work at the detailed code level rather than a logical design level. Working at this low level is time-consuming. They rely on the skills and experience of the developer to ensure best practice, quality, tested and comprehensive code is developed. A considerable effort needs to be expended on defining coding standards, practices, and patterns and ensuring conformance to standards. Most ETL tools only work with the ETL and not the database schema. This means a coordinated deployment is required to manage the schema and ETL.

Change Propagation

Take a simple example of just changing a column name or data type in a Staging table. This will affect the staging table, the load process that loads the staging table, and any Dimension or Fact transform process that uses the staging table as it’s source.

Using a traditional ETL tool, you need to change each of those code artifacts individually, and then coordinate their deployment to multiple environments. The ETL changes need to be coordinated with the schema changes. This process introduces a lot of effort and potential human error.

With MDW, the modeling and ETL tools are integrated, and the solution is captured in metadata rather than code, any change that is made to the design propagates throughout the solution. Just change the name/type of the column, and regenerate. All code artifacts are simply regenerated to reflect the change and the tool handles the deployment of both the schema change and ETL code eliminating errors.

Portability

Data platforms are continually changing. Change is happening at an ever-increasing pace. Keeping up with that change is difficult. ETL code written today is obsolete in 6 months’ time. ETL gets stuck at a point in time, and usually, the only option is to rewrite it to take advantage of the latest technologies.

Metadata Driven Data Warehousing is different. Because it is not aligned to any one technology, but rather, is captured design at a logical level, moving from one data platform or technology is as simple as taking an existing project and re-targeting to a different platform. The code is then re-generating with a new set of generation templates that target that new platform. The ETL patterns remain largely logically the same, but their physical implementation can be very different, utilizing the strengths of different technologies.

Productivity/Speed

ETL can get complicated quick. Teams can get bogged down in implementation details, and as the body of code grows, having to maintain and refactor existing code becomes a hinderance to progress. Defining ETL at a logical mapping level combined with code generation and dedicated data warehousing tooling that lets you import schema and automates mapping increases productivity by a factor of 3. The technical complexity is encapsulated in the generation templates. Developers work at a higher logical level and don’t get bogged down in writing, deploying, changing and testing ETL code. Change propagation and coordinated deployment mean developers spend more time doing development rather than the admin overhead of releasing code.

Cost

According to research at Wharton University the optimum team size for a complex task requiring coordination is 6…

“if companies are dealing with coordination tasks and motivational issues, and you ask, ‘What is your team size and what is optimal?’ that correlates to a team of six. “Above and beyond five, and you begin to see diminishing motivation,” says Mueller. “After the fifth person, you look for cliques. And the number of people who speak at any one time? That’s harder to manage in a group of five or more.”

It’s a bit more nuanced than that. You can read an article about the research here – https://knowledge.wharton.upenn.edu/article/is-your-team-too-big-too-small-whats-the-right-number-2/

Mostly smaller teams are more productive and, of course, cost less.

With MDW you empower each team member with tools that make each individual far more productive. Combine that tool productivity with the inherent productivity benefits of small teams (a luxury MDW enables) and you have the recipe for a highly productive team.

Consistency

Over time data warehouse teams change. Each developer brings their unique approach to ETL code to solve data problems. This can present challenges when the developer leaves or moves on to other tasks. It becomes difficult for others to interpret and understand their code. In the worst cases, it becomes necessary to re-implement.

With MDW the design is captured in a consistent way, prescribed by the tool and architecture adopted. The entire solution is encapsulated in a single solution, easy to find and follow. Even across projects, there is consistency in the approach.

Not to mention the consistency of the code generated by the generation templates that ensure that a small stable of code patterns are judiciously applied to every generated artifact.

Quality

Code quality is essential to a sustainable data warehouse solution. Does your code cater for unexpected events? Does it take an oversimplified approach (e.g. truncate and reload) because other approaches just take too much effort? Is it tested to ensure that every permutation of insert, update, delete and reinstate are catered for? Is it robust? Can it handle duplicates or intra batch changes for example? Can it handle table changes like new columns, or does this mean a reload is required? Is it keeping history? Does it perform? Can it cater for complex transformations? There is a perception that ETL is easy. Just extract data from one place and load it into another. This is what a lot of vendors would like you to believe. The truth is Data Warehouse ETL is complex and requires careful thought. When you examine all the possible use cases, it becomes even more complicated. This complexity also means manually writing robust ETL can be time-consuming and error-prone, even with well thought out patterns.

The MDW approach is to solve all those complex issues once in the generation template and then apply that to many entities in your project. The template is thoroughly tested to ensure it performs and correctly handles the many use cases. You can have confidence that every generated code artifact has the same high levels of quality.

Responsiveness to Business

“The only constant in life is change”

That is true for data and reporting. Business requirements and source systems are continually changing, coming, going and upgrading. All the benefits of MDW make your Data Warehouse team much more responsive to a changing environment. Without it, the team can be very resistant to change which can result in the business looking for other solutions to their immediate problems.

Focus

What does this all boil down too? Is your Data Warehouse team lost in the technology or focusing on solving business problems? Being business-focussed is key to building a powerful information asset like a data warehouse. Businesses that focus on core business concepts, processes, terms and measures are building a consensus that aids understanding, communication and insight throughout the business. Too often this understanding is concentrated with a few employees and can walk out the door when key employees leave. Having your Data Warehouse team lost in the technical detail, drudgery and churn the ETL, means they have already lost the battle to deliver a long term business-focussed information asset. An MDW lifts the team out of the drudgery and focuses them on the business at hand.