We have four videos already out; subscribe to our channel for future videos!

What is a Data Platform?

Finding Value

Cloud and Data Mesh

Data Processing

We also have the guide in beautifully designed PDF book format.

Firstly, an advert: I’m doing a related talk on “Past, Present and Future of Data Engineering“ in Leeds UK on April 18th, so if you’re in the area, it would be great to see you there!

I know, I know, another “you can’t kill SQL; SQL will be around for another 50 years!” post, but while researching the history of Data Engineering for a talk (see above), I was amused at how many times people have tried to replace SQL and relational databases over their history and had to make a blog about it.

And yes, I know, I’ve interchanged between SQL and relational databases a lot here, despite being two different entities, but the two are closely connected, in my opinion.

QUEL

Interestingly, relational data modelling was created a few years before SQL in 1970 by Edgar Codd and SQL had some stiff competition from QUEL*, which was invented by same person who went on to help create PostgreSQL.

QUEL had it’s own pros and cons, listed here and could be argued to be the better language that was also more suited to Codd’s vision, but SQL received major backing from IBM and Oracle, so QUEL was always fighting an uphill battle.

I also quite like QUEL’s copy syntax; I think it was written years before it would appear in SQL:

copy student(name=c0, comma=d1, age=c0, comma=d1, sex=c0, comma=d1, address=c0, nl=d1)
into "/student.txt" 

*The SQL name was apparently a pun on QUEL: “SEQUEL”.

Object-Oriented Databases

Back in the 1980s, when Object-Oriented Programming (OOP) was the coolest kid on the block, there was an attempt to bring databases into the realm of OOP so programmers didn’t need to learn more than one language. But it became a niche technology, arguably because relational models are more robust for transactional updates*. They sort of still live on in Oracle databases as SQL Object Types and a few niche vendors.

*Note, since it happened in the 1980s, it was hard to research on the internet to say exactly why Object Oriented Databases did not take off…

Subscribe now

Document Databases

In the 2000s, the use of semi-structured data exploded, and for a period of time, querying such data in SQL was a massive pain, giving rise to analytics usage with Document Databases such as MongoDB with MQL.

But SQL databases added JSON type, which made querying semi-structured data easier, and also, data ingestion technologies like Fivetran made ingesting 3rd party APIs into relational tables easier too. ElasticSearch is also still popular in some analytics niches, such as searching logs.

Graph Databases Languages (Cypher, Gremlin, etc.)

Graph databases have the sales pitch of being better in terms of performance and code conciseness for multiple joins than SQL and relational databases (5 to 10 joins). Though I’d argue they’ve stayed fairly niche, as it can be easier to remodel your data to do fewer joins than adopt a whole new database.

-- SQL
SELECT actors.name
FROM actors
 	LEFT JOIN acted_in ON acted_in.actor_id = actors.id
	LEFT JOIN movies ON movies.id = acted_in.movie_id
WHERE movies.title = "The Matrix"

-- Cypher
MATCH (actor:Actor)-[:ACTED_IN]->(movie:Movie {title: 'The Matrix'})
RETURN actor.name

Also, the SQL standard now supports Graph querying, which could limit graph database usage even more for data analytics. There is even serious debate recently about whether graph databases have any point at all.

That said, we could see a serious increase in Graph Database usage as they can add a lot of value to a LLM.

Java

Using Java and/or Scala was quite popular in data analytics in the 2000s for large data transformations (Hadoop), but Hadoop is now dead, and a lot of Hadoop usage ended up being done via SQL anyway using tools such as Hive (which itself has survived much better than the rest of the Hadoop ecosystem, partially due to being, well, SQL).

Java has had more success in real-time streaming and is still a popular language to this day for the use case, but we are also now increasingly seeing SQL being targeted as a primary language for new streaming products.

Why? You can point to Java being a verbose language to write and difficult to learn quickly unless you’re an experienced, expensive programmer, so it's a poor language for ad-hoc queries. I can still see use cases in large, complex streaming transformations, though.

Python

This is the most painful one for me, as five years ago I built everything with pySpark and thought I only needed SQL for ah-hoc queries now and again. I really liked the idea of using one language for Infrastructure, Business Intelligence, streaming, and Data Science. The reality is a bit harder to achieve after some bitter experiences.

One problem with Python is maintainability at large scale, which requires experienced, expensive teams of developers to manage well (though you can argue the same with SQL to be fair). Another bigger problem is migration and re-training, as most new data platforms are doing some kind of migration from a mostly SQL-based platform, and it’s easier to move between dialects of SQL than another language.

Python still has a role in data (we are currently using pySpark for a few clients right now), especially in Data Science, though even there, you can find SQL extensions that allow limited ML/AI capability.

I talk more in depth about Python and SQL here.

Low Code

While low-code is increasingly successful recently in some areas of Data Engineering and Analytics, such as Orchestration, Data Visualisation (Power BI/Tableau) and Data Ingestion, I think low-code data transformations at scale are a poor idea: lock-in, often poor debugging experience, DataOps can be an afterthought, and a more limited community than SQL. That said, I can still use cases for simple transformations for small, low-maturity data teams.

LLMs(?)

This is more of a guess than a history lesson, as there are many companies right now betting on enabling querying via natural language with LLMs. But at the moment, scientific papers suggest querying structured data with LLMs is only 30 to 70% accurate, and that’s only if you attach an expensive Knowledge Graph / Semantic Model to it.

This doesn’t even take into account asking the right questions to solve a business problem, which often requires years of experience in requirements analysis and in-depth knowledge of the domain you’re working in to be good at.

If AI does get good enough to answer vague business questions with near 100% accuracy, then it is also likely clever enough to take over the world, at which point we'll have bigger fish to fry. I still think LLMs have many great use cases, such as data discovery, code generation, and debugging, but I currently see LLMs more as a tool to enhance my SQL coding process than as a replacement.

So why do engineers keep trying to replace SQL?

SQL has become very close to the “universal data language” and it performs four quite different use cases across many different domains:

1. To quickly perform ad-hoc queries on data (DQL)

2. To transform data as part of data pipeline (DML)

3. Defining data models (DML)

4. To administer data access (DCL)

I think SQL does all four tasks well enough, but learning SQL can feel like a lot of effort to non-engineers for just ad-hoc querying, and certain complex and/or dynamic problems are arguably easier to do in an imperative language like Python and Java.

For the first problem, low code can sometimes be better suited in low maturity organisations, and there are some SQL-like languages that can fix the second problem to some extent (Malloy, PRQL), though knowing the past history of SQL, it’s more likely SQL will take the best parts of these languages rather than being replaced.

What I’m getting at is that SQL has become a “good enough” language for a high percentage of data tasks, but there will always be edge cases where other languages make better sense, and the important point is to be aware when those edge cases come up, and you’re not trying to hammer a square peg into a round hole when they do pop up.

The same goes for relational data modelling, which I talk more about in detail in my data modelling article.

Cover Photo by GR Stocks on Unsplash

Sponsored by The Oakland Group, a full service data consultancy. Download our guide or contact us if you want to find out more about how we build Data Platforms!

• Podcast: How Data Platforms affect AI and ML

• Practical Data Modelling

• RAG Using Structured Data: Overview and Important Questions

• Automated dbt Test Generation

• Databases in 2023: A Year in Review

• I spent 6 hours understanding the design principles of BigQuery. Here's what I found.

• Rethinking Stream Processing: Data Exploration

Podcast: How Data Platforms affect AI and ML

While I haven’t had time to write an article this month, I did record a podcast with MLOps Community on how Data Platforms affect AI and ML. There is a video and an audio (Apple, Spotify) version, so pick whatever format suits you.

Practical Data Modelling

Joe Reis, Co-author of one of the most important books in Data Engineering, Fundamentals of Data Engineering, has started a blog on Data Modelling, an area I feel lacks authoritative, up-to-date material, so excited to hear what Joe has to say.

Databases in 2023: A Year in Review

You could argue Andy Pavlo is one of the leading experts in databases: he is, after all, “Associate Professor of Databaseology” at Carnegie Mellon University and CEO of database tuning company Ottertune.

As expected, his thoughts on the latest database technologies and SQL language are highly insightful.

RAG Using Structured Data: Overview & Important Questions

As mentioned above, everyone wants a “Intelligent Data Platform” and to do that probably requires using RAGs to train LLMs on your data.

But how good are RAGs at reading structured data and outputting accurate data? Semih Salihoğlu, CEO of graph database company Kuzu and Associate Professor, reviews the latest scientific literature on the topic to give an overview.

I also highly recommend reading Semih’s sister article on using RAGs with unstructured data and the role of knowledge graphs.

Automated dbt Test Generation

While I use dbt for most of my data quality testing these days, I still miss Great Expectations ability to profile your data for you and generate tests, saving you loads of effort in configuring the test yourself.

Kevin McQuate has released test generation for dbt while it only generates tests for half a dozen types of tests, hopefully it will grow and match Great Expectations profiling capability.

I spent 6 hours understanding the design principles of BigQuery. Here's what I found.

While Google has struggled for market share in the cloud against AWS and Azure to a certain extent, it still loved by many Data Engineers and Analysts, mostly due to having great data products like BigQuery, which more than holds it’s own against the best Warehouses and Lakehouses.

Vu Trinh dives deep into BigQuery internals to find out why.

Rethinking Stream Processing: Data Exploration

How do you explore data that is constantly in motion? Shi Kai Ng, Calvin Tran and Minh Nhat Nguyenv from Grab, a ride-hailing and food delivery app (among many other features) that has hundreds of millions of users in Southeast Asia, try to answer that question.

Sponsored by The Oakland Group, a full-service data consultancy. Download our guide or contact us if you want to find out more about how we build data platforms!

• Reviewing 2023 and Not-So-Bold 2024 Predictions

• How Analytics Can Make a Massive Impact on the Bottom Line

• Understanding Parquet, Iceberg and Data Lakehouses

• Four pitfalls of spatiotemporal data analysis and how to avoid them

• Data Explained: Idempotence

• Is it Time for Composable Data Systems?

• DevOps in Snowflake: How Git and Database Change Management enable a file-based object lifecycle

*To those that follow the Gregorian calendar

Reviewing 2023 and Not-So-Bold 2024 Predictions

It’s that time of the year when loads of experts make bold predictions that 2024 is going to be the year of “X“ (and often X is also conveniently the exact thing the expert is selling to you).

Joe Reis makes an excellent argument about not bothering with any predictions, but new year predictions also give us a chance to reflect on the general state of data and what trends have been appearing. After all, 2024 is just a number, so expect many of the trends of 2023 to keep going into the new year.

So the list below is more of a “list of what is currently happening and will continue to happen in data, according to me“:

• Generative AI had a big breakout in 2023, though I can see a lot of firms ending up in the “trough of disillusionment” in 2024 without a sensible data and AI strategy.

• Relatedly, the phrase “Intelligent Data Platform“ has been thrown around a lot recently, though if I were being cynical, this is yet to mean any major differences to platform architecture beyond a few AI features being integrated into existing Data Platforms components. I’ll likely write a full article on this later.

• Last year didn’t have too much breakthrough innovation in data outside of generative AI to me. Some people might see this as a good thing: hopefully data companies are trying to make their existing products and features better rather than trying to reinvent the wheel every financial quarter.

• Related to the above, there has been a strong trend in the last year or two to figure out how to get a good Return on Investment (ROI) from a Data Platform rather than just building one and hoping it makes everything better.

• Will there be a new game-changing innovation in data in the next year? I don’t know; it’s like guessing when a black swan event will occur.

• Data Mesh hype seems to have died down a bit, though the idea of Data Products is still a popular concept. I think the Data Mesh architecture has a lot of merit but only makes sense for high-maturity, highly distributed organisations that are very data-driven.

• Cloud Data Warehouses and Lakehouses companies are still battling it out, though neither technology has significantly more mindshare than the other, in my opinion. They are also increasingly taking each other’s features, so the lines between them are becoming increasingly blurry.

• Real-time streaming has been slowly growing but struggles to fully replace batch processing in most pipelines. Also, streaming-only products will have to find a way to compete with the “good enough” streaming offered by Databricks and Snowflake.

• Data Modelling will keep making a sort of comeback after being somewhat sidelined by One Big Table models and query-driven analytics.

• Data Governance will keep getting more important as more controls are needed to keep Generative AI usage in check. Though separate data catalogues are still too expensive for most firms. What we are already seeing, on the other hand, is data catalogue features being added to data processing engines, like Starburst Gravity or Unity Catalog.

• In DevOps / Platform, Terraform is still king, though it may have a serious challenge in OpenTofu. Platform Engineering is gaining more traction as a replacement for DevOps, though I expect progress to be slow in the next year as I feels like it’s only needed for the biggest and most mature cloud platforms.

• Data quality is still very important, but like Data Catalogs, buying separate data quality products sometimes feels like a step too far for a lot of organisations, as they want to keep their data stack as simple and/or cheap as possible. Most seem happy with using free libraries or custom code where they can.

So not too many wild surprises above if you already follow existing data trends, but that’s the point: most people don’t like to make big gambles when spending on IT.

Subscribe now

How Analytics Can Make a Massive Impact on the Bottom Line

Ergest Xheblati, Solutions Architect at Wave40, shows how finding constraints in a process via Data Analytics can have major impacts on a company.

I’d also like to note that this article reminds me how important it is to capture your whole business process where possible in data; otherwise, it’s tricky to see how actions taken upstream affect downstream data, processes, and outputs.

Understanding Parquet, Iceberg and Data Lakehouses

Those coming from a Databases and/or Data Warehouse background can struggle with the concept of Data Lakehouses, and David Gomes, Director of Engineering at SingleStore, admits he was one of those people and wrote this informative article to help their understanding.

I’ve also tried my hand at explaining Lakehouses and how they compare to Warehouses and Databases, if you’re looking for another take on the topic.

Four pitfalls of spatiotemporal data analysis and how to avoid them

Analysing data based on time (temporal / time-series) can be tricky, and analysing data on geospatial data can be even more painful, so if you have to analyse data on both dimensions (spatiotemporal), then you could be in for a whole world of pain.

Gregor Aisch, former founder and CTO of Datawrapper, gives tips on how avoid common issues with spatiotemporal data.

Data Explained: Idempotence

Yes, I’m linking to another post by Matt Palmer, but he’s a great writer, and I think impotence is a) a key ingredient for building robust data pipelines and b) a concept that is not talked about enough in data.

Is it Time for Composable Data Systems?

While the last few years have proven there is a market for Data Lakehouses that has the main benefit of decoupling the storage and compute layers, should we go further and decouple the user interface, compute engine, and storage into three different composable layers?

Jordan Volz, Head of Field Engineering at Voltron Data, makes a case for them.

DevOps in Snowflake: How Git and Database Change Management enable a file-based object lifecycle

Vincent Raudszus, Software Engineer at Snowflake walks us through how to add git version control in Snowflake just using SQL commands, which I’ve never seen before (note it is currently only in private preview).

I see other data-processing products adopting this approach in the near future.

Sponsored by The Oakland Group, a full-service data consultancy. Download our guide or contact us if you want to find out more about how we build data platforms!

Photo by Moritz Knöringer on Unsplash

• Microsoft Fabric is Generally Available - Should You Adopt It?

• Investing in Knowledge Graphs Provides Higher Accuracy for LLM-Powered Analytics Systems

• Semantic Layer as the Data Interface for LLMs

• Data Explained: The State of Streaming

• Why You Need a Data Catalog to Build Data Products

• Automatic Data Platform Optimisation

Microsoft Fabric is Generally Available - Should You Adopt It?

So Microsoft Fabric, an all-in-one data platform, has been generally available for a few weeks, and as a Microsoft Partner with five years of personal experience designing and deploying data solutions in Azure, I have opinions.

I had some issues with Fabric when it was announced, and MS, to be fair, has addressed most of my concerns at the time, either in the present or on the future roadmap (which I’ll cover later).

One thing I should mention early on is that I’m asking the question “should you adopt Fabric as a data platform” and not as a Business Intelligence (BI) solution, which is a far easier question to answer as Fabric has adopted most or all of the features of Power BI, which is arguably the most popular BI application on the market right now and has had great success using in the past for clients.

Not quite best-in-class security

The consensus among Oakland engineers is that Fabric is great for proof of concept and for low- to medium-data maturity organisations that are not too fussed about having the best security.

Why? Fabric has no private link support, and most enterprise cloud security teams using Azure mandate them partly because Microsoft says including private links in all data solutions is security best practice, so you’ll likely stick to Synapse or Databricks on Azure in that situation.

Fabric security is still very good, but not the best.

It all depends on data maturity of the organisation

As I mentioned above, Fabric is a great option for low-data-maturity organisations, as it’s very easy to adopt if you already have Microsoft, Office 365, or Azure accounts. This may explain why I’ve seen lots of excitement among Power BI and Platform users as Fabric massively expands their data capabilities without having to learn cloud infrastructure.

But also, I’ve seen a lot of negative opinions from experienced data engineers for Fabric due to its focus on no- and low-code. Most experienced data engineers just want to write in SQL and/or Python (including me) and not be limited by their tooling, and Fabric doesn’t allow that in Pipelines and real-time analytics (yet).

For example, Microsoft has been spending a lot of time adding features to their low-code data flow and pipelines, whereas I would prefer if they added more support for managed Airflow or something similar to bring it closer in line with GCP or AWS.

This is also the second Microsoft data product refresh in a few years, and Azure data engineers are weary of having to adapt to a new suite of products rather than see existing products that haven’t been out very long (Synapse) catch up with the competition (Databricks, Snowflake, etc.).

Also note that there isn’t full automation support right now, so platform admins might be tearing their hair out over the prospect of managing Fabric on a large scale.

Roadmap

Though Private link are on the roadmap for 2024, and speaking of the Fabric roadmap, I’m quite excited about what’s in there:

• Better Data Factory / Pipelines git integration

  • The current iteration converts everything JSON at the moment, which is awful to review in pull requests and merging

  • It also has issues when using Infrastructure as Code (Terraform, Bicep, etc.)

• SQL for real-time analytics

• I’m hoping “On-premises data gateway (OPDG)“ means we don’t have to build another virtual machine for Pipelines / Data Factory to connect to on-premise again.

• Better automation support everywhere (SDK, REST API)

This feels a bit “jam tomorrow”: I’ve been burned in the past by great-sounding features turning out to be rubbish or never arriving, but I’m cautiously optimistic about Fabric’s future.

Subscribe now

Investing in Knowledge Graphs Provides Higher Accuracy for LLM-Powered Analytics Systems

As you can see in the table below, Knowledge Graphs can possibly provide massive benefits to text-to-SQL query accuracy:

What is interesting about this paper is that it was used on a somewhat realistic 13-table schema* based on the insurance industry rather than some noddy one-table dataset. Major kudos to the authors for putting in the effort and money to build this.

That said, it only gets 71% accuracy on simple results, which says to me that LLMs require an expert to second-check any outputs, so I think we’re still a bit away from a CFO just using text prompts to get analytical insights.

Even with the results above, I’m not sure it’s still worthwhile to use Knowledge Graphs with LLMs unless it’s a large-scale solution: you’ll have to build and maintain two data analytics systems rather than typically just one, and I can tell you, as someone who has designed and/or built more than half a dozen data platforms in the last 6 years, building just one analytical system is hard enough work, even if you can use SaaS solutions and the cloud.

Also, on top of that, you have to keep Knowledge Graph in sync with your relational database, which exponentially increases your maintenance problems (for example, you now have two sources of truth, not one).

*I know 13 tables is a bit on the small side for a large enterprise dataset, but it’s a much more complex dataset used to test LLMs than I’ve seen elsewhere.

Semantic Layer as the Data Interface for LLMs

But maybe semantic layers are actually better with LLMs than Knowledge Graphs? Jason Ganz, Developer Experience at dbt Labs, presents their promising initial findings on using dbt semantic layer with LLMs.

One issue dbt is going to face with this is that the two biggest developers of LLMs, Microsoft and Google, also sort of have a semantic layer in Power BI and Looker, respectively. Though dbt might have an angle here for organisations not looking to be locked into only one LLM stack or BI application.

Data Explained: The State of Streaming

I’ve mentioned quite a few articles on streaming and/or real-time data in this newsletter, but I don’t think I’ve shared an introduction to streaming here, and Data Engineer Matt Palmer does one of the best intros I’ve seen in this article.

I’d also generally recommend following Matt Palmer, especially if you like Jujutsu Kaisen references in your data engineering articles.

Why You Need a Data Catalog to Build Data Products

Last week, I was speaking to a data governance professional in Oakland about how it is difficult to get anyone outside of data governance to care about the topic.

It probably explains why some in data governance use comics or music to lower the bar of entry for non-experts.

Hugo Lu, Co-Founder and CEO of Orchestra, tries well-written prose to make the case for data teams to adopt data catalogs, I especially liked this sentence: “Catalogs offer a way for data practitioners to finally collaborate with business users effectively.“

Automatic Data Platform Optimization

With rising prices in both on-premise hardware and the cloud, there has been a lot of noise in the data world about reducing costs on data platforms.

That said, I haven’t seen many articles on comparing vendors that try to save you money on data storage and compute, so I love the article by Phil Dakin here.

Sponsored by The Oakland Group, a full-service data consultancy. Download our guide or contact us if you want to find out more about how we build data platforms!

• How Self-Service Analytics Impact Data Platforms

• Data Pipeline Orchestrators - The Emerging Force in the Modern Data Stack (MDS)?

• When to Build or Kill Your Data Product Ideas

• From Data Platform to ML Platform

• dbt Shows Off New Features For its Cloud Service

How Self-Service Analytics Impact Data Platforms

I believe most decisions made in data should ideally be on a continuum of two or more options or be a compromise rather than picking a binary extreme.

And I think Self-Service vs. Guided Analytics (pre-built reports) is one of those examples, as in a sufficiently large organisation, you are going to see a mix of both approaches.

Now I’m not going to make a typical comparison of the two, as there are already a number of great articles on that. Also, the comparison articles can’t cover all the numerous use cases and personas of analytics (see appendix below).

The trick is to find the right balance for your organisation based on your current analytics use cases. This balance will differ in each organisation and, most importantly, differ over time as your organisation becomes more data-informed, new technology comes on the scene, and data analytics use cases change.

Self-Service is Technically Everywhere

While the dream of some to have 100% guided analytics, as it’s fully governed, requires less data literacy training and documentation, the reality is that there is always a requirement for some custom ad-hoc analysis.

I also believe pretty much every organisation is doing self-service analytics; just all are not aware of it. If you’re using Excel for analytics, you’re doing self-service analytics!

And with Excel, you are doing ungoverned self-service analytics, as Excel doesn’t have much in the way of query logging, data lineage and version control features out of the box.

But self-service done right requires more upfront costs than guided analytics: you need the right tools, governance, and training in place so self-service doesn’t turn into a report-dumping ground.

But What Does This Have To Do With Data Platforms?

As I mentioned in my guide, ideally, a data platform should be designed backwards from it’s analytical use cases so we know the right tools to choose for our platform and how best to use them. How you do analytics can have a massive impact on your data platform.

Increased self-service use cases will impact platform design in a number of ways:

• Modelling: data models will require a more adaptable core data model

• Performance and Cost: self-service modelling will probably put more strain on your warehouse or lakehouse. Rather than having static indexes and partitions, you may want to look at more dynamic alternatives (Starburst, Databricks and Snowflake) based on recent usage.

• Data Governance: will be harder to manage with self-service, so it requires more platform processes and tooling to manage.

• Tooling choices: low-code and no-code tooling can increase self-service, but may come with a lack of governance features like version control.

Summary

The above considerations might put you off self-service, but as I mentioned before, it’s nearly impossible for some self-service not to occur, so it’s best to plan to govern it, plus it can give you business performance benefits such as a lower time to insights.

Likewise, I can’t see a fully self-service approach working for most organisations, as it will simply cost too much to train everyone to be a data analytics wizard, even if all staff want to be one (unlikely).

So it’s about finding the right balance for you right now and building the best data platform for that balance.

Appendix: Analytics Personas

While this didn’t fit well into my post above, I wanted to give you an idea of the range of use cases that fit into guided analytics and self-service by looking at the types of end users in analytics:

• Report viewers:

  • Require relatively lower data literacy training.

  • Very much a guided analytics use case.

  • Doesn’t have to be static reports, though the more filters you add to a report, the more training required for it

• Report Builders:

  • Now into the domain of self-service analytics, building ad-hoc experiment reports or production-ready reports for report viewers

  • Usually exclusively using BI tools such as Power BI or Tableau.

  • Should be trained in data tooling they are using and data literacy.

• Report Modellers:

  • Still often using BI tools, much also likely comfortable using data warehouses and lakehouses with knowledge of SQL

  • Will also often build the reports too.

  • They require a high level of data literacy: at least the basics of data architecture.

  • Likely data specialists with some domain expertise.

• Analytics Engineers:

  • Designing their own data pipelines with tooling like dbt.

  • Very quick time to insights, but can be hard to govern at scale.

  • Likely data specialists with high data literacy.

• Data Scientists:

  • Will often require access to conformed and even raw data.

  • Will likely still need training on how best to communicate their AI and ML models to non-scientists.

  • May design models that can be accessed in a self-service manner for investigation.

Subscribe now

Data Pipeline Orchestrators - The Emerging Force in the Modern Data Stack (MDS)?

Timo Decau, Chief Content Maker at Deepskydata, often writes great posts, and this is another: pointing out that Data Pipeline Orchestrators offer a common layer for various Data Platform / Modern Data Stack tooling similar to a fully integrated data platform.

He also does a deep dive comparsion into various Data Pipeline Orchrestrators and wonders if they’ll exist in 10 years time.

When to Build or Kill Your Data Product Ideas

There are lots of posts on LinkedIn telling you that your analytics “must have business value“, which I agree with, but how?

Stanislav Dmitriev, Head of Marketing at ellie.ai gives a nice intro into how to figure out which data product ideas should be built or not.

From Data Platform to ML Platform

This is a great (and accidental!) companion article to my article on why data foundations matter in AI and ML, as this looks at how to start with the data foundations of a typical data platform and add AI and ML functionality with a deep dive into MLOps.

Written by Ming Gao, Tech Lead Manager at Bytedance (Tiktok)

Don’t Use DISTINCT as a “Join-Fixer”

Great deep dive into the performance impact of DISTINCT SQL function by Aaron Bertrand at Staff Database Reliability Engineer for Stack Overflow.

However, Snowflake processes DISTINCT differently, so this advice doesn’t apply to all databases!

dbt Shows Off New Features For its Cloud Service

While I love open source dbt (or “dbt core”), I’ve always struggled to see much value in dbt cloud, especially at it’s current pricing. For example, I don’t think most organisations need a semantic layer, as they likely have one in their Business Intelligence (BI) applications, and it’s orchestration features are limited compared to best-in-class orchestration libraries.

But dbt mesh and dbt explorer sound interesting and could add value to organisations that have hundreds or even thousands of models across many data products.

Though note, there is already work in the open source community to replicate some of the dbt mesh’s features with dbt loom, Data Engineer Christophe Blefari shows off a working example of it.

Sponsored by The Oakland Group, a full-service data consultancy. Download our guide or contact us if you want to find out more about how we build data platforms!

Cover Photo by Stephen Phillips - Hostreviews.co.uk on Unsplash

• How Data Platform Foundations Impact AI and ML Applications

• Current 2023 Conference Retrospectives

• What is the Environmental Impact of Your Data? (Sponsored)

• Beware of dbt Tests…. Your Money May Disappear

• Pulumi Environments, Secrets, and Configuration

• Why data integration will never be fully solved

How Data Platform Foundations Impact AI and ML Applications

I’ve always told my clients and colleagues that traditional rule-based software is difficult, but software containing Artificial Intelligence (AI) and/or Machine Learning (ML)* is even more difficult, sometimes impossible.

Why is this the case? Well, software is difficult because it’s like flying a plane while building it at the same time, but because AI and ML make rules on the fly based on various factors like training data, it’s like trying to build a plane in flight, but some parts of the plane will be designed by a machine, and you have little idea what that is going to look like till the machine finishes.

This double goes for more cutting-edge AI models like GPT, where only the creators of the software have a vague idea of what it will output.

This makes software with AI / ML more of a scientific experiment than engineering, which is going to make your project manager lose their mind when you have little idea how long a task is going to take.

But what will make everyone’s lives easier is having solid data foundations to work from. Learn to walk before running.

This isn’t a new concept; it’s just more relevant in a new AI hype cycle. Previous AI and ML hype cycles gave birth to the idea that good AI, ML, and advanced analytics are best made on top of high-quality, well-modeled, and governed data:

The above pyramid is from an article in 2017 and could arguably be said to be a reworking of the DKIW pyramid, which can trace it’s origins back to the 1920s.

So wisdom, making the best decisions, is based on learning from past decisions, which in this context means you’ll have great decision-making capability using AI and ML if you invest in your historical data processing beforehand.

But you probably did not click the link to this post for a history lesson; you want to make great AI and ML applications and want some tips to make that happen. So let’s get started. What platform foundations do you need?

Data Quality

• If half your training data has wrong or no value, then it is likely to be half as accurate.

• Studies show you’ll often get more accurate forecasts by increasing Data Quality than by using a bigger and more expensive model.

• Also Data Quality tests tell Data Scientists what data is useable and inform them that data has drifted, potentially saving time and money in finding that out.

• I cover Data Quality more in depth here.

Data Modelling

• As the grand sage of data, Joe Reis, says, Data Modelling is more important than ever in the age of Gen AI, as you cannot use Gen AI on your own data well without sufficient modelling to make it consistent, understandable, and easily readable.

• Also, if done carefully, clean, deduped, and conformed data can save weeks or even months of work for Data Scientists.

• Preferably, you’ll want data not to be overwritten when conformed and curated; forecasting looks at all changes over time, otherwise, if there are any important changes missing due overwritten data, you are going to have less accurate forecasts

• Also, using aggregated and sometimes anonymized data can make your models less accurate, as you have less training data to work with.

• You might think then, “I’ll go straight to the raw data”. Oh, sweet summer child, expect to lose months of time trying to replicate work already done for Business Intelligence (BI) to clean data and build business-critical metrics.

• Try to weigh the trade-offs here between using raw and curated data for Data Science.

  • The curated layer will have less data to train on, but raw data will likely be more messy, harder to access, and have no business rules.

  • If you are heavily into ML/AI, you may want to invest in an append-only conformed middle layer of data (Data Vault, Activity Schema) if you haven’t already.

• Classic AI/ML models generally prefer One Big Table (OBT) data models, which often expect one big table (or matrix) of features and target values. Though OBT models have their weaknesses, it’s not uncommon to use multiple model types. I cover this in more detail in my Data Modelling post.

  • Though Generative AI like GPT can take almost any data model, but smaller and cleaner model will likely give you more predictable results.

Data Governance

• It’s easier to find the right or best data if it’s catalogued! Even a Data Dictionary of most organisational data in a spreadsheet will save days, if not weeks, and months of time for Data Scientists.

• All applications involving data often have some legal and/or ethical concerns you need to be aware of. This is doubly true for most AI and ML applications, as they often make decisions that will impact people's lives. An experienced Data Governance team can often help you navigate these constantly changing legal waters so you don’t get hefty fines from your government.

• If decision-making has large impact, you need to document your work

  • Black box models may be banned if you can’t show the ML or AI model how you came to a decision, which might be illegal or very risky in high-impact sectors (medicine).

• I go into more depth on Data Governance here.

MLOps

• While MLOps can use some different techniques and tools, you can save time if you already have a solid base of DataOps and/or DevOps expertise, services, and tools to rely on.

• You’ll also want to track if you’re product is as efficient and trustworthy as possible, which is where DORA metrics like “time taken to complete a change“, “how many changes cause failures in production“ can help.

• I cover DataOps in more detail here.

Business and Cultural Change

• Have you time and budget to train people on the application? Have you communicated how the application will change the organisation?

• How you use historical compared to AI/ML-created forecasting data requires different thinking for everyone that uses the application

  • It requires more “scientific” data analysis; how confident is the forecasting?

Cautionary Note

Having solid data foundations does not make building AI and ML applications easy - just a lot easier. You might have difficulty with having great data but not enough of it to train accurate enough models, for example.

Also, having a platform already in place does not mean no more modelling, re-architecture, and cataloguing needs to be done for AI and ML, but you should have to do far less of it if you already have a solid base.

Summary

There are a lot of elements to consider above, and I didn’t cover all the topics needed to deliver any successful AI/ML application, like working out the best business value or strategy, security implications, and effective team delivery.

Though if you have these foundations in place, typically found in a Data Platform, it can save you a lot of work on AI and ML applications and likely reduce your chance of failure too. It will also make your organisation better at reporting historical data, making it, in general, a more data-informed organisation.

Finally, I ask those exploring AI and ML projects to understand that while AI can bring big transformational change, be wary of buying into the hype too much: most of the AI and ML projects I’ve been on have been turned into more traditional rule-based projects, as we found out we only need cheaper rule-based software than a fancy, expensive model once digging into the requirements and data.

*There's an argument that most AI is just ML, which I’m somewhat partial to but is a well-covered topic elsewhere, so I’ll call it AI/ML and not digress here.

Subscribe now

Current 2023 Conference Retrospectives

Current is a streaming/real-time conference hosted by Confluent, which sells arguably the most popular cloud-independent managed streaming product, based on open source Apache Kafka.

Because Confluent allows any vendor who is related to streaming to attend their event, even their competitors, it is probably the biggest streaming focus conference of the year.

The two big themes were:

• Apache Flink, an open-source combined batch and streaming data analytics library that supports SQL, Python and Java APIs

• Streaming Databases

Apache Flink

Apache Flink adoption has been growing quickly, and now Confluent has brought out a managed, serverless version of it with a demo at the conference. Yaroslav Tkachenko noted that it was disappointing that you could only use it in Confluent Cloud with no bring your own cloud option.

As mentioned in this newsletter before, Flink could soon become a common element in the data stack, with many vendors like AWS, Aiven and Ververica offering enterprise-ready versions of it now.

It will be interesting if we see other big vendors (Microsoft, Google, etc.) jump on the Flink bandwagon too.

Though it does suffer from some headwinds, it often requires Kafka to ingest data into it, and the cost of running both Kafka and Flink at the same time will not be cheap. Also, Databricks and Snowflake both offer streaming analytics now, which looks like an easier option for those not needing streaming first libraries like Flink or super low (sub-second) latencies.

Also, even though Flink has a SQL API, I don’t think it’s really suited for use in Data Analysis: it doesn’t have direct connectors for Power BI, Tableau, etc., and it doesn’t have much support for authentication and authorization options in the open source version. It’s more designed like Spark: only for transforming data, not serving it as well like a Lakehouse/Warehouse, so it requires pushing data elsewhere, likely via the Kafka connector.

Streaming Databases

While I think Streaming Database suffers from the similar headwinds as Flink, we are seeing a lot of new companies and investment in this sector, so this is one area to watch over the next few years.

Hubert Delay, author of Streaming Data Mesh and currently co-writing a book on the topic, explores the topic more.

What is the Environmental Impact of Your Data? (Sponsored)

GreenOps is a phrase already thrown around a lot, but expect to hear more of it: climate change is having a bigger impact on our lives every day, and governments are getting tighter on their emissions regulations.

My colleague Luke Sharma, who is Oakland’s resident expert on GreenOps and Green Data Strategies, gives tips and guidance on how to work towards a mature GreenOps process in your Data Strategy.

Beware of dbt Tests…. Your Money May Disappear

Paul Marcombes, Head of Data at Nickel, looks at the hidden costs of dbt tests, though I think they can apply to most kinds of Data Quality tests:

Data Quality is of course important, and keeping costs low is important, but Data Quality tests rarely come for free. So what do you do?

The difficult answer is to not run a complete set of tests and only run tests that bring the most value. It is a difficult answer, as there’s no silver bullet but a tradeoff between creating trustworthy data and staying on budget.

One thing worth also considering is looking at SQL constraints to replace some of your testing, which are sometimes called “free Data Quality tests“.

Finally, have a look at Data Quality libraries and products, as they can design their software to run their tests more efficiently than most custom code. I spoke to Soda CTO Tom Baeyens about this topic in our interview.

Pulumi Environments, Secrets, and Configuration

Managing environments and secrets can be a pain on any reasonably sized platform, as you’ll be storing configuration for multiple cloud accounts, not to mention 3rd party vendors as well.

But Pulumi, an Infrastructure as Code (IaC) and now Platform Engineering product, is now offering a feature to store all configurations and secrets in one place.

While Pulumi has a lot of work to do to knock Terraform from the IaC throne, I would seriously consider it for large, complex platforms.

Why data integration will never be fully solved

Very much onboard with this article, as no-code or low-code data integration solutions like Fivetran can get you far, but often not all the way, if you can afford them.

Anna Geller, Product Lead at Kestra, looks at open-source alternatives and their own tradeoffs.

Sponsored by The Oakland Group, a full-service data consultancy. Download our guide or contact us if you want to find out more about how we build Data Platforms!

Introduction

Unless ChatGPT manages to figure out how to automatically make Data Platforms from a text prompt, you’ll need people to build your Data Platform.

More than that, you’ll need a team of people that can work well together to deliver your Data Platform successfully. So first we ask, what roles do we need to build a Data Platform?

So, What Roles are often involved in building a Data Platform?

• Data Engineers and/or Analytics Engineers

• Cloud and/or Platform Engineers

• Data Architects

• Solution Architects

• Data Analysts

• Data Scientists and / or Machine Learning Engineers

• IT Administrators

• Support Engineers

• Security Experts

• Business Analysts

• Subject Matter Experts (SME)

• Product or Project Manager

• Budget Holder, Data Owner, and/or Product Owner

• Senior or Executive Management

That’s a lot of roles! Note that most of the above will only work for days or even hours on the Data Platform, but they are vitally important none the less.

Also, many roles will be filled by the same person; in fact, we recommend generalists over specialists to make your data teams more resilient to business and staff changes. Though beware of overextending this; if a person wears to many hats, they’ll more likely get burned out from all the context switching.

As you may of guessed from the above, building a successful Data Platform requires a lot of teamwork and collaboration, so the focus of this section is on how to get people working as a cohesive, engaged team to deliver a Data Platform or Product, in a reliable but efficient manner.

So what guidelines are there for effective software delivery?

Delivery

There are many frameworks for delivering software: Waterfall, SAFe, Scrum, Kanban and lots more. I (and my employer, Oakland) do not have “one delivery method to rule” unlike other consultancies, but prefer picking the best framework for the project, product, or organisation at that point in time.

There is also nothing wrong with changing frameworks to suit your needs, say, from fast-build Proof of Concepts (PoCs) to slower, more audited production changes, other than the time taken to change.

Also, feel free to adapt the delivery to your needs; a delivery framework is just guidance, not a set of instructions. Though do note that your organisation may mandate a delivery method you have to adhere to.

The one required rule of agile to remember is to regularly reflect on work done to see how your team can improve in the future. Most agile framework diagrams are in a cycle, to reflect the commonly held wisdom that the most effective delivery is one that constantly adapts and learns from its experience.

Effective delivery also requires automation of common tasks, which we cover in the DataOps section. Using metrics from DataOps as well as team engagement can help you quantitatively work out if the changes you’re making are having a positive impact.

I would argue the most important organisational unit when it comes to delivery is the delivery team. So, assess people based on how they contribute to the team, not on individual metrics like lines of code written.

But what does a good data team look like?

Designing Teams

Ideally, aim for three to eight people per team:

• Avoid large teams (more than eight people) where possible. If a Data Platform build requires more than eight people, split up the teams to focus on an aspect of the platform (more on scaling teams later on).

  • It is generally too hard for a large team to keep track of everything at a deep and meaningful level, leading to increased levels of burnout and miscommunications.

  • Plus, no one likes to be in a 20 person daily standup that takes an hour and 90% of it’s content has little direct relevance to them.

• On the other side, teams of one to two people long term can make for lonely experiences and aren’t very resilient if one person goes off sick or gets pulled away to fix organisational emergencies, for example.

Try to split teams across Business Domain and then Products or Projects rather than role types; ideally, you want your team to design, build, and maintain an entire data solution together from start (source data) to end (reports) and not have to hand it off to other teams where information and context get lost.

As the number of teams grow, you may add cross-organizational services like monitoring and malware detection, managed by other teams (usually platforms).

We do note, however, that changing organisation structure is often very hard, if not impossible, in the short and medium term, so if you can’t change your team structure, be aware of it’s limitations and plan appropriately. In fact, changing teams too much is an anti-pattern; it usually takes a few months for a new team to reach maximum efficiency once they learn how to work together in the most effective way.

Communication and Meetings

If you speak to most engineers, they’ll complain that they are into many meetings; however, Project Managers often complain that they don’t get enough updates from engineers!

So there is a gap that must be bridged: making sure engineers are not attending unnecessary meetings, but also making sure engineers are updating progress on at least a daily basis so the project budget holders know we are making progress on a project while spending their money.

For me personally, I try to lean towards overcommunicating. I think the risk of colleagues taking the wrong action because they missed an important update is higher than the risk of colleagues getting annoyed because I’m communicating too much.

Scaling Data Team(s): from one person to many teams

1. Your first data hire(s) should normally be a Data Analyst or Analytics Engineer: focus on the data outputs or outcomes first. You’ll likely want to bring in project management and business analysis experts, even if they are only part-time.

2. After you’ve hired a few Analysts, you may want to apply some software engineering and/or manage data at scale, so you’ll likely hire some Data Engineers.

   1. Roughly here is where you think about building data platforms.

3. Once your team grows beyond approx. eight people, look to split them up into two teams, each with a manager, both likely reporting to a “Head of Data” or similar title.

4. Once you’ve hired enough people to make more than four to eight teams, you may need to add another layer of management.

5. Try to split the teams along Data Products (or, next best, projects); the easiest way is to have one team = one Data Product.

   1. Multiple teams on one product increase the chance of handoffs and blockers, which increases blocking. Look at splitting up the product if possible.

   2. On the other hand, if one team is working on multiple products, it increases cognitive load, which leads to more burnout and expensive context switching. Teams are usually fine with two to three simple products and rarely more than one complex product.

6. As you grow in the number of Data Products, you’ll want to provide them with consistent cross-organisational services such as (Cloud) Platform Engineering, Data Governance, Monitoring, etc. to reduce repetition.

Summary

A reasonably big section, but only touches the surface of effectively running a data team. I haven’t covered:

• In person vs. remote working

• Comparison of delivery methodologies

• Office design

• Delivery estimation

• Setting personal and team goals (though I have a section on product goals).

• Documentation (though I have a section on documenting architectural decisions)

Feel free to reach out in the comments or in a private message on LinkedIn if you have any questions or suggestions!

Finally, here is some research into this topic that has been a major help to me over the years and that I recommend to others:

• An Elegant Puzzle: Systems of Engineering Management by Will Larson

• Team Topologies by Matthew Skelton and Manuel Pais

• What Your Mother Never Told You About Agile Development by Aino Vonge Corry

I’d also like to thank Hannah Varley-Fodden and Lynne O'Donnell for reviewing this post and providing valuable feedback.

Sponsored by The Oakland Group, a full service data consultancy. Download our guide or contact us if you want to find out more about how we build Data Platforms!

Cover Photo by Nick Fewings on Unsplash

• The Fancy Data Stack - Batch Version

• Data Contracts at GoCardless - 6 Months On

• Not My Circus, Not My Monkeys Newsletter

• Is Data Mesh Only for Analytical Data?

• Are You Using Elementary for DBT?

• Principles of Data Layers in Data Platform

• Data - The Land DevOps Forgot

I will also point out that my “How to Build a Data Platform“ guide is almost done, with only one article to publish that should go out next week. So click the link above if you’ve recently subscribed and haven’t already seen them.

Also, leave a comment if I’ve missed any major elements of building a Data Platform, as the long-term plan is to go back and update the guide over time.

The Fancy Data Stack - Batch Version

This is a great idea by Christophe Blefari, a Senior Data Engineer, who presents his “ideal” data stack with few constraints. I find his choices and reasoning to be on point, though I will say every organisation is different, so it will have different “ideal” stacks.

Data Contracts at GoCardless - 6 Months On

Andrew Jones, author of a book on Data Contracts, which has many great reviews, has written a retrospective on implementing Data Contracts at GoCardless, an online payment processor with $30 billion in transactions in 2022.

Not My Circus, Not My Monkeys Newsletter

Site Reliability Engineering Coach and ex-Colleague of mine, Mark Ellens, has started regularly posting great insights into Monitoring, DevOps and Testing for the last few months, based on his vast experience of building and maintaining modern, large software systems.

It also often crosses paths into data often (what doesn’t! ;D). Highly recommended!

Subscribe now

Is Data Mesh Only for Analytical Data?

Piethein Strengholt, author of Data Management at Scale (see my glowing review here), has another great article.

I’ll digress here by saying the first draft of my article on Data Mesh vs. Fabric and Centralised Architectures did include how a Data Mesh can, in theory, mix both operational and analytics system outputs where it makes sense, but I wimped out and took it out. I wish I left it in now!

From my experience, I’d still setup separate deployment areas (Azure Resource Groups) for security and DevOps reasons, but the outputs can be combined if there is a good business requirement for it and ideally have the same data owner(s).

Are You Using Elementary for DBT?

Lead Data Engineer at New Relic, Leo Godin, presents a convincing argument for Elementary if you want more observability, alerting, and testing tools in your dbt transformations.

Principles of Data Layers in Data Platform

Siva Ilango, Principal Data Architect at JMAN Group, has put together a great diagram of Data Modelling layers and what kind of data model best suits that layer:

The article is also a great read, with lots of great Data Architecture advice.

Data - The Land DevOps Forgot

Great video from Michael Nygard, VP of Data Engineering at a large Latin American bank, Nubank, on how data comes with extra issues with integrating DevOps and how Data Mesh can help.

It also covers a lot of real-world pain issues with Data Meshes too, so far away from another Data Mesh sales pitch.

We’re Now Starting to See Major Companies Implementing DuckDB as their Data Warehouse

In this case, Okta. While I don’t think many enterprises will consider DuckDB, it is definitely worth keeping tabs on it to see if a lot of more tech-centric companies adopt it and if it becomes a game changer for them.

Sponsored by The Oakland Group, a full-service data consultancy. Download our guide or contact us if you want to find out more about how we build Data Platforms!

Photo by Anna Dziubinska on Unsplash

I’ve worn a number of hats in my career, but Data Governance isn’t one of them, so I can’t claim to be an expert.

That said, I have implemented governance controls on many Data Platforms, have opinions about it and don’t think I can talk about “How to Build a Data Platform” without mentioning Data Governance in good conscience, as I feel all data solutions benefit from some governance.

See this as an “Architects or Engineers guide to Data Governance” rather than any deep dive into Data Governance, for that, I’d check out my colleagues expert in-depth guide.

Why is Data Governance important?

Data Governance is often the last thing on an engineer’s or analyst’s mind while being swamped with stakeholder requests that needed to be answered yesterday.

But it’s likely they got into that situation partly because of a lack of governance:

• Stakeholders can’t discover data on their own will instead ask an overworked data team.

• Lack of data documentation inside the data team, which increases time to insight and generally making the data team miserable because everything is harder to do than it should be.

• Unclear data access specifications mean it takes weeks, months or even years to get access to data that should only take hours to get access to.

• There is no clear data ownership on data assets, so data access requests are a slow or impossible task and a lack of updates to data means it becomes a legacy dataset.

• Little or no usage metrics are collected on data assets, which means investment in data decisions requires more effort and is more likely to be spent incorrectly.

• Not enough Data Quality, so the data requires more work to do analysis on.

I would argue data with a lack of governance is like software technical debt, so in this case data debt, okay in small amounts, but can soon build up enough to become a millstone around the data team, if not the whole organisation, making everyone more inefficient.

I haven’t even mentioned the security and legal compliance aspects of Data Governance: if you collect personal data and have improper governance, you are at risk of embarrassing and costly data breaches and / or legal action.

Govern what you use, not every data asset you have

But comprehensive Data Governance in a large organisation is often expensive, and there is never enough budget, right?

One way to combat this is start governing data that is only actively used (in reports, dashboards, etc.). Once you have a handle on actively used data, then look at data not used often or at all with a view of seeing you can delete it without any impacts (though this can be done through FinOps efforts too).

It’s also worthwhile adopting agile strategy of starting in one area and getting feedback on what worked and didn’t work before rolling it out to other areas of the organisation.

Data Ownership and Standards Are More Important Than Tooling

I know, right, an engineer recommending less focus on tooling? Wild.

I’ve seen many Data Governance initiatives fail due to the organisation buying a fancy Data Catalog and then not doing enough work to make the Catalog an essential part of the organisation’s data ecosystem.

To help with that, we recommend implementing data ownership across all essential data and then implementing robust standards and processes, locally by the data owners themselves and globally by a central data team.

Data Owners (or Stewards or Custodians) mean someone (or a team) is responsible for the data assets: quality, documentation, security, and access, helping resolve many of the issues we spoke about at the top of the article.

After that, I’d look at implementing a cross-organisation metadata model or schema for all governed assets so that minimum Data Governance requirements are set. There may also be local policies set to extend the enterprise metamodel, like Finance wanting certain metadata tracked for legal reasons.

The metadata model can look like a complex, nested JSON document, but it can also be three tables in a spreadsheet: data sources, tables and columns.

Both of the above are often easier to do before adopting any tooling (re-factoring configuration is usually harder than starting from scratch) and still provide benefits if your tooling is only partly implemented or not at all.

With active data owners, you’ll have all data assets kept up-to-date in any tooling, and a consistent metadata model makes it easier to import and maintain any metadata in tooling.

Do I need a Data Governance Framework?

I know that setting out to build a framework for anything requires time, and you’ll have many competing concerns, so I understand if you feel reluctant to build one, especially if you are a small team with a limited budget.

But there comes a point where fighting lots of local battles with Data Governance becomes more inefficient than building out a framework to reduce Data Governance issues over the long term.

Like I mentioned at the start of the post, I’m not going to go in depth on frameworks, but I will say that whatever framework you use, make sure it’s cyclical so that it’s always improving and you are acting on any emerging issues in a timely manner.

You Might Not Even Need Fancy Expensive Tooling

Another budget hack is to just use a spreadsheet, which should do the job in a small Data Platform, especially if you have a low number of schema changes (a few times a month). The downside is that you can’t easily track who has changed what in the spreadsheet, which is itself a Data Governance concern…

Another option is to import metadata into a database or data lake, which is more work to setup but allows for more scale than a spreadsheet and can potentially track changes.

For cheap metadata visualisation and discovery, think about importing governance metadata into a Business Intelligence (BI) application like Power BI / Tableau so metadata can be discovered alongside the reports and dashboards.

What is Data Catalog and / or Data Discovery Software and do I Need It?

The main point of Data Catalogs (or the more aspirational title “Data Discovery“) is to collect metadata of data assets, including data ownership, column, and table names, import them into a data store (often graph-based) and then combine that with a User Interface (UI) to search by key terms so users can discover data.

But Data Catalog software can often seem expensive, starting at hundreds of pounds per month that can scale rapidly up. Why? Because they have many components that need to work well together: Search Engine, Relational Database, Web Server for User Interface, a Graph Database and maybe a Streaming Cluster. Collectively, this costs a lot to run.

I wouldn’t recommend building your own unless you have at least a few million dollars or pounds to burn (though some still do, as Data Governance is a bit different in every organisation, making it hard to find the tooling that exactly matches your requirements).

So are they worth bothering with?

At a large scale, yes, data discovery software has powerful search engines that search over millions of data assets in fractions of a second, which might take a long time or be impossible to search for in a normal spreadsheet or database.

They also often have:

• Integrations with popular data sources, making metadata import much easier to setup.

• Data Lineage.

• Real-time updates.

• Actions that are sent to Teams or Slack if there are any important metadata changes or issues.

• Integration with Data Quality tooling like Great Expectations.

• Automatic Personal Identifying Information (PII) detection.

• Some now even use AI to generate documentation for you.

One interesting trend we’re also seeing is getting Data Catalogs for “free” alongside other data solutions (Databricks Unity Catalog, Monte Carlo, Starburst Gravity and Dagster Asset Catalog) as "Data Catalog lite".

They lack many features found in a bespoke Data Catalog, but can help data teams have a smaller, cheaper on-ramp into Data Catalogs without buying another product to maintain.

Summary

As mentioned at the top, this is only a lightweight summary of Data Governance and we did not cover:

• Data Culture

• Data Literacy

• All the Data Governance Roles

• Data Quality (though have a upcoming section on this)

That said, my takeaways are:

• All Data Projects / Products and Platforms benefit from at least a little bit of Data Governance

• Start small and don’t initially focus on governing all the data.

• Ownership and Standards should ideally exist before starting to look at any tooling

• Once you do implement tooling for governance, there are a variety of options, from a simple spreadsheet to an enterprise Data Catalog that costs millions per year in licences.

I think my last comment on this is that, like Data Quality, successful Data Governance requires effort from all of the organisation, not just Data Governance professionals.

If you have any questions or comments on this article, leave a comment below!

Sponsored by The Oakland Group, a full service data consultancy. Download our guide or contact us if you want to find out more about how we build Data Platforms!

Cover photo by fabio on Unsplash

• What Should We "Shift Left" On?

• Podcast: The State of Databases Today

• Serverless is Still Not Designed for Data

• Why Prices are Still Going Up When Companies are Spending Less?

• Ditching Databases for Apache Kafka as System of Record

• They Said Streaming Would Overtake Batch.

What Should We "Shift Left" On?

There is a lot of talk of “shifting left” on a number of aspects of a data platform or product. But what is shifting left? Simply put, it is building a feature at the start of a project's or product's development that is normally built late in development. And if you want examples, I have some below:

• Shifting left on testing has been around since at least 2001 and has it’s own Wikipedia page.

• Shifting left on DevOps has also been around for awhile.

• Many data quality vendors talk about how they can help you shift left.

• So do some data governance vendors.

• And while I couldn’t find any data modelling articles on shifting left, there has been a lot of complaining about the possible trend towards less data modelling, especially lack of conceptual modelling, which is also typically done at the start of development.

• Also security, though if you’re putting sensitive data into your MVP, then I’d argue security should built in from the start.

So should we build our Minimum Viable Product (MVP) with testing, data modelling, data governance, DataOps and data quality built in on the first iteration so we can “shift left“? That’s a lot to cram into an MVP even if you buy a bunch of expensive off-the-shelf products! Especially if you have a deadline of, say, six weeks to prove business value.

Speaking of business value, that should be your number one goal; if you can’t do that, then you are unlikely to get an extension to your MVP. But proving value can be tricky, especially with more experimental products where we are not even sure if the product can be built.

But focusing on business value means that shifting left quickly becomes shifting right, leaving you with a potential mountain of tech debt that could take months to clear, reduce trust in customers (remember, no data quality), and slow down the development of new features.

So what do we do? There are no silver bullets, but a few things can help:

• Testing, DataOps, governance, quality, and modelling are not on-off switches; you could pencil in just enough of them to keep technical debt to a reasonable level while not turning each of them into a multi-month project in their own right. The downside to this is that it can lead to a lot of burnout-inducing context switching and work in progress.

• If you’re not sure the budget holders will give you extra budget for “shifting left”, present it as a series of options in the project proposal while also mentioning the tradeoffs described above for shifting left or right.

• Hire more people and break up work into parallel workstreams. This can work if you are pretty sure your product will add value, but rarely does doubling the number of people mean you double the production; in fact, it can slow down production.

• Buy off-the-shelf software that suits your exact needs (or near enough). The downside of very specialist software is that it’s likely to be less adaptable in the future as the business and requirements change. Also, more expensive licence costs.

• If you are building the same feature again and again for projects, convert it into a service or architectural pattern so future projects save time. For example, a well-tested monitoring service or cloud infrastructure module. This won’t help you with your next MVP, though.

• I’m tempted to say DataOps should come before the other items I mentioned above, as improving DataOps improves the efficiency of new features. Data Quality is another strong contender, as poor quality can lead to a loss of trust among stakeholders that is hard to win back. In reality, this is not a hard rule and depends on the use cases.

• As a consultant, I can’t help but mention that you can hire a team of experts to help kickstart the development process, especially as many of them have pre-built accelerators. I also have firsthand experience that this is no silver bullet; we cannot always turn water into wine, and it’s more expensive than in-house staff.

If you have any other ideas on how to shift left under difficult time constraints, leave a comment below!

Subscribe now

Podcast: The State of Databases Today

This is an interview with Andy Pavlo, CEO of OtterTune and “Professor of Databaseology” at Carnegie Mellon.

Andy Pavlo does great articles reviewing databases each year, so it was great to hear him in audio form talking about relational databases vs. other types and the future of the database market, among other topics.

Serverless is Still Not Designed for Data

I’d personally rename this to “Microservices is Still Not Designed for Data“ as there are serverless databases available everywhere now, but the article is still an interesting read from Ciro Greco and Jacopo Tagliabue of Bauplan, as it is curious how Microservices have become a big thing on the operational side and not the analytical side.

Why Prices are Still Going Up When Companies are Spending Less?

Analysis from Jason Quinn from Vendr, who helps companies buy SaaS software.

Ditching Databases for Apache Kafka as System of Record

Andreas Evers, CTO of KOR Financial, pitches an idea that many may see as crazy, as Kafka is mostly used to get lots of data from a to b very quickly.

They Said Streaming Would Overtake Batch.

A very good comparison of streaming vs. batch and why streaming hasn’t taken over Data Engineering by Data Engineer Daniel Beach.

Sponsored by The Oakland Group, a full-service data consultancy. Download our guide or contact us if you want to find out more about how we build Data Platforms!

Cover Photo by Nick Fewings on Unsplash

Data Security is a Trade-off between Risk and Cost

I don’t need to tell you how important security is to a Data Platform: large data breaches hit the headlines almost daily, which can have massive monetary, legal, and reputational costs.

However, it costs money to enforce data security: expensive cybersecurity tooling, “SSO tax”, hiring security experts, and added security processes that can slow down innovation if not designed correctly.

So it’s a tradeoff between how much risk of a data breach you're willing to bear vs. how much of your product or platform budget should you spend on security?

So how do we know what is “good enough“ security? Fortunately, there are well-known security recommendations and guidelines to aim for, such as NIST (US Gov), CAF (UK Gov), CIS, Mitre Att@ck, ISO 27001 and OWASP, not to mention that every major vendor will have a guide on security best practises for their product.

There is also one security strategy that is repeatedly used in the industry for secure software: defence in depth.

Defence in Depth

Just like a multi-layer backup strategy guards against data loss, a defense-in-depth strategy guards against data breaches by having multiple layers of security, so if one layer fails, you still have other layers to secure your data and minimise the blast radius of any breach.

We’ll go through each of the layers in turn:

1. People and Policy

People are the number one reason why data breaches happen: they accidentally send data to the wrong place or store sensitive details like passwords or personal information in a public place.

This is why security training is usually a mandatory first-day exercise for many organisations.

You also want to foster a culture of proactive security, where software teams are always on the lookout for any possible security issues so their software is as secure as possible.

2. Physical

I’m not an expert on physical security, so I can’t talk about how to make your data centre bombproof; therefore, I won’t comment much on this section. Typical recommendations here are to always keep devices that have access to sensitive data, like laptops and phones, close to you or in a secure location.

3. Perimeter

You can think of all organisations as having a “security perimeter”, where you have to login and have the correct credentials to get inside the perimeter.

The two main ways to setup a perimeter are Virtual Private Network (VPN) and Zero Trust Architectures (ZTA), with ZTA being the newer, generally preferred approach, especially if you are using the cloud. Sometimes both are used!

In the cloud, there are lots of organisation-wide security features for:

• Authentication (AWS, Azure)

• Authorization (AWS, Azure)

• Security monitoring (AWS, Azure)

• Data loss prevention (AWS, Azure)

• Plus a half-dozen other options available.

4. Network

I think most engineers can get away with just knowing the basics of the first three layers, but this and the latter layers are where you want to get to know the best practises, as these are the layers engineers are going to be building, interacting with, and enforcing.

The practise of networking is about connecting compute and storage together, so best practise dictates trying to give each other access to as few services as possible.

In reality, this can be onerous to enforce, so there can be a trade-off here: for example, an organisation-wide service could give access to all internal services, even if they're not used by all services.

Other major aspects of network security are:

• Firewalls to block traffic to malicious websites and reduce the attack surface. Many internal firewalls also block all inbound connections, so you can’t connect to most or any data or compute from outside the network aside from a few monitored endpoints.

  • It’s common to have multiple firewalls for redundancy and reducing the attack surface: organisational firewalls for all traffic, internal firewalls, application firewalls, and server (Operating System) firewalls.

• Private Networks (AWS, Azure) create internal perimeters for each application or set of sub-components of an application, each with their own firewalls.

• Encryption in Transit: encrypting traffic between servers reduces person in the middle attacks. The most common encryption used in transit is HTTPS.

• Private Network Endpoints (AWS, Azure) keep all traffic internal, so no traffic goes through the public internet, which again reduces person-in-the-middle attacks.

5. Host / Operating System

While in the cloud, you may often use managed or serverless services, so you don’t have to worry about configuring the server's Operating System (OS) according to security best practises.

If you do need a server, one option is to use preconfigured hardened servers and containers in the cloud marketplace, such as CIS server images saving you a lot of hard work, though not everything, as you’ll want to have the minimum required access as well. For container security, I’d recommend reading docker’s security documentation. For servers, security is such a big area that you’ll likely end up reading multiple books or entire websites on the topic.

You may also want to install anti-virus software on the server, especially if users are going to be actively logging in to the application.

You also should think about a server patching policy, which can actually be very painful to setup as you need to be careful not to push software updates that break your software (note, automated testing is great here), but also make sure you push updates often enough so you don’t have software with vulnerabilities.

6. Application

Most applications will have some kind of authentication, though it is increasingly likely that applications are linked to a single organisation-wide authentication service to enforce Single Sign-On (SSO). Third party vendors generally charge extra for enterprise SSO.

Limiting what you can access once you are logged in is usually done through Role-Based Access Control (RBAC). Here, you assign everyone one of many roles that each have different levels of access: typical roles are read-only, user, and admin.

It is also worthwhile scanning all code for vulnerabilities on each change as part of your DataOps pipeline. Another advantage of a fully automated DataOps pipeline is that you can limit access to sensitive areas, such as production environments, to only a few administrators rather than every developer.

7. Data

You can use your own data to limit access to other parts of the data using Column and Row Level Security, which is also known as Attribute Based Access Control (ABAC). This works by having a list of users with some user attributes, and then attributes are looked up to check for access. Some data storage and processing vendors come with managed features built in, usually at a premium. There are also open-source initiatives like Open Policy Agent (OPA) to build a cross application RBAC and ABAC authorization language.

You’ll also want to add Data Encryption at Rest, which encrypts data when it's stored in data storage. Most major cloud providers and data vendors provide encryption at rest, though many also charge a premium for it.

Finally, Data Anonymization or Data Masking: some vendors provide this, but you can also do this yourself, though it requires some maintenance overhead. One issue with this is that it can make some analytics difficult or impossible, and you may have to pay extra for it. It is also worth knowing the difference between anonymization and pseudonymization.

Security Monitoring

Monitoring and alerting are another big element of security. Alerting can tell you of a breach the moment it happens, and if there is a breach, you want to know who has access to what for legal purposes.

One issue with extensive monitoring is cost at scale, as all these logs will cost money to store and sometimes process. You need to think carefully about what long-term log data will be useful for, so you are not paying any unnecessary costs.

Also, monitoring data is itself a security risk, as it logs network and server details that can be used against you if leaked. Because of this, monitoring is usually centralised, and raw log data is often only accessible by a few administrators and developers.

Security at Scale

Building and managing all these layers can be quite difficult and expensive for one data team, as well as distracting them from their original mission of providing value to the organisation through data.

You can help ease the burden by providing a number of security services for data teams to adopt, usually in the first three to four layers, such as SSO, organisation wide cloud security policy and Network Firewalls.

Also, providing common architectural patterns can be a massive help here, as they can and should include best security practises. There are some overheads in keeping them up to date, though, and there will always be edge cases that don’t fit existing patterns.

Federation

As you scale, the security team will know less about the context of the data team(s) and can easily get overwhelmed by the number of requests it receives for permission changes. This is where it can make more sense to move permission management to the product or project owners, so they can manage access.

This can save time on getting the required permissions, put less strain on the security team, and provide the best security as the owners know their applications the best.

This does have the downside of requiring the right tooling to do this, though many large cloud providers allow for permission groups that can be owned by people who are not admins elsewhere.

Summary

Thanks for reading! My takeaways are:

• Security is a tradeoff between risk, usability, and cost.

• Defence in depth is a great strategy for security.

• Monitoring is very useful for security but also poses a security risk if configured incorrectly.

• Federated Security is very useful at scale.

This was a vendor-neutral article; if you’re interested in Azure Data Platform security, I also recommend my colleague Abigail Brown’s article.

Thanks to Phil Bent for reviewing this article. Photo by Jason Dent on Unsplash

If you have anything to add or any questions, feel free to comment below!

Sponsored by The Oakland Group, a full-service data consultancy. Download our guide or contact us if you want to find out more about how we build Data Platforms!

• Small Book Review: Team Topologies

• Data Governance is Broken

• Modern Data Stack Festival 2023 Recap

• The Streaming Plane

• Practical Introduction to Data Vault Modelling

• Finding the Right Balance in Data Mesh Implementations

• Why AI Can’t Pass This Test

Small Book Review: Team Topologies

I read this book after Piethein Strengholt recommended it in his “Data Management at Scale“ book (which I also reviewed). It is authored by Matthew Skelton and Manuel Pais, both experienced consultants and trainers.

And in short, it’s a really great book for many IT leaders, especially those interested in implementing Data Products. It lays out a framework for creating teams that best fit the Agile and DevOps ways of working rather than trying to crowbar Agile in an existing organisation without any changes.

The authors are trying to ease the pain of engineers having to make many interactions and handoffs to get work done, which increases inefficiencies, burnout from too much context switching, issues caused by miscommunication, and therefore reduces employee engagement too.

This can happen in data by having a single cross-organisation team for:

• Business Analysis

• Data engineering

• Data Science

• Data Analysis

• Data Architecture

• Cloud Infrastructure

I could go on. This can create, as an example, several handoffs between teams to get a business requirement into an extra metric for a report.

What can also happen is Conway’s law (building software that reflects the organisation), instead of building software that matches the organisation’s value chains, which again can create unnecessary handoffs. The authors propose the opposite direction: ‘Inverse Conway Maneuver’ (ICM - building teams to match an organisation’s software services).

So we ideally want only one service to provide value to the organisation in a single, focused way (for example, a sales forecasting report) and with only one team entirely dedicated to making that service the best it can be. If we have multiple teams doing the above, it increases the number of handoffs needed, which in turn likely increases the number of miscommunications and misalignments too.

Doing the above will create a number of cross-functional stream-aligned, service, product, or feature teams, like:

And we’ll want to add a “Platform Team” for any central services like organisation-wide infrastructure, security, data governance, and monitoring:

There are also two other types of teams, called enabling teams and complicated-subsystem teams, which are specialised and less common but provide important cross-product, non-platform teams.

And if you want to see how this looks at scale, Docker has a blog on how they implement Team Topologies:

For those who already practise agile, this doesn’t look ground-breaking, and all I can say is that I had the same thought as someone who’s been working on cross-functional teams for many years now. But the authors here go into more depth about team design and interactions than I’ve seen elsewhere, so I still got a lot out of this book.

For example, thinking about and tracking all the interactions a team makes and then checking if they are the right interactions, the right number (too much equals burnout), and the right type of interactions. I also liked the idea of thinking about documenting team interactions like APIs - which made me think of Data Contracts or a standardised metadata model for a Data Product.

It’s quite easy to map this book to Data Products and Data Mesh and to be honest, I wouldn’t be surprised if Zhamak Dehghani was partly inspired by this book when creating the Data Mesh. And just like Data Mesh’s, I would say this team structure is best suited to larger, more decentralised Data Platforms.

Now, many experienced leaders fear a reorganisation because, at best, it reduces productivity and employee engagement in the short term as everyone adjusts to their new role and colleagues, and if done badly, it can leave permanent damage. The authors of this book are very open about this, saying their methods take time to work. So I’d understand if you read this book or review and thought, “This sounds great, but I don’t have time, money, or political clout to implement it.“.

I would say to that, yeah, this book is less useful to you and leaders of smaller teams, but still has many merits: you can still think about the interactions aspect in an existing team or re-think how best to collaborate on new projects and products where there are easier opportunities to create and redesign software teams.

Right, on to the links!

Subscribe now

Data Governance is Broken

While I’m not sure I agree with the title of this article from Paolo Platter, CTO of Agile Lab, This is a good summary of issues that can happen in Data Governance: with a nod to my review above, look at all those handoffs and opportunities for miscommunication!

I also liked the recommendations for making it work better in organisations, especially the one about starting on governance as early as possible: governance is like cleaning the house; it’s easier and more efficient to do in lots of short bursts rather than in small amounts of big bursts of effort after rubbish has piled up.

I think the real issue with Data Governance is that, frankly, it doesn’t happen enough, especially at project or product inception. The same goes for Data Quality and Modelling. The reason why this happens is simple: in the very short term, governance doesn’t always bring value to a product, unlike, say, building another feature for a product like another set of charts for a report.

So governance problems like lack of documentation and data sharing pile up and get harder and harder to clean up as bad practises become habits (see my cleaning metaphor again).

The question for me is: should Data Governance (plus quality and modelling) be part of a Minimum Viable Product (MvP)? And if not, when should it be implemented?

Modern Data Stack Festival 2023

This was a great online conference hosted by Secoda two weeks ago, with talks and panels by Joe Reis, Taylor Brownlow, and Chad Sanderson.

So why am I linking to it now? Well, most of the videos have gone live for all to view:

• Day 1 Playlist

• Day 2 Playlist

• Day 3 Playlist

You’ll likely find at least a few videos you’re interested in.

The Streaming Plane

The gap between operational data/application teams and data analytical teams has to be one of the most important issues today for Data Platforms, hence why Joe Reis has written not one but two articles on it.

Hubert Delay, co-author of the Streaming Data Mesh book, thinks the gap can be bridged with a Streaming Plane.

Practical Introduction to Data Vault Modelling

While I hear a lot of noise about data architects and engineers implementing Data Vaults, this is possibly the first time I’ve come across a good introduction to them that is not in a book or behind a paywall.

Not only that, Nuhad Shaabani has provided code to implement your own example Data Vault.

Finding the Right Balance: Socio-Technical Balance in Data Mesh Implementations

One of the benefits (and issues) of Data Mesh architecture is that it goes beyond technology and has solutions for how an organisation should structure it’s processes and teams. It’s not a technical architecture but a socio-technical architecture.

Omar Khawaja, Global Head of Data and Analytics at Givaudan, explains why socio-technical architectures are important and also breaks down the four core tenants of the Data Mesh into their social and technological parts to show what kind of effort is required for each of the tenants.

Why AI Can’t Pass This Test

While this is very much a video for a general audience, this video makes an important point: AI is currently great at certain things but hopeless at others. It can be easy to get caught up in the recent hype and think everything can be solved by GPT and similar models, but in reality, it’s best currently for a subset of tasks.

Sponsored by The Oakland Group, a full-service data consultancy. Download our guide or contact us if you want to find out more about how we build Data Platforms!

Do I Need any Data Modelling?

With the rise of Query-Based Modelling adopted by large companies and enterprise “super” data models being broken up by Data Products, you may be forgiven for thinking data modelling trend is decreasing, maybe even dead. Even if that is the case, I think there is still a case for some modelling in all datasets.

Why? Well, almost all data has some kind of structure, so putting data in a consistent shape and style can allow data teams to make changes faster as you scale up your Data Platform. Thinking a bit about your data model before implementing it is like drawing a chair diagram before you cut the wood to build it.

This doesn’t mean you have to spend weeks or months modelling every dataset, but at least think about rules for consistent labels like columns and table names with sufficient documentation, starting with your most used data, like you would with any well-written code, even if you’re only using a Data Lake or Real-Time Streaming transformations.

Data Modelling can also help with Data Governance and debugging data issues if done well; you are documenting the data ecosystem with clear diagrams and easy-to-understand data models that relate to the business.

Data should be able to be easily visualised in reports and dashboards while also having good enough performance that analysts are not waiting hours or even days for queries to run; both are heavily affected by how you model data.

Data Modelling Workflow

Just like in Solution Architecture, you want to start with the Business Processes and Business Rules: what is the Business Process from source to report for your data and what rules must it comply with? Of all the steps, this is arguably the most crucial to get right, as you want to build a data solution that matches stakeholders expectations.

This stage is often called requirements gathering, which is a whole topic in itself and has many textbooks on it.

The next stage is Conceptual Modelling: if dealing with lots of data assets, it is usually easier to understand if they are grouped into business domains like Finance, Human Resources and Sales and entities like Employees, Customers and Projects. This part is sometimes skipped, especially for small models that only have one or two concepts.

You’ll want to use the same language as the business domain at the first two stages so you can have Subject Matter Experts (SMEs) in the business domain verify your modelling.

Logical Modelling: this stage formally defines the relationships between tables, what unique identifiers to use and what columns tables will have. This part is also sometimes skipped, especially on small models where it is combined with a physical model stage to save time writing two different models. One reason to keep this stage is it is largely technology-independent while driving deeper into the model details.

The last stage is Physical Modelling, which defines the column types and data constraints. After this stage is complete, can use tooling such as SqlDBM or Erwin Data Modeler to convert a physical model diagram into SQL code to save a lot of time creating tables by hand.

I’d recommend checking out Sonny Rivera’s post which goes into more detail on modelling stages if you’re interested.

Model Layers

The other important consideration is how we process our data through our data pipeline, from source to report. Generally, there are three layers: Raw, Conformed and Curated (there are many naming variations, with Databricks confusing everyone with Bronze, Silver and Gold, but they achieve the same aims).

• Raw layer is most of the time a straight copy of the raw data with essential metadata and maybe some transformations to fit the technology the data is being moved to, so it requires little modelling.

• Conformed layer cleans and deduplicates data to make a Single Source of Truth (more detail later). May also add some enterprise-level business rules too. This is arguably the hardest layer to design, as you have to balance use cases for today with adaptability in the future.

• Curated layer is analytical data that adds business value in some way directly, so often has a very narrow focus of answering one or two business questions. You may end up splitting this layer into business domain Data Marts.

I’ve seen some data models and pipelines combine both Conformed and Curated together, which could save some time on smaller data models, but does make your conformed layer more narrow in focus and harder to adapt in the future.

Finally, you have to be careful with the curated stage so that you don’t end up with hundreds, if not thousands, of unused tables used for one-time queries. Combat this with careful pruning based on usage metrics or by putting more experimental and/or exploratory analytics in a separate layer and/or schema.

How to Model Data?

Agile, Value-Oriented Modelling

Model what you need to, not every data source you have to get quicker Time To Insight (TTI). As mentioned earlier in the guide, you should first look at your anticipated analytical outputs and then only model the data that is needed for those outputs.

Maybe your Data Platform is small and only has a few tables, in which case model the entire Data Platform. However, if your Data Platform will potentially contain hundreds of tables and will take years to implement, we recommend only modelling on data you will use in the next few weeks or months.

Model Types

The most common data model type historically has been the Star Schema or Kimball model, especially when it comes to reports and dashboards in Business Intelligence (BI). For example, some testing has shown Power BI is often better modelled as a Star Schema.

This does not mean Star Schemas are always the best type of model, and with newer technologies like Columnar Data Warehouses, streaming, various flavours of NoSQL and Data Lake, you may require slightly different methods of Data Modelling, so model your data according to your use cases and technology.

Arguably the most common alterative model is One Big Table (OBT), where it creates a much simpler model where the most commonly used attributes and measures are put in, well, one big table, with possibly small reference tables to supplement it.

The benefit of this style is that you’ll have to make fewer joins with other tables, which can be an expensive and difficult to debug operation if there are lots of tables to join. OBT also matches the data model required for most Data Science, AI and ML model algorithms, which often expect one big table (or matrix) of features and target values.

The downside is that you can end up with a massive table of dozens, if not hundreds, of columns, which can make it hard to understand and use. Also, as mentioned above, BI applications like Power BI generally prefer Star Schemas.

Another big factor in how you model your data is how often your data structure changes. This can have a big impact on your modelling; you may even decide to use semi-structured data storage if data changes structure a lot (from data point to data point or on a daily basis).

Activity Schemas offer an alternative here by storing data in a tabular database with columns for unique identifiers and metadata but storing dataset attributes and/or semi-structured elements in a JSON column to keep data models smaller for less maintenance and potentially improve performance.

The downside is that it requires more effort and expertise to use JSON columns in BI applications and queries. I wrote a longer post on Activity Schemas if you want to know more.

Also consider Data Vault models for better write performance while keeping to a more tabular structure. The downside of Data Vault models is that they can make your models bigger, so they may require more maintenance. Also, like Activity Schemas, usually not ideal for direct use with BI applications with a small number of data assets.

With all these modelling styles that each support different use cases, it may make sense to use multiple types of models if you have varied workloads and analytical technologies. It is a common pattern to use modelling patterns that may suit better performance, like OBT, Activity Schema or Data Vault closer to the source data and then use Kimball or Star Schema closer to the consumption side for easier understanding of large data models.

Non-Database Data Should Still be Modelled

Data Lakes still need modelling: they should be structured in such a way that it’s easy to find the data you want, and large Data Lake files should have thought put into how it is partitioned to get the best performance.

The same goes for streaming, logs, event data and semi-structured data (JSON, NoSQL): think and document about how the data will be used, joined, and change over time will help create a consistent structure to make refactoring and debugging issues easier in the future.

Even metadata should ideally be modelled, so you have clear and consistent requirements for data ownership and usage on all datasets for easier governance.

Single Source of Truth

A lot of modelling effort can go into creating a Single Source of Truth - one record, dataset or model to rule them all for each entity. For example, the customer or employee dataset that (nearly) everyone should use in an organisation. This makes new analytics and fixing issues much easier, as you always know the best, most trusted place to look.

Getting to this deduplicated nirvana can be very difficult, especially if duplicated data is already widely used. Master Data Management (MDM) can help massively here, but it can cost a lot unless you are a large enterprise and will take time and effort to implement.

What is Master Data Management (MDM) and do I need it?

Master Data Management can check multiple data sources, combine them into one correct record, and send it back to the data source if the source needs updating.

You can use algorithms such as Levenshtein Distance or Cosine Similarity to compare two possible duplicate records, and then use other metadata like update time to work out which record details are correct to give you a Single Source of the Truth.

Though today’s off-the-shelf MDM software often uses more complex Machine Learning algorithms to do such matching.

One example of MDM in action is customer records: large organisations likely store customer data in multiple places across departments, and it can be important for marketing and legal reasons to keep them consistent and up-to-date.

Off-the-shelf MDM software can be expensive (starting at >£10k per month) and can take awhile to integrate, so it can be cheaper to roll your own solution for simple use cases, though for complex, large-scale cases, you may find MDM is value for money. For example, scanning a table of billions of rows for a “fuzzy match“ can be expensive in computer resources, so high performance off-the-shelf MDM software can provide performance benefits without you trying to find them yourself.

I also haven’t mentioned that you often get a nice User Interface, alerts, etc. with MDM software, so it can be used beyond Data Engineers.

I should also mention Reverse ETL, as it’s typically much cheaper than MDM, but just sends analytical data back to the data sources like Salesforce with no matching algorithms. This is great if you already have clean, deduplicated data in an analytical data store and want to send it back to the data source.

There is also Entity Resolution and Identity Resolution, which feel mostly like rebranded MDM for the cloud age, possibly without the moving of deduplicated data back to the data source.

If you do implement MDM, I would look at first setting it up in registry style, especially in a complex environment, where you deduplicate incoming data and assign unique identifiers, but don’t use it to conform to a golden single source, which can involve a lot of work in removing and changing existing data. You can use this style initially as a trial run to see if MDM is producing the data you want it to.

One other important implementation consideration with MDM is whether you send deduplicated, cleaned data back to source* or not (consolidation style): while sending data back to source will lead to better Data Quality, the data source technology might not be able to be updated by a third party source, and the data source owner may not have the budget or the will to incorporate MDM into their systems.

*There are two ways to do this: centralised, which queries the master record before creating a record in a data source or transactional data system; or coexistence style, where data sources are ingested post-creation, then cleaned, deduplicated, and sent back.

Summary

Another big topic with lots of takeaways:

• All data that is used by your customers (internal or external) should at least have a bit of modelling and documentation, at least to keep it consistent and easy to find.

• Model only what you need.

• Think about multiple model types for varied workloads.

• Single Source of Truth and MDM are very powerful but can be very expensive and time consuming to implement.

I hope you enjoyed reading this; leave any comments or questions below!

Cover photo by Karen Vardazaryan on Unsplash

Sponsored by The Oakland Group, a full service data consultancy. Download our guide or contact us if you want to find out more about how we build Data Platforms!

Why Invest in Data Quality?

This can seem like a rhetorical question: you should always invest in Data Quality! But I argue we are still not investing enough: surveys show Data Quality issues are increasing in most organisations and, on average, take up 34% of a Data Engineers time, instead of creating value by adding new features. This increases to 50% for large Data Platforms.

All these Data Quality issues add up, with bad Data Quality costing organisations on average $15mil a year.

Data Quality investment is also an investment in high-quality AI and ML, as you’ll likely get more accurate AI and ML results from improving Data Quality than changing your AI model and code.

Having Data Quality checks in place helps reduce “data downtime" for outages and fixes, which subsequently increases the overall reliability of the Data Platform. Highly reliable data leads to more trust in data and better informed decision-making.

Better decision making should increase profitability, productivity, and confidence of the whole organisation, which in turn usually leads to more investment in data and, as a result, going back to the start to increasing the quality of data again.

All this creates a “Virtuous Cycle” of Data Quality, constantly improving your organisation:

If Data Quality decreases the opposite happens with a negative cycle.

Better Data Quality testing should also reduce the blast radius of the issue to a few Data Engineers rather than hundreds or thousands of users as more issues being found earlier:

The fewer users impacted, the smaller the cost caused by the issue, which should again pay back any investment in Data Quality in large multiples.

Do I need a Data Quality Framework?

I know that setting out to build a framework for anything requires time, and you’ll have many competing concerns, so I understand if you feel reluctant to build one, especially if you are a small team with a limited budget.

But there comes a point where fighting lots of local battles with Data Quality becomes more inefficient than building out a framework to reduce Data Quality issues over the long term.

I’m not going to go in depth on Data Quality frameworks, as they are often tied to wider Data Governance frameworks, and we have a guide on that here. I will say that whatever framework you use, make sure it’s cyclical so that it’s always improving and you are acting on any emerging issues in a timely manner.

How Do I Test for Data Quality?

Classically, Data Quality tests are a set of rules that test between the actual and desired state of data. The desired state may not be perfect, but ‘good enough’. What counts as ‘good enough‘ varies from dataset to dataset, which makes Data Quality more challenging.

What do we normally test in a dataset, though? The DAMA International’s Guide to the Data Management Body of Knowledge says there are six dimensions to Data Quality:

• Accuracy: does the date look how we expect it to?

• Completeness: are there any unexpected missing values?

• Uniqueness: no duplicates!

• Consistency: does a person’s data match in two different datasets?

• Timeliness: is the data out of date?

• Validity: does the data conform to an expected format? Think postcodes, emails, etc.

Tracking all these dimensions for every dataset at every stage of your pipeline is a lot of work, probably too much work. Therefore, a trade-off is often required to focus on areas where Data Quality will have the most impact on the business.

You also have to beware of false positives or minor issues being blown out of proportion, overwhelming your engineers with too many issues that can be fixed. It can help categorise your Data Quality issues by severity just like other software issues.

You also have to take into account the mental wellbeing aspect too: few Data Engineers and Analysts want to spend a large percentage of their time fixing Data Quality issues over a long period of time.

There is help, though, with software frameworks to help you write Data Quality testing:

• Great Expectations

• dbt tests and dbt-expectations

• Datafold

• Soda

• Monte Carlo

Most of the above profile your data and setup recommended tests for you to use, saving you some time configuring them yourself.

But in reality, we see a lot of custom-made Data Quality testing, partly because Data Quality isn’t often given a large amount of investment, so is done when possible in an organic, ad-hoc manner.

The above products work best in development and staging environments, so you can find issues before they enter production or use them as circuit breakers, to stop a Data Pipeline if the incoming or outgoing data is of poor quality.

It is also worth mentioning that you can use constraints in a Warehouse or Lakehouse schema, which has the benefit of not requiring another software library but not as feature rich (would likely have to setup your own notifications for alerting).

It is also important to inform your users of any Data Quality issues as soon as possible so they don’t waste time finding out for themselves or use data that is untrustworthy. This can be done through notifications and alerts, though we’ve also had a lot of success creating Data Quality dashboards that sit alongside existing reports and can be easily referred to by users.

Latest Concepts in Data Quality

There is quite a lot of innovation in Data Quality in the last few years, so we present below the concepts to take your Data Quality process to the next level.

This will require more investment, but it will give you an edge over your competitors to make better informed decisions as you’ll have more trustworthy data. This investment should also pay back long term with less time wasted fixing Data Quality issues.

What is Data Reliability and do I Need it?

Data Reliability gives Data Quality more of a support focus, which makes sense as most Data Quality issues in production will be dealt with as a support issue to a Data Platform.

Data Reliability takes a lot of its thinking from Site Reliability Engineering (SRE), which treats support as more of a engineering problem, where you examine your past and current support tickets and look to decrease them with engineering or better processes.

With Data Reliability, you would look to get a baseline of Data Quality issues per week or month and then look at ways to reduce them and monitor to see if the changes have reduced the number of issues and/or reduced the amount of time spent on issues.

The changes to improve Data Reliability can be technology-based:

• New or updated tooling

• Better automation of when a pipeline fails or automated actions to respond to a data issue

Or the changes can be process-oriented:

• Writing better documentation to avoid common issues

• Incident playbooks so the whole team can more quickly respond to a issue in an consistent way.

You can rather cynically say Data Reliability is just Data Quality with a feedback loop and a time series graph, but it is there to make sure you don’t think too short term about Data Quality and think about long term improvements that will make a more efficient and trustworthy Data Platform.

You may also set targets such as “99.9% of data will refresh on time” or “A maximum of 33% of engineer time should be spent on support issues“ as well. As mentioned before, it can be impossible to achieve perfect Data Quality, so aiming for a reasonable target instead can avoid engineer burnout.

What is Data Observability and do I Need it?

Data Observability is about gaining a Data Platform or organisation-wide understanding of your Data Quality.

It arguably goes beyond Data Quality by adding metadata features normally found in a Data Catalog: cataloguing schemas of datasets and data lineage. These features allow you to more quickly find a Data Quality issue by tracing the lineage of the issue and also you gain the ability to see how much Data Quality is impacting your organisation.

Data Observability software can often also come with Machine Learning (ML) algorithms to detect anomalies in data, so you can be warned about issues you haven’t even thought of yet.

We’ve seen products either extend a Data Quality framework with Data Catalog features such as Monte Carlo and Big Eye. Or existing Data Catalogs add Data Quality functionality, such as Datahub, which imports Data Quality tests created by Great Expectations and dbt tests. Both Soda and Monte Carlo have integration with the Data Catalog Alation.

What are Data Contracts and do I Need Them?

Data Contracts are well, making a contract between a data producer and a data consumer, so the consumer knows what data to expect from the producer.

While you can replicate some of Data Contract’s benefits by tracking the schema of the data produced, Data Contract is meant to go beyond that by giving you a full suite of metadata about the data:

• The data’s schema.

• How the data is calculated.

• Who owns the data?

• What is the data lineage?

• How to access the data.

• What is the data’s expected quality, availability, etc.

• Plus anything else that is relevant to the data.

You may think Data Contracts are redundant if you have a well-maintained Data Catalog, as they capture similar information, but Data Contracts are designed to be checked during every run of a Data Pipeline and have some action in the pipeline if the Data Contract is broken:

• Stop the pipeline with a circuit breaker.

• Alerting.

• Moving data that doesn’t meet the contract to a manual checking table.

For an example, Paypal has open-sourced their Data Contract template.

This should create more positive collaboration between data producers and consumers because they have a collective agreement of what the data should look like. It is not uncommon to have a poor working relationship where a producer makes changes without telling consumers or consumers accessing data in way not recommended by the producer, so good use of contracts

One issue with Data Contracts is that they are a new concept, so require more work to implement at present, though that will likely change in the near future as more companies adopt them.

Most of the examples of Data Contracts we’ve seen so far use Apache Flink and the Kafka Schema Registry, so assume you are using streaming, though there are some examples that use batch processing.

Data Governance and Data Quality

Good Data Governance can also improve the quality of data. It is important to know where data is coming from, who owns it, for what purpose data is being transformed, and finally, what is the impact of poor availability and data quality: all helped by having Data Governance properly implemented.

Some of the above concepts (Data Contracts and Data Observability) can also improve Data Governance, so investing in Data Quality can also be an investment in good Governance too.

We go into Data Governance in more detail in a later section.

How Does This All Fit Together?

We've covered a lot of concepts, so it might be difficult to picture how all this fits together. So I’ve drawn a diagram below as one example of how this could all fit together in a high-maturity organisation:

• Any code changes are tested in development and/or test environments with Data Quality Tests to check that any changes won’t have a negative impact on Data Quality.

• Source Data at the start of the data pipeline is checked to see if the Data Contract is held; if not, a circuit breaker may kick in, stopping the data pipeline early to avoid processing unsuitable data.

• Data Quality tests are also run in production, which can feel like duplication from testing in development, but there may be changes caused by moving to a production environment (different data, etc.).

• Data is collected for observability checks by Data Observability software, looking for any anomalous data: a department budget that goes from £10k to £1mil or 10 10x increase in rows for a table, for example. This can replace a lot of tests, but not all of them.

You’ll also be collecting Data Quality metadata to improve your Data Reliability.

Making all this work together seamlessly is not cheap and will take time, but as mentioned, poor Data Quality will also cost an organisation a lot of money. So I would recommend tackling this in an agile manner by improving Data Quality in small increments, one change at a time, starting where it will have the most impact.

Summary

Data Quality is a difficult subject to tackle, due to it being a slightly different problem in every organisation and never being able to get it perfectly right. That said, there are lots of options to help improve the quality of your data, so you should be able to get to “good enough“ if you give Data Quality enough priority and forethought.

Sponsored by The Oakland Group, a full service data consultancy. Download our guide or contact us if you want to find out more about how we build Data Platforms!

Also, we have the usual selection of great articles to share:

• Hashicorp Makes Terraform Licence Less Open: What is the Impact?

• Massive 70-Page Guide to the data quality resolution process

• How to Create a Secure Azure Data Platform

• After the Modern Data Stack: Welcome back, Data Platforms

• The Art of Building Your Own ELT

• The Complexities of Entity Resolution Implementation

• Mind the Gap: Seamless data and ML pipelines with Airflow and Metaflow

Justin Borgman Interview

First, the video, but for those who prefer to read, I’ve edited the video transcript below:

Jake: Hi Justin, would you like to briefly introduce yourself?

Justin: I'm Justin Borgman, co-founder and CEO of Starburst. Starburst is the company behind an open-source project called Trino. Essentially, we provide a Data Lake analytics platform that allows you to query data that lives both in the lake and outside of the lake, so we can really query data that lives anywhere. And that's part of what sets us apart from other platforms in the industry.

Jake: You were once the founder of Hadoop startup Hadapt. So considering you were very involved in the Hadoop ecosystem, what do you think are the key lessons from the Hadoop era that can be applied today? Because I feel like it's file-based structure is something that has come back around again with Data lakes.

Justin: Yeah, I think you're absolutely right. I think that is the lasting legacy of Hadoop. I mean, really, the notion of a data lake was created during that period. I mean, the first data lakes were absolutely Hadoop data lakes, and that's where that term came to be.

I think the concept of a data lake is going to potentially live on forever because there are so many natural benefits, like the notion of leveraging open data formats, something that was pioneered by Hadoop with the creation of Parquet files and Avro files, and all these different ways of storing data in a columnar fashion to still get good performance. But laid out in this really inexpensive commodity storage system.

I think that's another key: the legacy of Hadoop is this notion of let's store as much data as we possibly can in the cheapest possible place, where we can store it, and that's either going to be, classically, the Hadoop file system or, more often, object storage, S3 on Amazon, Azure Data Lake Storage, or Google Cloud Storage. All of them represent inexpensive storage.

Lastly, I would say this notion of open architectures was really pioneered back then, at least as it pertains to data warehousing analytics, with the notion that not only are your data formats open and your file storage, you know, inexpensive and open, but also that the means for how you process that data should perhaps also be open.

And so you have the rise of technologies like Presto and Trino, which we’re the creators of, or Spark, you know, as another example, and the idea that these engines can all query the same open data formats, I think, is a really important aspect of the architecture that, again, lives on today.

Now, while Hadoop itself may be waning in popularity, you know, again, a lot of these concepts live on and have simply moved to the cloud and cloud object storage.

Jake: Then you moved into Teradata, which looks from the outside in like a very different kind of company than a startup.

Did working for Teradata alter your thinking about how to approach future startups when you came back to founding Starburst?

Justin: Yeah, I think I’ve got two great observations from my time at Teradata: first and foremost, that enterprise customers, where Teradata really is very strong, have particular needs and requirements that internet companies don't have; they need more robust access controls, they need better governance, they need metadata management and catalogue support, and, just Kerberos integration, LDAP integration, so many different sorts of features and capabilities that frankly, somebody like Facebook just doesn't really care about. But mainstream enterprise customers very much do. And so that left an impression on sort of what it means to be “enterprise grade”.

The other observation was maybe a little bit more specific to Teradata, which is that they were really the pioneers of this idea of an enterprise data warehouse, which was always about centralising all of your data into one central data warehouse. And yet, when I got there, I realised that not one of their customers actually did that. Not one of them actually had everything in the enterprise data warehouse; they all have these different data silos, Data Mart's different applications, device data, and multiple clouds on-premise.

There is so much heterogeneity inherent to the operation of their business that this enterprise data warehouse model seemed like it was actually impossible. And that was pretty interesting to see from the Teradata vantage point because, of course, at least at the time when my company was acquired, they had $2.7 billion in revenue and were the industry leader. So this was like the best-in-class, you know, version of the enterprise data warehouse.

And yet, even their customers didn't truly centralise everything. So that was really what started to get me thinking: we need to design an architecture that actually accommodates data living outside of this central data warehouse.

Jake: Yeah, I probably agree from my experience of seeing large companies try their best to centralise everything, but finding out that getting people to even use one cloud platform can be just too much work, let alone trying to get everyone to use one technology,

I would say there are two major opinions of Data Architectures out there right now: Data Fabric and Data Mesh. I'm wondering what your opinions are on them.

Justin: So I think, you know, first of all, I'll start with the commonality, which is that I think both of them recognise that you need to be able to account for decentralised data. And I think the differences lie in sort of how you actually put that into practise, how you govern that, and how you manage that.

And at least the way that we think about it with our customers is that there are certain elements that you certainly want to centralise, and those are the things around security and access controls, management, and administration of the data platform. Those aspects likely need to be centralised.

And then I think it really depends on your organisational maturity in a way: your data maturity and data literacy. Perhaps in terms of who produces data products and who's responsible for data quality, and that perhaps influences just how much further you decentralise those processes or not. And that's more of a people-process challenge than a technology one.

And that's sort of the advice that I give customers, like, you need a technology that can handle your decentralised data, whether you choose to centralise the management, the administration, the governance, and even the data quality or not. It comes down to sort of the maturity of your decentralised teams, in the greatest embrace of a Data Mesh philosophy, you would have those domain owners being the ones who curate and create those data products, and they are responsible for that data quality. And I think there are a lot of benefits to that approach.

But I think you have to ask yourself, are we ready for that today? Do we have the people in place to facilitate that level of decentralisation? Or does it make more sense to sort of still have a centralised team that can simply reach out and access the decentralised data?

Jake: I have also seen some people talk about sort of merging them both together. So you get like a best of both worlds approach, almost like a Goldilocks solution, where you use the knowledge graph of the Data Dabric with decentralisation: I don't know if you've ever seen it or have thoughts on that?

Justin: Yeah, that's interesting. I certainly haven't heard it expressed that way before. Yeah, no, it's an interesting idea. I mean, I think it makes sense conceptually.

Jake: So moving on to data products, I'd say Starburst has been leading the charge in terms of Data Products, and is now becoming the de-facto way we look at breaking up our data in an organisation. So now that we're starting to see an adoption of Data Products in a lot of companies, have you seen any unexpected benefits or issues with Data Products, now that we're starting to see them come into production?

Justin: I think one unexpected, or maybe it shouldn't be expected, benefit that we see, or maybe it's more of a benefit than people expect, I guess, is that people are talking more and there's greater collaboration between consumer and producer than I think historically there has been because, you know, in the more traditional model, the data producer is sort of an anonymous, faceless, you know, human that we don't know as a data consumer. Right? And so if there are areas to be improved in that data product, whether it's adding additional fields or, or what have you, you know, I don't really even know how to begin to send that feedback back to the to the user, or even to ask questions of the data. Sometimes it's simply trying to understand the data that exists better.

But what data products were really facilitating that, that consumer producer interchange, if you will, an interaction and I think that's where people really start to see some of the unexpected benefits to your question of a data products model, data product centric model.

Jake: Yeah, I seen many struggle with the relationship between data producer and consumer, but there has been a lot of talk to improve that with data contracts. I’ve also been thinking, will we start seeing a blurring of the boundary between operational and analytical planes where they work closer together? Or will it stay as two distinct planes, as it might be better to keep them separate?

Justin: Yeah, no, that's a good question. I think that hopefully, we will see more collaboration between those two planes in terms of, you know, I guess the the interaction around like, what the schema should, should contain, right? Like the operational side very often is not necessarily aware of what the analytical side finds valuable in terms of that data capture. And so hopefully, there is greater collaboration between the two.

I don't think the actual underlying database systems will necessarily converge. I mean, every now and then, people talk about that. Well, we have an OLAP and OLTP database that does both. I think that's a little bit more of a challenge around the physics of just being read optimised versus write optimised.

But I think, hopefully greater collaboration between the humans involved on those two sides, because I do think there's a lot of value.

Jake: Moving on to Data Catalogs. With Starburst Gravity recently being released, what I've also started to notice is Data Catalog are being introduced into Data Processing products like Databricks and Unity Catalog, or even into Data Orchestration with Dagster. Do you think this is going to be an increasing trend where we see products lean more into data governance?

Justin: Yes, I think in conjunction with some of the points you brought up earlier around data being decentralised, the role of the catalogue becomes more important and more essential, almost like table stakes for the whole thing to work. Because you now need to account for data that lives outside of one single database. And so yes, I think this is a theme and is becoming increasingly a required capability for customers.

Jake: To be a bit sneaky, are you going to start to expand out Gravity as you go on over the years, or is that something behind a NDA?

Justin: Yeah, no, we do intend to expand it. I think our mission is to make distributed architectures as easy and seamless as possible for our customers. And that means that anything that we can do to make that a more streamlined experience is within scope from our standpoint, and that's everything from metadata management to access controls, which, of course, we now provide in Galaxy, our SaaS product, we provide attribute-based access control, role-based access control, and very fine-grained access control.

We think these are sort of table stakes. And being able to centrally define those and enforce them across all the different data sources you connect to has a lot of value for customers. So yes, I think all of those things are within scope.

Jake: There have been a few people in data saying that big data is dead. I don't know if you've seen the famous MotherDuck article claiming that? As someone who's from the original days of big data, you might know some people who say big data never actually happened. So good to have your take on whether it happen? Is it still happening?

Justin: Yeah, sure. I mean, I disagree with the idea that big data is dead. I mean, I think that it makes sense for MotherDuck to say that, and, look, I think DuckDB is a great database system. There will always be point solutions and specific use-case solutions for specific problems.

You know, one of my greatest influences from a database perspective is Mike Stonebreaker, who's a professor at MIT here in Boston, where I'm based. He is the creator of Ingress, Postgres, Vertica, and so many other different database systems; you won the Turing Award; he's probably one of the leading computer scientists of our time, certainly as it pertains to databases; and one of his most famous statements was that there's no one-size-fits-all database and that you're always going to have purpose built database systems. And I think that's absolutely true. And I think Duck DB is just an example of that.

But the idea that big data is dead, that we don't need to analyse all of our data, or something to that effect, I think is certainly not true. And I think with the increasing attention towards LLM and Generative AI, if anything, I would say big data is more alive than ever before, because your models are only as good as the data that you train them on. And very often, at least as it pertains to LLMs, more data is usually better to have an even more accurate model. So, I think understanding your business holistically is really important. So, you know, while we don't think you'll centralise all your data in one data warehouse, we think that having access to as much data as possible is really, really important.

Jake: Thanks, and that brings us nicely to my next question: I'm wondering what’s going to be Starburst's future with Large Language Models (LLMs)? Say, can you start integrating them into your products or finding other ways of communicating with them?

Justin: Yeah, I think so in a couple of ways. First of all, within the product itself, we've already actively prototyped. And something that we'll be introducing soon is the notion of natural language to SQL translation, and I think that's low hanging fruit for the industry. I don't think that we'll be alone in offering that; I think many, many, many vendors will likely offer that it's actually fairly straightforward to just be able to take natural language and turn those into SQL queries.

I think the harder stuff that is maybe even slightly more interesting because it's hard, is the automatic curation and creation of data products, back to your earlier question (on Data Catalog), you know, can we help facilitate and present data products that consumers are going to find valuable in their analysis. That's a little bit longer lead time, but it's also an area that we think is really interesting.

And then I'll say, broadly, that by virtue of us being able to provide access to all of the data in your enterprise, we want to be the access layer to help you, as the customer, build and train your own LLMs that are really specific to your business and derive a proprietary competitive advantage for you. Right? I think Chat GPT is a great technology to experiment with and have some fun with. But I think where the real value is going to come from enterprise customers is when they're training themselves using their own proprietary enterprise data.

And that won't be Chat GPT, most likely, right? You certainly don't want GPT training on your data. So I think those are the types of things that are very interesting from our perspective.

Jake: Yeah. I'm constantly interested to see how it's going to turn out, for example, where we are going to end up with like open source models, like we had with the previous ML cycles, or everyone's going to end up using the Open AI, which has some concerns with one company dominating.

Justin: And I will say, and obviously, we're big believers in open source in general, that I do think the open source LLM world is just getting going. And I think that'll be a really big force. I think, to your point, people aren't going to want to trust a vendor with their data. And I think these open-source models are getting really good.

Jake: Yeah, agree. Moving on to data leadership, how do you balance running a large business while still keeping an understanding of your customers needs and delivering the technology they need to run their businesses?

Justin: I think the key is, you have to find the right balance between time spent internally and externally. And what I mean by that is that spending time with customers, to me, is almost like oxygen; I need it, and it helps inform our product roadmap.

And, you know, it goes back to that old saying: if you ask somebody 120 years ago, what they wanted in terms of personal transportation, they would say I wanted a faster horse. You know, you have to really spend time with customers to understand they actually don't want a faster horse; they just want to get from point A to point B faster. And that maybe a car would actually be the better way to do that.

And so, you know, if you want to make revolutionary change, you know, disruptive change, and not just incremental change, you really have to spend a lot of time with your customers, empathising with and understanding their challenges at a very root level, because otherwise, you just listen to what they say.

And an average product manager can do that, you know, okay, faster horse, we'll go see if we can work on the horse’s diet or something and get 5% more speed out of a horse, but the person that really lives with that customer and spends a lot of time with them realises, you know, the real underlying requirements and can come up with creative solutions that might actually be more transformative to their lives.

So, to me, that's huge. And something that I don't think any leader should ever lose sight of. But at the same time, you also need to spend time internally; otherwise, you know, things don't get done. So it's about finding the right balance.

Jake: To me, there’s like a three-way thing where there are your internal customers, your current (external) customers, and the wider ecosystem where your prospective customers are, as well as keeping up with the latest trends coming around the corner.

Justin: I totally agree. It's a dynamic ecosystem that's always changing and evolving. And you need to be consciously aware that nothing in our space is ever static.

Jake: And finally, to finish: the future. So we’re already halfway through 2023, so I can't tell you what trends are going to emerge in 2023! So what trends do you think will emerge in the mainstream in 2024?

Justin: Well, I have to say, You're the first one to ask me for 2024 predictions. So I guess this will be my first 2024 prediction here in the summer. But I will say, you know, I think that, you know, going back to some of your questions around Generative AI, I think vector databases will be an interesting thing to watch.

I think open file formats, or, I'll say, modern open file formats or next generation open file formats like Iceberg, will gain a lot of momentum. We're seeing that already. I think that will continue.

I think Iceberg, you know, will win the table format war, maybe that's the most provocative thing I'll say today, and it will just become the dominant table format.

And then, you know, broadly, I'll just say I think increased focus on collaboration and data products, like, how do we create that? That collaboration, and that two-way dialogue between, you know, the data domain owner and the data consumer. So I think data products will be big next year as well.

Subscribe now

HashiCorp's new license is still open source-ish

Most modern Data Platforms use Infrastructure as Code (IaC), which allows you to define your infrastructure as code in development and then run the same code in production, so you cut out errors from making manual changes in production.

Hashicorp’s Terraform is arguably the most common IaC solution, but they have changed their licence from allowing anyone to run their software for free in production to now only allowing companies who do not compete with Hashicorp’s products.

Now, 99% of companies will still be able to use open-source Terraform with this licence change, but it will impact one of the biggest strengths of Terraform: it’s community of tooling around it, which allows you to extend Terraform in a number of useful ways (integrations, security scanning, etc.). With this licence change, some of those tools may no longer be able to use open source Terraform, potentially killing off some of the community overnight.

In response, some of the Terraform community has started OpenTF, which is threatening an open source fork of Terraform that will split Terraform usage in the same way ElasticSearch and OpenSearch did, which is not great news long term.

I can also see lawyers for large enterprises vetoing Terraform, as it can be difficult to work out what exactly is “competing“ with Hashicorp products.

On the plus side, maybe I’ll get to use Pulumi more, which is fully open source and I suspect works better at scale than Terraform modules, with it being able to be written in a existing programming language.

Guide to Data Quality Resolution Process

Considering Data Engineers can spend on average, up to 50% of their time fixing Data Quality issues, Mark’s in-depth 70-page guide on how to best respond to these issues could be the most impactful guide you read! (Aside from mine 😉).

Please note that this is not a guide to preventing Data Quality issues; that’s a separate book in itself.

How to Create a Secure Azure Data Platform (Sponsored Article)

My employer, Oakland, has a lot of experience building secure Data Platforms in critical sectors like Telecoms and UK Government organisations, where only Data Platforms with the highest security get approved for production.

Oakland Senior Data Engineer Abigail Brown shares some of this experience on how to build a secure Azure Data Platform so you too can build a solution that has “Defence in Depth”.

I’ll also have a similar article that is platform neutral for my guide coming out soon!

After the Modern Data Stack: Welcome back, Data Platforms

There are quite a few “Modern Data Stack (MDS) is Dead!“ articles out there, but they're usually written by vendors trying to kill off MDS to increase their profits and therefore usually miss why MDS exists in the first place.

This is written by Timo Dechau, Founder of Deepskydata, a training platform for marketing data collection, who offers a refreshingly more vendor-neutral take by explaining why the Modern Data Stack took off and is very popular: but also why you wanted to adopt a more closed Data Platform.

I will say these “Post-Modern Data Stacks” likely suit small to medium teams who have a decent budget (you are putting an expensive layer on an already expensive data stack), but I can see the point of them if you fit this audience.

The Art of Building Your Own ELT

Ever wanted to build your own ELT / Integration / data movement software like Fivetran? Hugo Lu, Founder of Orchestra, show you how to build the basics of one.

The Complexities of Entity Resolution Implementation

Entity Resolution or Master Data Management (MDM) software is expensive, so how difficult would it be to build your own solution to match and deduplicate data?

Stefan Berkner, CTO of Entity Resolution software Tilores, points out that it can get very difficult indeed.

Mind the Gap: Seamless data and ML pipelines with Airflow and Metaflow

I’ve heard some complain that Airflow is not the best place to run for Machine Learning (ML) or Artificial Intelligence (AI) pipelines, as there are dedicated tools like Metaflow designed around that, giving you better integration with ML and AI software and models.

But Metaflow is not designed for Data Engineering pipelines, so you may end up doubling the maintenance of both by running them side by side.

But Michael Gregory and Valay Dave, Engineers at Astronomer and Outerbounds respectively, offer a better way(?) by running Metaflow inside of Airflow.

Sponsored by The Oakland Group, a full service data consultancy. Download our guide or contact us if you want to find out more about how we build Data Platforms!

• Short Book Review: Data Management at Scale, 2nd Edition

• Cloud Cost Management: How to Optimize and Control Cloud Expenses

• Reviewing “Data Modeling with Snowflake”

• Platform Engineering is Just DevOps with a Product Mindset

• How to Check Two SQL Tables Are the Same

• A Comparative Analysis of Open Source Data Quality Libraries: Great Expectations and Soda Core

• DuckDB ADBC - 38x Better Performance Than ODBC

Now some bad news: I’m going to change my newsletter schedule to twice a week, so I can balance finishing off the guide and working on exciting new projects. I may still post weekly if there is a big news event and if my bandwidth increases.

Short Book Review: Data Management at Scale, 2nd Edition

I’ll start with a summary for busy readers: I highly recommend this book if you architect large Data Platforms or are just looking to get started in this area. It will give you a wealth of ideas and examples to use in practise while also mentioning their tradeoffs.

It is written by Piethein Strengholt, who is the Chief Data Officer of Microsoft Netherlands and a former Principal Architect at Dutch ban, ABN AMRO. He also has a great blog that I’d recommend reading if you’re interested in Data Mesh architectures and Azure.

While the book subtitle is “Modern Data Architecture with Data Mesh and Data Fabric“ and the book delivers that, by giving detail on both approaches and also spends a lot of time devoted to Data Products, so if you don’t believe in them, this probably isn’t the book for you.

The book does stray for a chapter or two outside analytical data: there is a chapter focusing on how data is created in the operational data plane and having a strong look at the pros and cons of Event Driven Architectures (EDA) which is an increasingly common architecture in large organisations and often has a large impact on how you analyse your data.

While some might think focusing on the operational data plane is a distraction and skip those sections, it does give you more insight into how data is created at scale.

Another aspect of the book I liked that it was not purely a technical architecture guide, it also covered constructing teams, processes, platform services, Master Data Management (MDM) and data governance which helps give a 360 view of Data Management.

The only minor negative is all the architecture diagrams are drawn using Azure, Microsoft and Databricks components, though Piethein keeps his description of architectures fairly vendor neutral, so I feel confident AWS and GCP users can still get a lot of insight from this book.

I wouldn’t recommend making this your first book in data: I’d look instead at Joe Reis and Matt Housley’s excellent Fundamentals of Data Engineering and possibly read Zhamak Dehghani’s Data Mesh book, as Piethein doesn’t go into the details on why you’d might build a Data Mesh as much as Zhamark does.

Also, this is less useful if you run a small data team (3 to 6 people), as you’ll have little need to adapt to the scalable Data Platform patterns mentioned in this book, as Piethein says in one of his blogs:

“If your company has a lower level of data management maturity, a centralized approach in the beginning is more appropriate.”

Which I’d absolutely agree with.

Subscribe now

Cloud Cost Management: How to Optimize and Control Cloud Expenses

Managing cloud costs is often a constant battle, especially during a period of high inflation, as costs are usually going up for running the same services while budgets stay flat.

Sonny Rivera, Senior Analytics Evangelist at ThoughtSpot, goes through some valuable tips to reduce cloud costs and also adds his thoughts on why managing cloud costs can be difficult.

I will also use this opportunity to shamelessly post that my colleague Jack Evans wrote a blog on managing cloud costs based on our many years of experience on this topic.

Reviewing “Data Modeling with Snowflake” by Serge Gershkovich

It’s been awhile since I’ve used Snowflake, so while I’ve seen this book get rave reviews, I haven’t had much time to read it myself. But here is one of those glowing reviews from Daan Bakboord, Managing Director Data & AI of Pong.

I’m also glad to see a modelling book focused on modern cloud data warehouses, which sometimes require thinking beyond just implementing another Kimball data model as warehouses like Snowflake operate very differently from regular database and can have different performance characteristics.

Platform engineering is just DevOps with a product mindset

A lot of the problems in data can come back to infrastructure and DevOps, as data can only move as fast and just important, change as fast, as the infrastructure it’s built on.

Luca Galante, Product at Platform Engineering company Humanitec, talks about the common issues in DevOps and infrastructure and how Platform Engineering could solve them.

Want more background on how to automate your data infrastructure? I wrote a section on it as part of my “How to Build a Data Platform“ guide a few weeks back.

How to Check Two SQL Tables Are the Same

Database PHD Student Remy Wang investigates how to compare two tables in a database, which isn’t quite as straight-forward as you might think it is.

Hacker News also has a great discussion on this, talking about what tooling you can use as well to compare tables.

I’ll also add my thoughts on this: where possible, compare numbers, not text, if you can, as text data is more likely to differ even though the content is the same due to different character encodings being used to create the data. It’s also often much slower to compare text than numbers.

A Comparative Analysis of Open Source Data Quality Libraries: Great Expectations and Soda Core

For fans of data quality content, Bruno Gonzalez has just started a newsletter dedicated to it, with one of their first posts doing a in-depth comparison of two of the most popular open-source data quality libraries.

My personal thoughts on this are: open-source Great Expectations has more features but is arguably harder to use than Soda CL and Soda has a more mature managed cloud version than Great Expectations (which is still in beta).

DuckDB ADBC - 38x Better Performance Than ODBC

This post is about how DuckDB, a analytical variant of SQLite (so a file-based analytical database) has now adopted a new type of connection driver, Arrow Database Connectivity (ADBC).

What is most interesting is the performance benchmark than blows ODBC out of the water:

Why does this matter? Most database connections with clients use ODBC/JDBC and I’ve configured connections using these drivers dozens of times in the past. Even if you don’t use ODBC/JDBC directly, it’s very likely you are using some kind of paid managed integration solution like Fivetran that uses the same drivers in a preconfigured connection.

So adopting ADBC may give you massive performance improvements when used with columnar database (DuckDB, Snowflake, …).

Sponsored by The Oakland Group, a full service data consultancy. Download our guide or contact us if you want to find out more about how we build Data Platforms!

Plus, we have usual excellent set of videos and articles to share:

• Starbursts Smart Indexing and Databricks Liquid Clustering - Using Query Metadata to Improve Performance

• Building a Real-Time Analytics Database

• What Is Query Driven Data Modeling?

• What Are “Data Clean Rooms”?

• Reverse ETL 101

• Building and operating a pretty big storage system called S3

Tom Baeyens Interview

Jake: Hi Tom, would you like to introduce yourself?

Tom: I'm co-founder and CTO at Soda. I'm passionate about data. My common theme has been to build Domain-Specific Languages (DSL) that allow more people to automate their work. I've done this first on workflows and now in data quality.

I’ve also contributed heavily to open source: building software that is open source is not only good for business, it also helps the world become a better place.

Jake: How did the idea of building a Data Quality Platform come about?

Tom: We were working in the data management space. As more data became available, we realised that data quality was becoming the main bottleneck. New cloud -native data technologies also required new approaches to data quality.

Soda Platform

Jake: Could you give an overview of Soda and the challenges it solves?

Tom: Businesses increasingly thrive on analytical data. Soda helps companies prevent bad data from killing good business. Imagine the confusion in a board room when a report indicates business is going down and that there is doubt if the data is correct or not.

Another example is a recommendation engine that is trained on bad data and starts to recommend the wrong products. For large retail shops, recommendations can drive up to 35% of the revenue. It's easy to see that this has a major impact and large potential damages.

The way we help is by checking new data each time it is produced. Engineers as well as analysts can configure what must be checked in a declarative checks language. There is the ability for both automated and very explicit checks.

Jake: What makes Soda stand out from the competition?

Tom: The declarative language in which both engineers and analysts can express data quality checks. Many people in the organisation know and work with data, but often only a fraction of them know how to code in, say, Python. The SodaCL language is created so that all those people can contribute checks and keep the data in great shape.

Our tooling helps data teams take ownership and accept contributions from anyone in the organisation. A second unique aspect is that we have a built-in metric store that enables anomaly detection and other change over time thresholds for checks.

Jake: What is an ideal high-level workflow for using Soda in a Data Platform?

Engineers embed a Soda scan into the data producing workflow. The check files can be added to the existing git repository file structure. Engineers can add checks for all the assumptions they make for the correct operation of their pipelines, for example, schema, uniqueness or null checks. After that, analysts can propose checks to be added. If they can write SodaCL, they can do it self serve.  

Analysts without software engineering skills can contribute data domain knowledge by asking engineers to add a SodaCL check for example: "In shops where the volume is greater than 10 million, all critical customers must have field technical contact filled in with a valid email address". Soda excels at bringing the producers and consumers together and allowing them to build out a decent suite of checks that prevents bad data.

Jake: The Soda documentation mentions you can integrate Soda throughout the Data Pipeline; could you give a high-level overview of how you use Soda at data ingestion, transformation, and serving?

Tom: The most common way to integrate Soda is by adding scans as steps to your orchestration job. A scan is a single execution of all checks related to the data produced in that pipeline. Right after ingestion is a very good place to run a first scan. And right before data is passed to consumers is another good point in the pipeline where a scan is appropriate. These two places provide a very good basis for proactively finding and later debugging data issues. If you want, you can go more fine grained, but we only see that in very data-mature organisations.

Jake: Soda now supports high-code with Python and low-code Data Quality checks with YAML. What are the use cases for both?

Tom: All checks are authored in SodaCL, which is a YAML based language. The execution of a scan can be coded in Python. That makes it very easy to embed scans anywhere in the pipeline and enables advanced features like setting custom variables.

Jake: What have been the most interesting and/or exciting use cases of Soda you've come across?

We've seen often that SodaCL is exposed to data consumers like analysts. They don't all have the skills to put code in production. But by deploying SodaCL checks, they can manage self-service. This removes a large part of the bottleneck for engineers that otherwise get constant requests to create or update checks programmatically.

Common Data Quality Implementation Issues

Jake: From personal experience, it can be hard to get investment in Data Quality until something goes badly wrong in the organisation. Do you have any advice to give on getting investment in proactive Data Quality efforts?

Tom: I think those times have changed. A few years back, we called this 'a compelling event': an data issue that got management exposure, leading to more awareness that data quality is important. Today, we see that almost all companies experience data issues regularly, and there is already much more awareness that data quality needs to be addressed as part of their day-to-day operations.

Jake: I've found in the past that extensive Quality checks can be expensive to run; how does Soda do anything to reduce this impact and/or have any thoughts to reduce the economic impact of DQ tests? 

Tom: Great question, as I love to talk about this part. We spent a lot of effort optimising the load on warehouses to run data quality checks. It starts by leveraging the data inside the existing SQL engines, which prevents any unnecessary moving or copying data. But the crux is that Soda can process a large set of checks in a single scan. That allows for optimisations. Many checks are based on the same metrics. We ensure that those are only computed once. And the biggest cost reduction comes from merging many metrics into a single query.

By grouping metrics in the same query, we drastically reduce the number of passes that the warehouse has to make on the data. It's not the first part when explaining data quality and Soda, but a very important topic when adopting data quality at scale.

Note: Tom also mentioned he’s working on a article on this subject. I’ll add the link later when it’s uploaded.

Jake: Another issue I've come across is that I've seen organisations take a techno-centric approach to improving Data Quality, which can lead to poor adoption of Data Quality standards. Do you have any advice on this?

Tom: SodaCL has been designed exactly to cope with this. It's a compact DSL for engineers. That means that engineers can write advanced data quality checks in a few lines of SodaCL making them very productive. But at the same time, SodaCL is very readable and enables the more tech-savvy analysts to become self service. Self-serve authoring of checks with Soda Cloud goes beyond the engineering-only approach and opens up the data quality to more people, but all on the same foundation.

Future

Jake: What can we expect in the future from Soda in terms of features?

Tom: We are looking into applying data quality checks on streaming systems so that we can detect issues earlier in the pipelines.

Jake: Do you think this wave of generative AI technology will change how people approach Data Quality? If so, how?

Tom: Absolutely. Chatbots have been added in more and more applicable use cases. For data quality and SodaCL in particular, generative AI has proven a tremendous help for the many non-tech people that know the data domain. They can just use natural language to describe the check they want to build and our SodaGPT feature translates that into SodaCL checks. This expands checks authoring even further to include more people who have intimate knowledge about the data.

Jake: Finally, any trends do you think will emerge in data and/or Data Quality in the next one to two years?

Tom: Embedding data quality checks will become the most normal thing to do when building or changing data pipelines. The abnormal thing will be to skip that part.

We also believe that there will be a big change in how data ownership is handled. At the moment, it's really hard for engineers to take full ownership of the data they produce because they don't know all the details and guarantees of the data that they use as input to their pipelines. If producers upstream provide formal data contracts, then engineers can also take ownership of the full datasets they produce in their own pipelines. That's definitely a trend to watch out for.

Starbursts Smart Indexing and Databricks Liquid Clustering: Using Query Metadata to Improve Performance

Partitioning is a very popular technique to use on large datasets to save time and money on compute. It works by splitting up your datasets into files by columns that get filtered (WHERE clauses, etc.) most often by queries, so compute only has to read a subset of the data (say, the last 7 days of data, if partitioned by day) rather than scan all data.

Though they can be tricky to configure, as your query patterns on a dataset might change as the data and your business evolve. This is especially problematic as partitioning on columns that don’t get filtered often in queries can increase costs.

As a solution to this, Starburst has brought out “Smart Indexing” and Databricks “Liquid Clustering” within weeks of each other. They have different names but have the same rough idea: review the history of recent queries run on the dataset and build a partitioning/clustering strategy from that, so your partitioning strategy never goes out of date.

I suspect not everyone will need it, but pretty cool tech.

Building a Real-Time Analytics Database

This talk is another banger from the GOTO Conferences Youtube channel, albeit also a cleverly worked advert for StarTree by Vice President of Developer Relations, Tim Berglund: it takes the viewer through the decision making process of building a Real-Time Analytics Database in a Choose Your Own Adventure style, which made the video a fun and informative watch for anyone intrested in data, not just people interested in building or using Real-Time Analytics Database.

What Is Query Driven Data Modeling?

Another insightful article from Data Consultant SeattleDataGuy/Ben Rogojan again showing why he’s one of the more popular voices in Data Engineering.

Here, he has a balanced take on the relatively new practise of just coming up with a new data model, pipeline and dashboard when it is asked for by data consumers:

Now I’ve seen a lot of angry responses to this lack of Data Modelling, especially from Data Architects, as frankly, it outs them out of a job somewhat. Also, this style of modelling likely doesn’t work in regulated industries, especially banking, where it’s recommended to show the data lineage of your financial reports.

Can you imagine performing data lineage on thousands of pipelines, most of which were created to answer one question? Even if you use technology that can easily capture data lineage, it’s still likely to be expensive to perform and difficult to look through it all.

And that’s before mentioning how more expensive this all could be, as Ben calls out in his article.

However, there is something very agile, lean and “just in time” about this method: you’re doing just enough work to complete the task and no more. It will also be very appealing to industries like retail and advertising that need analytics in minutes or hours and not want to wait months or even years for a team of Data Architects to work out what data model they should have.

That said, if you do take this methodology, I’d recommend viewing it like a support function, where you hold frequent retrospectives to look at how you can improve the “time to insights” by pre building some tables or making analytical data more consistent, which requires, of course, Data Modelling.

What Are “Data Clean Rooms”?

While this article by Investigative Data Journalist Jon Keegan, is aimed at a more general audience, I felt it was still worth sharing as Data Clean Rooms solutions are everywhere (AWS, GCP, Snowflake, Databricks) these days and don’t think are well understood.

This article also asks “Why Are These Becoming More Popular?“ (legal requirements), “Are Clean Rooms a Silver Bullet?“ (no) and the numerous privacy concerns associated with using them.

Reverse ETL 101

I’ve written about Reverse ETL before, where you send data back to the applications / operational plane rather than out, but Madison Schott, author of The ABCs of Analytics Engineering, does a much deeper dive into the topic, especially focusing on how to use it with dbt.

Building and operating a pretty big storage system called S3

A monster 6k article from Andrew Warfield who is a Distinguished Engineer at Amazon, give insight into how S3 operates. Lots of great technical information here, but the most interesting points for me was how to manage people and yourself when working on such a large and complicated service.

Sponsored by The Oakland Group, a full service data consultancy. Download our guide or contact us if you want to find out more about how we build Data Platforms!

First, let’s talk about what aspects of a Data Platform often get automated:

• Updates to the Data Platform code: data transformations, pipelines and infrastructure

• Schema migrations

• Security Scanning (scanning for vulnerabilities in internal and external code libraries and platform configuration)

• Code Linting for code consistency and complete documentation

• Testing

All of the above can be done manually, but it requires a developer to actively do it. So if you automate the above tasks, you potentially free up more of your developer’s time, though we should remember that sometimes a task is more effort to automate than to do manually:

But time isn’t the only benefit, traceability is another major benefit, as manual tasks are hard to audit. For example it may be difficult to trace when and who made the change that caused issues with the Data Platform, but with automation, you can add monitoring of updates.

There can be a security benefit too to automation: the more you automate, the fewer changes need to be made directly to production, so it can be possible to remove developers direct access to Data Platforms making your platform more secure as you reduce the attack surface.

It is often recommended for any production software system that access is limited as much as possible; only the automation server has access and time-limited access for developers in emergencies.

What is DataOps and why might I need it?

DataOps translates to Data Operations, which is the act of doing automated data operations on a Data Platform so the changes are done faster, more securely and more reliably than doing the same operations manually.

It also sets out an agile culture of shipping early to get customer feedback as early as possible and a process for making and tracking changes in small iterations, so you can adapt quickly to business and technology changes while maintaining quality software.

It is based on DevOps, which is Developer Operations, and to be honest, the differences between DevOps and DataOps are not massive.

We prefer to have data teams implement their own DataOps processes, with the Platform Engineering team providing common services such as user admin, firewalls or logging across the organisation (though in small to medium organisations, data teams may manage the platform as well).

Having a separate team for building DataOps processes can create a slower, more waterfall process where the data has to wait for the DevOps team to make their changes for any changes in infrastructure or automation.

DataOps Manifesto

While every organisation will implement DataOps differently, there is a DataOps manifesto that can be an excellent starting point for most teams.

Though at its core the DataOps philosophy to me is these 5 main points:

1. Always design, build and support while thinking about the customer and business requirements.

2. Get feedback as often as you can from all stakeholders to continually improve.

3. Data teams need the power to build, run and test as much of their solutions themselves as possible: waiting for IT, Security, Business Domain etc. will slow them down, reduce innovation and fail to keep up with business and technology changes.

4. Embrace changes as technology, data and the business will never stop changing.

5. It’s a team effort - balance sufficient communication while allowing time for people to get work done.

Achieving the above can be difficult in certain organizations, especially in highly regulated environments, so adapt as needed, but we think this should be your ultimate aim.

In our experience, the hardest change is culture and process, as we’ve seen many DataOps or DevOps initiatives fail as they were shoehorned into a Waterfall process.

The other thing to note is that the faster you develop, the happier your developers and data consumers will be:

• Developers want to build cool features they can deploy quickly, not be stuck doing monotonous manual tasks

• Data consumers want the platform to stay up-to-date as possible so they have most up-to-date data to make decisions.

Happier employees are more productive, creating a positive feedback loop.

DataOps Metrics

One of the core tenants of DevOps and DataOps is to continually improve developer practices and metrics can help keep track of whether your team is continually improving or not.

You do have to be careful with productivity metrics, as sometimes there can be mitigating circumstances that reduce developer productivity, like illness or especially difficult requirements.

But without hard data, it is hard to know whether any changes to developer practices and process are working.

But what sort of metrics could you use to track? Well, we can take some inspiration from DevOps metrics and adapt them to data:

• Number of changes per developer that cause issues over a time period: ideally, you’ll want this to be stable at a low level; if you see this rise then you may want to prioritize reliability over new features until this number comes down.

• Deployment frequency per developer over a time period: or the number of changes or improvements pushed to the Data Platform. There can be number of reasons for this to drop, so view with caution, such as an issue with the DataOps process, the Data Platform technology or requirements, among others

• Lead time for changes: how long does a requirement go from being accepted to being deployed into production? Very important, as the quicker you can react to changes made in the business, the more valuable your Data Platform is.

• Time to restore service: how quickly can you recover from failure? It can be important if the analytical data you hold is of critical importance to the business, though it’s often expensive to have backup servers and databases to recover from failure quickly.

• Number of Data Quality issues per developer over a time period: how trustworthy is your data? Again, if you see this rising, then you may want to prioritise data reliability over new features until it comes down.

Note that it costs money and time to track these measures, so we’ve come across many Data Platforms (usually small, older platforms) that do not track the above fully or at all and therefore use the “gut feel” of the Engineers and Managers to decide when to prioritise reliability over new features.

Though large, complex Data Platforms will be more in need of the above because one person or team cannot truly grasp what is happening until they start measuring the above, it also gives you hard evidence when building a business case for Data Platform improvement and investment.

How do you Automate Updates to Your Data Platform?

First, start with building a pipeline to put all your deployment tasks in: Continuous Integration/Continuous Development (CI/CD) pipelines automatically deploy infrastructure, code, database schema and data loading in one pipeline.

The pipeline can be kicked off automatically by a code change (commit) or manually.

You’ll build the infrastructure for your Data Platform first and the most automated and reliable way to build infrastructure such as compute (servers), data storage, databases and networking between them is with Infrastructure as Code (IaC).

Then you can deploy your software (Data Pipelines and calculations) on top of the infrastructure.

It is also common to run a battery of tests after the infrastructure and software are deployed to a test environment: this will ensure your Data Platform works as expected and meets its requirements.

Once the tests pass, there is likely one or more manual review processes before deploying to a staging, User Assessment Testing (UAT) or production environment.

One Way and Two Way Decisions

You’ll likely want to have more checks for certain kinds of changes that are hard to reverse, such as large data migrations, which Amazon Engineering calls ‘one-way decisions’. They also call changes that can be reversed ‘two way decisions‘ and these changes should be more automated with less checks as they can be reverted.

It’s goes without saying you should always be looking to turn one way decisions into two-way decisions wherever possible.

How many environments do you build? The balance between infrastructure cost and developer productivity

This is a bit of a digression, but a important one, as one of my biggest pet peeves is not having the budget for enough environments for a Data Platform, slowing down developers who cost 10x to 100x more per hour.

If you build just one environment for your Data Platform, you have to be very careful that any changes you make will not break the Data Platform, especially any changes that are irreversible (for example, deleting data) though try to keep any irreversible changes to a minimum as well!

So you may build a test and/or staging environment so any changes can be tested before going live; however, this doesn’t scale on it’s own as you’ll likely want to develop as well as allow for testing and you don’t want your development to affect testing, so you build a development environment as well.

Now you have two or three environments, but more may be required with a large team of developers as one developer could make a change that affects other developers, blocking work all other developers or worse, confusing developers if they have a issue (is it my change or someone else’s change that caused this?).

So ultimately you may have two to three environments plus one for each developer. This starts to cost a lot, not to mention the maintenance overhead. But there are lots of ways to save money here:

• Automated IaC builds often only take minutes or hours to build: because you only maybe need production up 24 hours a day, 7 days a week, you can build the environments when you need them, dramatically saving infrastructure costs.

• Developing and testing on a subset of data to reduce processing costs (or a subset of synthetic or anonymised data for security reasons). The downside to this is could miss some critical issues that are caused by a part of the data you did not test.

• You may only build just two Stream Processing, Data Warehouses or Lakehouses - one for production and another to be shared across testing and development with different schemas. Maybe even just one: though it does make infrastructure changes more risky, trading off the increased risk for lower costs.

• Some data vendors make switching environments fast and cheaper by offering “Shallow“ clones of your data: Snowflake Zero Copy Cloning or Delta Lake Shallow Cloning are two examples.

Speed vs. Reliability vs. Budget Trade-offs

Getting this balance right is difficult, as attaining 100% reliability in a complex platform probably means you are pushing virtually no changes and improvements to the platform - this can make a Data Platform out of date quickly and irrelevant to the business.

However, having high reliability is important for people to trust your platform and is critical when data is used in safety systems where quality of the data is a life or death issue.

Investing more in your DataOps processes can allow you to move fast but in a reliable manner. However, it can be expensive and may take a while to show value as your engineers adjust to any new processes or technology.

It finding the right balance isn’t easy, though can easier with metrics as discussed above.

Summary

This was a big post, as DataOps will be implemented differently in every organisation, and even every engineering team in a organisation and I wanted to cover all the major tradeoffs and decisions you make.

I hope this was useful and inspired you to build more efficient and reliable Data Platforms! Feel free to comment if you think I missed anything or have any extra advice on how to implement DataOps.

Sponsored by The Oakland Group, a full service data consultancy. Download our guide or contact us if you want to find out more about how we build Data Platforms!

• A short rant about the Medallion Architecture

• Is Kimball Still Relevant?

• Hello, World of CDC!

• Grai: Open Source Data Lineage

• You Can’t Master Data in a Database

• Data Parallel, Task Parallel, and Agent Actor Architectures

• Standardized Data Product Metadata Examples Based on Real-World Published Data Products

What makes the Medallion Architecture Different?

I’ve seen a few comments on social media about Medallion Architecture, saying it looks no different from the classic three-tier structure of Raw, Conformed and Enriched found in batch Data Warehouses and it’s just meaningless buzzwords created by Databricks to get more sweet Venture Capitalist money.

Could Databricks come up with less vague layer names than Gold, Silver and Bronze? Maybe, I suspect they wanted a set of names that doesn’t tie themselves to one modelling style, to show how flexible Medallion Architecture is (and sell more Databricks).

But it doesn’t change the fact Medallion Architecture does differ from other architectures.

What makes the architecture different is that Databricks supports both batch and streaming using the same technology across all three layers, whereas classic Batch, Lambda and Kappa Architectures have separate batch and real-time processing technologies.

This arguably makes it better than all of the above as it’s flexible in supporting streaming and batch while having lower maintenance than Lambda and Kappa as you only need one data processing product, not two (though you’d still keep the streaming and batch pipelines separate).

And just so it doesn’t look like I’m just shilling Databricks, this architecture can also be applied in Apache Flink and maybe Snowflake?, if Lakehouses are not your thing.

Though I will say if you’re doing no streaming processing, then yeah, it’s just classic batch processing.

Is Kimball Still Relevant?

Joe Reis, co-author of the excellent Fundamentals of Data Engineering book, woke up in a fiery mood on Friday:

Here’s the deal. If you’re aware of the various data modeling approaches and can pick the right approach for your particular situation, terrific. You’re a competent and thoughtful professional. To completely ignore data modeling is professionally negligent, and I’ll argue you’re unfit for your job. We can do better as an industry. Don’t burn down data modeling just yet…

I’m tempted to frame the above paragraph. The rest of the post is just as good.

I will add this though: while the world of Data Engineering (DE) may feel a bit lukewarm on Kimball models as there are some arguments it doesn't scale as well as Data Vault, Activity Schema or One Big Table, I feel Kimball is more in use than any other point in time due to being the default way to model data in Self-Service Business Intelligence applications (BI): Power BI and Tableau.

And BI is 10 times bigger in usage than DE, I say that as a DE myself.

Though if you hate the idea of Kimball models in your BI apps, I’d check out Narrator, which uses Activity Schema.

I think an argument can be made that we're living in an era where it is common in a large organisation to use multiple types of data models, whereas 15 to 30 years ago you could only use Kimball and you'd be called crazy to question it (though I could be wrong, I was still in school then!).

Hello, World of CDC!

I’ve covered Change Data Capture (CDC) in previous issues, but this three part series (so far) by Ryan Blue, former Senior Engineer at Netflix and now CEO of Tabular, goes arguably into more depth about implementing CDC, what issues you might run into and how to solve them.

Grai: Open Source Data Lineage

Grai is a new start-up offering Open Source Data Lineage with a cloud option. It also has features to show the downstream impact of failing Data Quality tests.

You Can’t Master Data in a Database

This is a great article on something that I’ve been thinking about for awhile: Master Data Management (MDM) / Customer 360 / Single View of the Customer should be done as close to the operational data processing as possible rather than implemented post import of data into an Analytical Storage.

You want to master data at the source, or as close as possible to the source so data duplicates have less impact than if data is exported to the analytical plane to be mastered. Steve Jones of Capgemini lists the above and many other reasons why MDM is the solution to a business operations problem and not an analytical data problem.

Though, I will argue that it can be hard to get this view across in a large organisation, so MDM ends up closer the analytical data because because that is where the most pain is felt of having no mastered data.

This article is also great and on a very similar theme, talking about putting MDM in the close to operational data in a Data Mesh context.

Data Parallel, Task Parallel, and Agent Actor Architectures

If you’re a big nerd like me and want to know how distributed data processing solutions like Spark, Flink and Ray work under the hood, this is the perfect article for you.

Zander Matheson, Konrad Sienkowski and Oli Makhasoeva of the Streaming Processing product, Bytewax, go through three common types of distributed compute, each of their pros & cons and what their use cases are.

Standardized Data Product Metadata Examples Based on Real-World Published Data Products

While there is a lot of talk about how Data Products in Data Mesh should have consistent metadata model across the organisation, we haven’t seen many examples shared in public, likely because organisations that adopted Data Products don’t want to share their meta-model for fear it would give away company secrets or increase security risks.

To help organisations figure out what metamodel their data products should contain, Jarkko Moilanen of API Economy Hacklab, has co-authored an open source specification with a few examples.

While the specification looks like a great starting point, I will say I don’t think this is the final say on the matter, as I would like more detail on the Data Quality section, including what tests are run.

Sponsored by The Oakland Group, a full service data consultancy. Download our guide or contact us if you want to find out more about how we build Data Platforms.