First off, I promise no World Cup references in this edition—except to say that the Neo4j-powered Global Chalice is absolutely worth checking out if you enjoy mixing data with soccer football.

The first part of this edition will cover a broad topic: internal data tools and how their position and value are (very) quickly evolving as AI disrupts software in general. The second part will then zoom into the implications of this shift for one category of internal tooling: BI, and my perspective on how & why the acronym should shift from business intelligence to business intent. This connects with my recent article in Towards Data Science: "BI is Dead, Long Live BI."

So, without further ado, let’s talk data & analytics while the espresso is still hot.

Internal tools, anyone?

Historically, and specifically during the Modern Data Stack era, data teams were particularly spoiled in terms of the footprint of their internal tooling. The data stack was usually a collection of 20+ tools and open-source packages, moving, transforming, and interacting with data across a lengthy (and often unnecessarily complex) path.

This tool sprawl resulted in a bloated stack with many moving pieces, but also many rough edges where a gap between the features of two tools remains open or a critical piece of functionality remains elusive. An ideal solution to circumvent this would be developing lean in-house tools that fill these specific gaps, but realistically, no one had time for that—we were all, collectively, jumping from one broken dbt model to another while updating a query filter and adding a new Fivetran connector to a spaghetti-like Airflow DAG.

Around this time last year, I wrote about how AI can be a true enabler for these “missing last-mile pieces” because it writes software that’s good enough for internal usage, where bugs are usually acceptable and downstream dependencies are limited (so tech debt is cheap). I used the example of a dbt SLA tracker tool. The tool aimed to recreate Airbnb’s brilliant data timeliness UI, which features sophisticated lineage and neat visual styling to show which specific jobs/steps are causing SLA misses for a given data pipeline.

The result last year was (barely) acceptable. I had functioning lineage at the dbt-model-level with nice coloring based on model run metadata, but it was clunky and lacked all other functionality. Getting to that result took a few hours of vibe coding with Replit + Claude Sonnet 3.7.

This year, around the same time, knowing the massive leap AI models had made in coding, I tried the exact same exercise with Claude Code + Claude Opus 4.8 Max. The results were simply mind-blowing. With just the Airbnb blog post and my ideas as a starting point, it built an end-to-end tool with a very sophisticated (yet elegant) and intuitive UI that covers all key features discussed in the blog post. I was able to add rather complex functionalities (running via Docker, refreshing data via a button, providing YAML config, etc.) with just one prompt at a time, without any significant corrections or back-and-forth. (The full GitHub project is available here.)

• The overview page for a given dbt project:

• The timeline page for a particular run:

• The other pages of the app exhibit the same level of “sharpness”, all courtesy of Claude Opus 4.8 raw capabilities.

This result, given the current capabilities of state-of-the-art models, isn’t surprising (and partly made Anthropic’s near-1-trillion valuation somewhat reasonable). But it made me realize an uncomfortable fact about where the future of internal tools lies.

Claude built a fantastic UI, sure, with a brilliantly engineered backend (I checked!). But at the end of the day, my actual goal of “which models are slowing my pipeline and what are my SLA trends?” becomes totally answerable without any of these mechanics. They turn into redundant code that serves no purpose other than vanity “oh cool UI!” dynamics. Claude can just look at the raw dbt runs metadata, figure out everything worth figuring out (either on its own or based on my guidance), tell me about it, and—in 95% of cases—make the most optimal code and process improvements directly.

Internal tools were built for a world where humans needed to find the insight on their own, making every visual cue helpful. But AI agents don’t need a UI; they extract insights directly from the raw data. So, as access to agents gets democratized, internal tools ecosystem... quo vadis?

The BI link

This realization is even more pertinent when we think about BI tooling specifically, which for decades (disappointingly) served the sole purpose of “this data says X, and I can’t tell you anything beyond that or why you should care”. If Claude can look at the data and surface everything worth surfacing, what’s the benefit of maintaining an expensive and demanding BI platform?

The nuance here, however, is that running too quickly to the “ok let’s replace BI with AI chatbots” path misses the whole point of BI. The key question all BI tools ultimately should aim to help answer isn’t “what’s the data answer to this specific question?”, but instead “based on the data, what questions should I be asking?”.

Right now, dashboards suffer from what I call the “closed-window problem”: they only show you what someone already decided to measure. They can’t surface the patterns no one thought to ask about. To actually answer “what should I be looking at?”, we need to stop optimizing the query layer and start building for the business intent layer (defining the business outcome, and letting the agent watch the data autonomously).

This exact shift (moving the acronym from Business Intelligence to Business Intent) is the focus of my latest article in Towards Data Science. In it, I dive into the future of BI in an AI-Agent-powered world: where it’s currently headed, why intent is the missing piece, and what the new model actually looks like in practice.

Hope you enjoyed this edition of Data Espresso! If you found it useful, feel free to share it with fellow data folks in your network.

Feedback is always appreciated as usual, so feel free to share your thoughts in the comments or reach out directly – I’d love to hear your take on the edition’s topics!

Until next time, stay safe and caffeinated ☕

We are officially settling into 2026, and I hope the start of the year has been kind to you all!

This edition will be a deep dive into just one topic. I’ve spent the last few weeks reflecting on where data teams stand today, and I believe we are at a critical crossroads: We are paralyzed by tech debt and “model sprawl” (thanks MDS!) right when the business needs us to be more agile than ever. So, instead of a news digest, I’m sharing a survival kit for the next era of data—covering the strategic pivots we need to make to escape the “service trap” we’re in today and build the missing foundations for an era in which AI agents will be the main (only?) data consumer.

So, without further ado, let’s talk data & analytics while the espresso is still hot.

The Survival Kit: 6 Imperatives for the Agent Era

The dust from the Modern Data Stack era is finally settling, and it has left us staring at a very uncomfortable reality: the architecture we spent the last five years building is fundamentally incompatible with the future we are now living in.

For half a decade, we (data teams) optimized for volume. We scaled our warehouses, treated every problem as a nail that needed a dbt hammer, and bought a specialized SaaS tool for every minor step in the pipeline. We built an ecosystem that was easier to manage than Hadoop, sure, but we optimized for human eyeballs—building dashboards that, let’s be honest, often went unread.

But the new primary consumer of data isn’t a human analyst anymore; it’s the AI Agent. And unlike a human, an agent doesn’t care about your star schema or your “best-of-breed” lineage tool if it lacks the unified context to reason about the data.

Yet, instead of moving fast to prep for this shift, teams are bogged down managing thousands of fragile models and a sprawling vendor list. We were caught off guard: We built a stack for 2021, and it is failing us in 2026. (For a group of people who work with predictions daily, we turned out to be terrible at predicting our own future.)

And so, fellow data person, it is now time to pivot.

In this month’s deep dive, I am sharing a survival kit for 2026 and beyond, consisting of 6 strategic imperatives that data teams need to adopt to re-align the data stack with the new era of use cases:

1. Put the Stack on a Diet: The era of buying a specialized SaaS tool for every feature is over. If your platform offers it natively, use it—consolidation is the only way to lower the “cognitive tax” enough to move fast.

2. True Decoupling (Storage is Yours): To be future-proof, your central gravity must shift to Open Table Formats (Iceberg FTW!). Your data should live in a neutral state that any engine (Snowflake, Spark, or something completely new) can plug into without a migration.

3. Stop Being a Service, Start Being a Product: The “ticket-taking” model is the wrong path forward. You cannot survive by answering every ad-hoc question; trust is won through focused excellence on a few high-value Data Products, not mediocre ubiquity.

4. The Context Library: An AI agent is context-blind. We need a semantic layer that captures business logic, not just metrics—and the best way to build it is to have AI write the initial documentation for you.

5. From “What Happened?” to “What Now?”: We must stop using data just to validate past decisions. The goal is now Navigation—automating the response to a change in a given metric. This requires shifting from the passive BI model to automated AI-powered processes that perform data-driven actions.

6. The New Persona: “I write SQL” is no longer a career. The new data role moves Upstream (enforcing governance with engineers) and Downstream (acting as a PM to ensure data drives workflows).

I unpack the nuance of all these points (and more) in the full article on Towards Data Science. If you are trying to figure out how to position your team—or your own career—for the next 12 months and beyond, this is the full blueprint.

Hope you enjoyed this edition of Data Espresso! If you found it useful, feel free to share it with fellow data folks in your network.

Feedback is always appreciated as usual, so feel free to share your thoughts in the comments or reach out directly—I’d love to hear your take on the edition’s topics!

Until next time, stay safe and caffeinated ☕

We meet again after a long(-ish) break, mainly due to a busy few months on both the personal and professional sides. As you might’ve noticed if we’re connected on LinkedIn, I left my role at Sifflet and joined Neo4j to work on graph analytics (something I’ve always been passionate about!) - this won’t impact the content of the newsletter: I’ll keep discussing the overall data and analytics space and how to build data products & platforms at scale. (But maybe we’ll have a graph-related “twist” here and there, since I’m getting more familiar with graph analytics use cases.)

Now let’s briefly go through what we’ll cover in this month’s edition: we’ll first discuss the semantic layer renaissance (really, it’s back in style!) and why we need a new approach (other than defining aggregations and joins in YAML) for an AI-ready semantic layer, then we’ll go through how the data stack is shifting into a “hub and spike” model instead of a “shopping list” architecture, and finally we’ll end with a note on DuckDB’s new release: DuckLake.

So, without further ado, let’s talk data & analytics while the espresso is still hot.

The Semantic Layer Renaissance (and why it’s actually different this time)

Back in 2021/2022, during the peak of the Modern Data Stack craze (R.I.P.), the semantic layer felt a bit like… “unnecessary” homework.

In theory it was an essential component, sure, and I was a strong believer in its value - but it ultimately consisted of thousands of YAML lines just to define how customers joins with orders. The promise was “metric consistency,” but deep down, many of us knew it was overkill: The use cases weren’t there yet, and all the semantic layer was doing was adding friction between a BI tool (where analytics lived at the time) and a data warehouse.

The friction was unnecessary because human analysts are smart - they have implicit context. They know not to count test accounts in churn figures. They know who to ask if a number looks weird. Building a massive configuration layer just to save them a JOIN often didn’t feel like high ROI.

But the landscape has shifted.

We are moving from a world of Analytics (humans looking at charts) to Agents (AI taking actions). And unlike human data analysts, an LLM is context-blind: it doesn’t know your business nuance. So if you feed it raw SQL snippets or simple metric definitions, it will hallucinate.

This is where the need for a “new” semantic layer comes in: it’s no longer about optimizing for SQL generation; it’s about optimizing for reasoning. It shifts the job from defining the Math (”Here is the formula for Churn”) to defining the Meaning (”Here is why we track Churn, who owns it, and what business goal it impacts”).

A couple of weeks ago, I wrote a deep dive on Medium about this shift. If you are building for AI, or just tired of writing YAML for dashboards that nobody uses, give it a read: Building a Semantic Layer for the AI Era: Beyond SQL Generation.

The Death of the “Shopping List” Architecture: Why You Just Need a Platform (and a Few Exceptions)

For years, we treated data architecture diagrams as a shopping list - the standard advice was to buy (or build, for the brave souls) a specialized “Best-of-Breed” tool for every single area of the stack: A data catalog, an orchestrator, a data observability tool, etc. But in 2025, with data platforms like Snowflake and Databricks rebundling the stack (via features that cover everything from orchestration to metadata management), the “baseline” requirements for governance, observability, and execution are now built-in - especially given how AI is minimizing the surface area of human involvement in data work. And so if you’re just starting out, buying a dedicated catalog before there’s a concrete need for it is definitely not the best place to invest money or resources.

Instead, we are moving toward a “Hub and Spike” architecture. The “Hub” is your massive, all-encompassing platform that handles the boring, day-to-day utility work. The “Spikes” are the specialized tools you buy only when you hit a specific, wall-sized problem that the platform can’t solve - like orchestrating a job across an on-prem legacy component and Salesforce, or governing 20 years of legacy Oracle data.

This marks a subtle but critical shift: Yesterday, we bought tools to build a Stack. Today, we buy tools to fix a Spike. You no longer buy a tool just to “fill a category” (e.g., “we need a catalog”); you buy it to solve a bounded, painful edge case that your central hub can no longer contain. In this context, my recommendation (for most cases) is the following: start with the platform, and earn the right to buy the specialist.

The Duck Is Out of the Lake: What if Metadata Doesn’t Need to Be Distributed?

DuckDB disrupted the data industry a few years ago with a simple yet brilliant observation: most “Big Data” isn’t actually big. They proved that for 99% of use cases, data fits on a single machine, so using a distributed cluster makes no sense (to an extent). Now, with the introduction of DuckLake, they are applying that exact same philosophy to the Data Lakehouse architecture.

While the new wave of distributed table formats (hi Apache Iceberg 👋) treat metadata as “Big Data” (scattering it across thousands of distributed files in object storage), DuckLake asks the obvious question: Why distribute the metadata if it doesn’t need to be distributed? Their approach keeps the data distributed (Parquet in S3) but centralizes the metadata in a standard, ACID-compliant database. It recognizes that while your row count might be massive, your catalog operations are almost always “small data.”

This adds a much-needed nuance to the stack, challenging the “distributed everything” default that dominated the Iceberg/Delta/Hudi conversation. It’s a healthy correction towards simplicity that aligns better with how most teams actually operate. If you want to dive deeper into why centralized metadata might be the future (again), MotherDuck’s CEO Hannes Mühleisen gave a fantastic talk breaking it down.

Hope you enjoyed this edition of Data Espresso! If you found it useful, feel free to share it with fellow data folks in your network.

Feedback is always appreciated as usual, so feel free to share your thoughts in the comments or reach out directly – I’d love to hear your take on the edition’s topics!

Until next time, stay safe and caffeinated ☕

This month, we’re diving into the big comeback of data modeling (and how to adapt it to today’s data-product-driven world), the data product we all need (analytics and insights combining data from all the apps we use), and how to find data problems worth solving. So, without further ado, let’s talk data engineering while the espresso is still hot.

Data Modeling for Data Products: A Practical Guide

For the last couple of years, it feels like we’ve all time-traveled back to 1999. Nearly everyone in the data space is talking about Kimball, Inmon, and Data Vault again. This isn’t just nostalgia; it’s a very reasonable (and much-needed) reaction to the chaos of the last decade, where we threw endless compute and thousands of dbt models at every problem without a coherent strategy. After data budgets got much tighter in 2022, we all collectively realized that we needed a process to clean up the mess.

But here’s the tricky part: while we absolutely need the discipline of modeling, we can’t just copy-paste the old playbook. The slow, rigid, waterfall approach that defined the data warehousing era is not a great fit for modern data teams trying to efficiently build and ship data products at scale.

So, how do we bridge that gap? How do we take the timeless principles of good modeling and adapt them for a decentralized, product-driven world?

I decided to write down my full thoughts on this in a long-form article for Data Engineering Things. It’s a practical guide that lays out a modern playbook for data modeling, covering (among other things):

• A "Go Wide, then Go Deep" strategy for balancing big-picture business alignment with the focused work of building a specific data product.

• Why decentralized ownership isn’t just a buzzword, but a prerequisite for making this approach work.

• The specific tools and frameworks (Metric Trees, the Semantic Layer, etc.) that help turn these principles into practice.

It’s my take on how we move forward: keeping the good parts of modeling without the old-school baggage. If you’re grappling with these questions on your own team, I think you’ll find it useful. (You can read the full article here.)

A Spotify Wrapped for Everything Else

Every year, Spotify Wrapped comes out and provides users with a wide range of analytics about the music they listened to throughout the year - and unsurprisingly, people love it. It’s a genuinely great data product that gives you a fun, insightful look at your own habits. But it’s also a rare exception. We use tens (hundreds?) of apps and services on a daily basis, yet we actually get back very little data from them.

What were my most productive hours according to my calendar and code commits? How did my reading habits on Kindle correlate with my running activity on Strava? What does my Uber history say about my social life? Some apps do sprinkle high-level metrics here and there, but there are mountains of personal data that are yet to be used for anything other than ads.

Getting the answers today would mean wrestling with a dozen different APIs and trying to stitch it all together myself. Unfortunately, but unsurprisingly, nobody has time for that.

This is where I think the current AI wave gets interesting. Connecting the dots across various services and generating useful insights about our habits and who we are is a very interesting "last-mile" problem for AI. The idea of a personal AI agent that can securely connect to the APIs of all the tools I use, pull the data, and just tell me something interesting about myself feels... actually useful.

It’s not about creating more dashboards. It’s about getting a coherent narrative out of the fragmented data of our own lives. That’s a data product I’d actually want to use.

(This is after all a data engineering newsletter, so I can’t talk about Spotify Wrapped without mentioning the fascinating article about the data engineering magic behind it.)

Finding Data Problems that are Worth Solving

For a myriad of reasons, data teams built a reputation for being disconnected (to an extent) from the business. The narrative is that we get excited about the tech and the new tools - and sometimes lose sight of whether we’re actually having an impact. (i.e. we end up building things that are technically impressive but practically useless.)

Recently I came across a great article by Sven Balnojan called "Why Internal Data Teams Build the Wrong Things", in which he provides 15 extremely relevant and useful learnings to ensure you don’t walk the wrong path.

But IMO we can distill things even further. In my experience, breaking out of this cycle and building things that are valuable for the business can come down to a simple, two-step path.

First, you have to find the low-hanging fruit to win some initial momentum. There’s always a team that is obviously starved for data. A classic example is the marketing team trying to figure out which campaigns are actually working. Building them a solid attribution model is a clear, contained project with an obvious business value. It’s a quick win that builds trust and opens new doors with other stakeholders who may have yet more interesting use cases.

But the real magic happens with the second step: truly understanding the business by diving into the details, and finding new (relevant) data use cases. This means going beyond shipping what’s obvious/requested, and instead watching how the business operates and identifying gaps where the data can shine.

A perfect example is product analytics. Most product teams have a tool like Amplitude or Mixpanel that gives them great high-level metrics. But the real gold (the raw, granular event stream) is often just sitting untouched in an S3 bucket or a production database. The high-value data product here isn’t about replacing the existing tool, but building a much richer analytics layer on top of that raw data. By modeling it properly, the data team can unlock granular, feature-level insights that are impossible to get otherwise. This is how you empower the product team to go from asking "How many active users do we have?" to "How does the adoption of our new search filter impact 30-day retention for our enterprise customers?" while also building a foundation for a myriad of new vertical use cases for product data.

These are the kinds of opportunities you only find when you go looking for them - by observing how the business actually runs and identifying the real-world friction that data can solve.

Hope you enjoyed this edition of Data Espresso! If you found it useful, feel free to share it with fellow data folks in your network.

Feedback is always appreciated as usual, so feel free to share your thoughts in the comments or reach out directly – I’d love to hear your take on the edition’s topics!

Until next time, stay safe and caffeinated ☕

This month, we’re diving into how AI is finally democratizing the Data Platform’s “last-mile” layer, dbt’s new Fusion Engine, and why a healthy dose of skepticism is always needed when it comes to data. So, without further ado, let’s talk data engineering while the espresso is still hot.

How AI is finally paving the data platform’s "last mile" for everyone

We’ve all seen the incredible advancements in the data space over the past years – powerful warehouses with limitless scale, slick ELT tools, and dbt bringing software engineering rigor to data pipelines. Yet, that final layer of polish, the seamless “last mile” experience seen in Big Tech tools (like Airbnb’s data timeliness UIs or Netflix’s integrated notebooks), still feels out of reach for most data teams, often a luxury only Big Tech could afford. Even with all the capabilities of the Modern Data Stack, data teams are still swamped with firefighting and ad-hoc requests, leaving little room for building these bespoke experience layers.

But what if that’s changing?

In my latest article, "How AI is Finally Democratizing the Data Platform’s Last-Mile Layer", I dive into how AI is emerging as a powerful “last-mile enabler”. I experienced this firsthand via a personal experiment that I discuss in the article: I persisted the artifacts of a dbt project using the ‘dbt-artifacts’ package, and then using Replit's AI, I built a basic dbt run execution timeline visualizer in under one hour – a task that previously might have seemed daunting (especially when using something as “delicate” as D3.js). Sure, it wasn’t nearly as sophisticated as Airbnb’s internal platform, but it worked. I had a functional app showing dbt model lineage, run statuses, and execution times, built by one person, in less time than it takes to watch a movie.

This isn’t just about building cool internal tools faster. It’s about:

• Democratizing Excellence: AI is lowering the barrier of the experience layer, allowing teams of all sizes to achieve the kind of platform sophistication once reserved for tech giants.

• Boosting Productivity: Enabling data professionals to focus more on insights by spending less time wrestling with clunky interfaces, hunting for metadata, or navigating complex access procedures.

• A Richer Ecosystem: Potentially fostering more collaboration and a vibrant open-source landscape for these “last mile” components (imagine all the dbt packages waiting to be built!).

Of course, AI isn’t a magic wand (for now at least), and I touch upon the necessary cautions. But the potential for AI to help us finally build truly complete, user-centric data experiences is immense.

Want to explore the Big Tech examples, the “how-to” with AI, and what this means for the future of data platforms? Check out the full post on Medium here.

New dbt engine, who dis?

So, dbt Labs finally shared their detailed roadmap for the post-SDF-acquistion world, and it’s pretty exciting stuff! The star of the show was the new Rust-based dbt Fusion engine. This is a big deal for quite a few reasons: Immense performance improvements compared to dbt Core (up to 30x faster parsing), smarter SQL understanding for things like real-time code validation, and state-aware orchestration that can actually cut down your warehouse compute costs.

The new engine will co-exist with the Python-based dbt Core engine (which dbt Labs will continue to maintain), but will understandably have the more restrictive ELv2 license (you can use it internally but can’t just take it and build a competing SaaS).

Maintaining the two engines separately is a smart strategy. The dbt language stays consistent, which is key for “universal” dbt concepts. If you're already running dbt Core, you can continue as is, or you can choose to adopt the ELv2 Fusion engine for the performance boost without worrying about migration costs.

dbt Labs announced a wide range of dbt Cloud features that augment the new engine and provide impressive quality-of-life improvements to the dbt experience, like a supercharged VS Code extension and new/improved components like dbt Canvas, dbt Insights, and an expanded dbt Catalog.

In my opinion, dbt Labs pulled off something pretty impressive. Throughout the early years of dbt, the “distance” between dbt Core and dbt Cloud was relatively minimal, making it difficult to justify the cost of the “upgrade” from Core to Cloud. However, with this release (and the additional updates made throughout the past two years to areas such as the semantic layer), the value proposition of moving to dbt Cloud is stronger than ever. They’ve finally created that necessary distance between Core and Cloud to warrant internal discussions about upgrading for most data teams.

Although the open-source offering is still part of the picture, the big focus on dbt Cloud features means teams using dbt Core (or even the new Fusion engine) will inevitably start to wonder why they’re not using SQLMesh instead, which continues to champion a more “compelling”, feature-rich, open-source-first narrative.

A shot of skepticism: thinking critically about our data conclusions

As data professionals, we’re immersed in extracting insights and building data-driven solutions (or at least we like to think so). But this very closeness to the data can sometimes lead to a subtle trap: we might become a bit too quick to believe we’re drawing the right conclusions, simply because the data “said so”.

I recently read a fantastic book (which I bought a few years ago and forgot about) that serves as an effective reminder of the dark side of data: “Calling Bullshit: The Art of Skepticism in a Data-Driven World” by Carl T. Bergstrom and Jevin D. West. While it tackles broader themes of misinformation, its lessons are incredibly relevant for us data folks. (And it’s a fun read!)

The book highlights how easy it is for anyone—even experts—to be misled if we’re not rigorously questioning our inputs and interpretations. For us in the data world, this means constantly asking:

• Are we truly asking the right questions to begin with?

• Have we considered all potential confounding factors or biases in the data?

• Are we diligently distinguishing between mere correlation and actual causation?

Even with sophisticated tools and vast datasets, the critical thinking we apply—the art of healthy skepticism towards our own findings—is as important as ever. If you’re looking to sharpen your internal “bullshit detector” and ensure your data narratives are as robust as your pipelines, this book is a worthwhile addition to your reading list.

Hope you enjoyed this edition of Data Espresso! If you found it useful, feel free to share it with fellow data folks in your network.

Feedback is always appreciated as usual, so feel free to share your thoughts in the comments or reach out directly – I’d love to hear your take on the edition’s topics!

Until next time, stay safe and caffeinated ☕

This month, we’re diving into the inevitable Apache Iceberg era, the often-overlooked benefits of frequently revisiting design decisions, and the upsides of working in constraint-heavy environments. So, without further ado, let’s talk data engineering while the espresso (or cappuccino if you’re feeling adventurous) is still hot.

How to navigate an Ice(berg) age

Apache Iceberg is all the rage in the data space. After a half-decade showdown between Hudi, Delta Lake, and Iceberg, the latter has emerged as the de facto modern table format, becoming the central piece of the increasingly composable data stack. While there’s been plenty of great content about Iceberg’s ascent, in this edition of Data Espresso, I want to offer my perspective on two questions many data teams are grappling with:

1. “I bet on Hudi or Delta Lake, should I migrate to Iceberg?”

If you built your data platform around Apache Hudi or Delta Lake, you might be wondering how Iceberg’s dominance will impact your data platform in the coming years, and whether a migration is warranted.

Here’s a heuristic I’ve observed: once a data technology gets its own dedicated conference, you know it’s going to dominate the discourse (and feature prominently in data stack diagrams) for at least a few years. This "de facto standard" status translates directly into ecosystem focus. While Hudi and Delta Lake aren’t disappearing overnight, the reality is that new tools and integrations across the data landscape will likely prioritize Iceberg compatibility first.

This shift will result in some friction for Hudi/Delta Lake shops. It’s not necessarily that existing Hudi/Delta integrations will break, but rather that the Iceberg path will likely receive new features, performance enhancements, and smoother integrations more rapidly. You might find yourself waiting longer for support or missing out on optimizations readily available to Iceberg users — and this gap is likely to widen over the next few years. A good historical parallel is the Parquet vs. ORC situation: While ORC remained in use, Parquet consistently had broader and earlier support across the ecosystem, making it the smoother choice for many.

So, does migration make sense for you? I highly recommend a structured evaluation:

• Assess the tangible risks of sticking with your current format (e.g., integration limitations, potentially slower adoption of new query engines, developer friction).

• Estimate the potential benefits of switching to Iceberg (e.g., access to specific features, performance gains, simplified architecture, broader community support).

• Factor in the cost and effort of migration.

• Crucially, run a Proof of Concept (POC) with Iceberg on your specific workloads.

It’s also worth noting that Iceberg has rapidly closed the feature gap with Hudi and Delta Lake, thanks to significant contributions from numerous “big tech” companies. A use case where Hudi/Delta might have been the clear winner a couple of years ago might be well-served, or even better served, by Iceberg today.

Migrating now, if the evaluation points that way, means doing it on your terms – defining a timeline that minimizes disruption while positioning your platform to leverage Iceberg’s momentum. However, if your assessment clearly shows minimal risk and limited benefit in switching right now, your engineering resources are likely better invested elsewhere — don’t migrate just for the sake of it.

2. “I’m not currently using a modern table format, should I add Iceberg to my stack?”

This really breaks down into two sub-questions:

• Should I add any modern table format to my platform right now? (either Iceberg or Hudi/Delta Lake)
  This is the classic "it depends", and the answer needs to be based on the value the tool would bring to your specific use cases. Considering the capabilities of modern table formats (and their increasingly available managed offerings like Iceberg Tables on AWS, GCP BigLake, etc.):

  • How specifically would it benefit your platform and increase the business value derived from your data?

  • Can it streamline your architecture, perhaps removing redundant layers or simplifying data pipelines?

  • Does it unlock new use cases by allowing different compute engines (Spark, Trino, etc.) to seamlessly work with the same data?

  • Does it solve specific pain points you have today (e.g., schema evolution issues, partition management overhead, concurrent write conflicts)?

• If I decide to adopt a modern table format, should it be Iceberg?
  In most cases today, the answer is likely yes. Iceberg has achieved feature parity (or near enough) for many common use cases, and its widespread adoption means it’s the safest bet for future-proofing and ensuring broad compatibility across the data ecosystem.

Work through these kinds of questions honestly. The goal is to ensure your decision stems from a genuine, explainable need that Iceberg addresses, not just because it’s the technology du jour.

If you decide to adopt Iceberg:

It’s a fantastic technology that truly enables a more composable architecture (think true separation of storage and compute). However, be prepared for a journey. Expect some rough edges, especially with tooling maturity around specific engines or complex migration scenarios. Plan for a multi-phase rollout to de-risk the transition and gradually realize the benefits within your specific environment.

An exercise that always pays off: Revisiting design decisions

Within the data space, the focus is always on the new cool thing — be it a paradigm like data mesh or a technology like Iceberg. This focus, combined with a natural engineering tendency to build cool things with cool new tech (worrying about business value is never fun, after all), makes it easy to overlook significant improvement opportunities (in cost, maintenance, or performance) that aren’t necessarily "cool."

I recently came across a great article by GumGum detailing their switch from Snowpipe to a Data Lake ingestion pattern using Snowflake External Tables over S3 data. The result? A staggering 60% cut in their data ingestion costs. This might sound counterintuitive, since Snowpipe is often positioned as the go-to for easy and efficient Snowflake ingestion. But for GumGum, the combination of their scale, data structure (already nicely partitioned in S3!), and access patterns meant the Snowpipe approach – which involved copying data into raw, internal Snowflake tables – created unnecessary overhead and cost (think expensive table scans for processing and retention on unpartitioned raw data). By switching to querying External Tables directly (and so leveraging the S3 partitioning), they eliminated Snowpipe’s compute costs, avoided data duplication in Snowflake storage, and drastically cut down query times. It’s a prime example of how the ‘best’ approach is highly contextual and definitely not static.

As we chase the cutting edge, the tools and platforms we already use are constantly evolving: New features, pricing models, or even entirely new tools might make previously optimal design patterns suboptimal today. That’s why I believe every data team should schedule regular (every six months for example) review sessions to revisit their past design decisions and identify areas of improvement based on tech progress or changing business needs. Such sessions are a great opportunity to take a hard look at your current architecture:

• What are the major cost drivers?

• Where are the performance bottlenecks?

• What are the biggest maintenance headaches?

• Have the capabilities of your core tools (or viable alternatives) changed significantly? Are you leveraging their new features?

• Have your business requirements or data volumes shifted?

This deliberate review often uncovers areas where significant gains can be made, sometimes with surprisingly simple changes, driven by technological progress or evolving business needs.

Out of the comfort zone: The joy of working in a constraint-heavy environment

I recently watched a fantastic talk by John Crepezzi from Jane Street about how they built their AI coding assistant. What struck me was the sheer ingenuity required to operate within their unique context: a codebase predominantly in a niche functional language (OCaml) and an environment lacking some tools many engineers take for granted (like having an in-house version control system instead of a Git-based system).

On the surface, this type of environment could frustrate many engineers (myself included), especially if constraints feel arbitrary or lack clear business or technical justification. (As in: "Is this limitation truly necessary, or just historical baggage?")

However, if you get the chance to operate within well-justified, albeit tight, constraints, you might find that it pushes you to elevate your game by having to be creative and solving complex problems. You’re forced to:

• Think deeply about the core problem.

• Navigate ambiguity with more rigour.

• Be incredibly deliberate about architectural choices, minimizing irreversible "one-way door" decisions.

• Often, find simpler, more fundamental solutions when fancy off-the-shelf options aren’t available or suitable.

I’ve personally worked in a couple of similar high-constraint environments (my investment banking days come to mind - so fun). While the roadblocks were certainly frustrating in the moment, looking back, those experiences were incredibly valuable learning grounds. They force a level of resourcefulness, experimentation, and rapid iteration that you don’t always encounter in less-constrained settings.

Hope you enjoyed this edition of Data Espresso! If you found it useful, feel free to share it with fellow data folks in your network.

I always appreciate hearing your feedback. What’s your take on the Iceberg wave, revisiting design decisions, or working with constraints? Share your thoughts in the comments or reach out directly – I’d love to discuss!

Until next time, stay safe and caffeinated ☕

This month, we’re diving into how (and why) streaming is becoming more accessible in the data space and the fate of the data platform’s raw/bronze layer. So, without further ado, let’s talk data engineering while the espresso (or cappuccino if you’re feeling adventurous) is still hot.

Will 2025 be the year real-time analytics finally goes mainstream?

Since the Hadoop era and the early days of Big Data, one prediction kept coming back year after year: “Next year will be the year of streaming.” There was a constant expectation that eventually, most (all?) data pipelines would evolve to real-time patterns instead of batch processing. Yet, despite the hype, real-time analytics has largely remained limited to tech giants and niche industries with highly specialized streaming needs.

By this time, the general consensus in the data space is “you don’t need streaming” / “batch is (mostly) always enough”. However, I believe 2025 might finally be the year streaming makes sense for common data use cases. This year, two key factors might allow real-time analytics to break out of its niche and finally hit the mainstream.

Streaming’s unfulfilled promise

Streaming is exciting. The ability to process and analyze data at scale in real time, unlocking insights and enabling immediate action, has been a core promise of data platforms ever since the Hadoop era. But then the realities of its technical challenges, coupled with the lack of concrete and valuable-enough use cases, often lead data teams to the catch-all “But do we really need it?” counter-argument.

Technologies like Apache Storm and Apache Flink solved many of the technical hurdles, but the fundamental constraints remained: Building and maintaining streaming pipelines required complex infrastructure, specialized skill sets, and significant financial investment. As streaming for pure data movement gained traction in the software engineering world, with patterns like event-based architectures and a meteoric rise for Apache Kafka, the data world remained stuck with batch processing, and architectures like the one below became the standard:

This architecture often resulted in data arriving at the data platform’s landing zone (typically an object store) in real-time but only being moved and transformed in batches, often on a daily schedule. Two arguments typically justified such architectures:

• Ingesting and transforming data in real-time within the warehouse is more complex and expensive than batch processing.

• The value generated by real-time use cases doesn’t justify the added cost and complexity.

However, these arguments are rapidly losing their validity. In my latest article on Towards Data Science, I discuss two major shifts that could make 2025 a tipping point for real-time analytics.

Out of the comfort zone: One layer too many

Tech companies have spent the last few years championing "efficiency" — often by streamlining their organizational structure and removing layers (i.e., middle management) to cut costs. The data world is experiencing its own efficiency push, with a focus on building business-value-driven data pipelines and minimizing the number of tools within the stack — but there's one area that should be the center of our efficiency discussions: data layers.

While most high-level architectures showcase the classic three-layer medallion approach (bronze/raw → silver/curated → gold/consumption-ready), many data teams in practice maintain far more “implicit” layers. Sometimes, data is even replicated multiple times within each layer due to ad-hoc use cases and lack of governance. So, as we strive to reduce costs and increase velocity, maybe it’s time to challenge our traditional layering approach?

A recent article by Adam Bellemare and Thomas Betts dives deep into this very question, arguing that it's time to rethink the bronze layer of the data platform by shifting it left. They propose several compelling options for building data products before the data even reaches the data platform's landing zone, rather than simply dumping everything into object storage and leaving data teams to untangle the complexity. If you're working on a data platform and looking to optimize costs or streamline your architecture, this article is a must-read.

If you enjoyed this issue of Data Espresso, feel free to recommend the newsletter to data folks in your entourage.

Your feedback is also very welcome, and I’d be happy to discuss one of this issue’s topics in detail and hear your thoughts on it.

Stay safe and caffeinated ☕

One of the conversations that helped me the most during that time was with Anis, a good friend who had made his own successful transition from engineering to sales a few years back. Anis shared many valuable insights that felt like they should be readily available online, not confined to a one-on-one conversation. So, a few months after my own transition, we had another conversation, this time about writing this very article: a roadmap with practical guidance for engineers looking to make a career pivot in tech. In this post, we'll share our personal experiences – Anis's journey into sales and mine into product management – to give you a behind-the-scenes look at what these transitions entail. We'll cover:

• Why Change? Is a career pivot right for you?

• De-Risking the Change: How to research, gather information, and prepare for a smooth transition.

• Two Journeys: Real-world examples of moving from engineering to product and engineering to sales.

• Navigating the New Role: Tips for thriving in your first few months.

• If It Doesn't Work Out: Strategies for adjusting and finding your path.

• Key Takeaways: Lessons learned and resources that helped us along the way.

Our goal is to provide a candid, practical guide for anyone in tech considering a career change. Let's dive in.

Part 1 - Is a Career Pivot Right for You?

Before we dive into the "how" of a career change, let's address the crucial "why." Switching roles, even within tech, requires time, effort, and a willingness to embrace the unfamiliar. So, how do you determine if it's the right move?

Start with Self-Reflection:

• Envision Your Ideal Day: Imagine your perfect workday. What are you doing? Who are you working with? What impact are you making?

• Identify the Gaps: Honestly compare that ideal day to your current reality. What's missing? What aspects of your job do you genuinely enjoy, and what do you dislike? Are there recurring patterns?

• Revisit Your Motivations: Think back to why you chose your current career path. What's changed since then? Have your priorities shifted?

• Consider Your Long-Term Goals: Where do you see yourself in 5 or 10 years? Does your current trajectory align with that vision?

Gather External Insights:

Introspection is crucial, but don't stop there. Seek diverse perspectives to gain a more complete picture:

• Learn from Others' Experiences: Read books, articles, and blogs by people who've made similar transitions.

• Tap into Your Network: Talk to people doing the job you're considering. Ask about their daily routines, challenges, and rewards.

• Consult Your Mentors: If you have mentors, solicit their advice. They may offer valuable insights based on their own experiences.

• Sounding Board: While your friends and family might not grasp the intricacies of your industry, they can still offer valuable support and help you process your thoughts.

Three Potential Realizations:

This process of reflection and information gathering will likely lead you to one of three conclusions:

1. You love your current role: Fantastic! You're already on the right path.

2. You need to tweak your current role: Perhaps a new project, a different team, or a shift in responsibilities is all it takes.

3. You need a fundamental change: This is also great news! Recognizing the need for something different is the first step towards finding a more fulfilling career path.

Remember: Change isn't just a risk; it's an opportunity for growth, a chance to discover a path that truly aligns with your passions and goals.

Part 2 - De-Risking Your Career Pivot: Preparation is Paramount

So, you've done the soul-searching and you're seriously considering a career change. Now it's time to strategize. Approaching this transition thoughtfully will minimize risk and maximize your chances of success.

1. Understand the Terrain:

• Explore Potential Paths: Research roles that align with your interests and skills. Analyze the pros and cons of each.

• Analyze the Market: What's the current job market like for your target roles? What are the projected trends for the next 6-12 months? Are there specific industries or companies that look particularly promising?

• Assess Your Skills: Honestly identify the skills required for your desired role. Which do you already possess? What are the gaps you need to bridge?

2. Bridge the Gap:

• Targeted Learning: Focus on acquiring the necessary skills. This could involve online courses, workshops, certifications, or even self-directed learning through books and online resources.

• Expand Your Network: Connect with people already working in your desired field, attend industry events and meetups, and join relevant online communities.

• Gain Practical Experience: Seek opportunities to gain hands-on experience in your target role. Could you shadow someone at your current company? Could you take on a relevant side project or volunteer your time with a non-profit to apply your new skills?

Pro Tip: Explore whether your current company offers personal development programs. These programs can often provide valuable resources and support for your learning journey.

Part 3 - Two Journeys: Engineering to Product & Sales

Let's get into the specifics. We'll share our own experiences transitioning from engineering, highlighting the challenges, surprises, and key learnings.

Mahdi's Journey: From Data Engineering to Product Management

My transition to product management felt like a natural progression. I've always been passionate about problem-solving and understanding user needs. As a data engineer, I found myself increasingly drawn to the "why" behind the features we were building rather than just the "how."

The Itch: It began as a subtle feeling. While I enjoyed coding, I felt a stronger pull towards product strategy and the bigger picture. I was that engineer constantly asking, "Why are we building this?" and "Who is this for?"

Finding My Niche: I realized my technical background could be a significant asset in a product role. I focused on products in the data space, where my data engineering experience gave me a unique edge.

The Mindset Shift: Moving from engineering to product requires a significant shift in perspective:

• Trusting Your Team: As an engineer, I was used to having complete control over technical implementation. As a PM, I had to learn to trust my engineering team's expertise while still providing valuable input and guidance.

• Becoming a Translator: A big part of the PM role is bridging the gap between technical and non-technical stakeholders. I had to become fluent in both "languages."

• Embracing Ambiguity: Product management often involves navigating uncertainty. I had to get comfortable making decisions with incomplete information and iterating based on feedback.

The Essentials: I quickly realized I needed to master these core PM skills:

• Prioritization: Learning to prioritize features and make tough calls about what to build (and what not to build) is crucial.

• Metrics & Measurement: Understanding how to define and track key metrics is essential for measuring product success.

• Collaboration: Working effectively with designers, marketers, and other stakeholders is a daily necessity.

Anis's Journey: From Engineering to Sales

My transition into sales was intentional, yet it still came with its share of uncertainties. While I enjoyed my engineering work, I found myself more excited about explaining our projects than actually building them. I got a real kick out of transforming complex technical ideas into relatable stories. I craved more of that—more conversations, human connection, and opportunities to help people solve problems.

This inclination started during university. I loved presenting our projects—not just explaining them, but truly selling them. I found ways to make them relevant, demonstrating their value beyond the classroom. That passion for communication and connection led me to a part-time business development role at a startup. It was my first real taste of sales. Some days were exhilarating; others were tough. I learned firsthand about rejection, how to handle it, and how to move forward. I discovered that I thrived on the ups and downs, the competition, and the challenge of working with people—not machines.

After graduation, I knew I didn't want to abandon my engineering background entirely, but I also didn't want to be confined to purely technical roles. I yearned for something different. That's when I pursued a Master's in International Business Development. It wasn't just about the degree; it was a strategic step to facilitate my transition. I learned about negotiation, networking, and operating in the business world. But the most valuable lessons came from the alumni network—hearing their stories and seeing how they navigated similar career changes. Their experiences gave me the confidence to believe I could do the same.

Key Differences:

• Human vs. Machine: Sales is fundamentally about people. It's about understanding their needs, building trust, and earning respect. My engineering background helped me connect with technical customers, but true success in sales came from listening, asking insightful questions, and demonstrating a genuine desire to help.

• Shared Frameworks: Some engineering principles translated surprisingly well. KISS—Keep It Simple and Stupid—became a guiding principle. The simpler I made things for my customers, the better the results. Sales isn't about overwhelming people with complexity; it's about making their lives easier.

• Networking and Branding: In sales, networking isn't just a skill—it's a lifeline. But it's not solely about revenue. It's about cultivating relationships, building a personal brand, and opening doors for the future. Over time, I've seen these connections blossom into friendships, mentorships, and unexpected opportunities.

The Learning Curve:

The transition wasn't a walk in the park. There were stumbles, and I had to learn quickly.

• Patience: Building relationships and closing deals takes time. I learned that the hard way. Success comes from consistently showing up and proving your trustworthiness.

• Adaptability: People are unpredictable. Unlike machines, there's no easy debugging. Every conversation is unique, and I had to learn to read situations, adapt, and respond in real-time.

• Embrace Mistakes: Early on, I mistakenly thought sales was about working harder. It's not. It's about working smarter. I learned to listen more, talk less, and focus on the customer's goals—not my own.

Sales has taught me that it's not about having the perfect pitch. It's about showing up, being present, and helping others achieve their goals. My engineering background remains a valuable asset—it helps me understand my customers' technical challenges and offer solutions that resonate. But now I also have the tools to connect, to inspire, and to lead conversations.

Pro Tip: If you're contemplating a similar change, my advice is this: don't overthink it. Take the leap. Your skills are more transferable than you might realize, and the journey will teach you things you can't yet imagine. You might surprise yourself. I know I did.

Part 4 - Navigating Your New Role: Mastering the First Few Months

You've made the leap! Now comes the critical phase of settling into your new role. These first few months are crucial for setting yourself up for long-term success.

1. Embrace the Learning Curve:

• Be a Sponge: Absorb as much information as possible. Ask questions, seek out resources, and learn from your colleagues. It's important to filter this information, though. Prioritize ruthlessly, focusing on knowledge directly relevant to your immediate tasks and goals.

• Seek Feedback: Don't hesitate to ask for feedback, both positive and constructive. It's the fastest way to identify areas for improvement.

• Set Realistic Expectations: You won't become an expert overnight. Give yourself time to adjust, learn the ropes, and define both realistic and ambitious goals to track your progress.

2. Build Relationships:

• Connect with Your Team: Get to know your colleagues on both a professional and personal level. These relationships will be vital to your success.

• Find a Mentor: Seek out an experienced person in your new role who can offer guidance, support, and valuable insights.

• Network Internally: Build relationships with people in other departments. This will broaden your understanding of the company and foster more effective collaboration.

3. Reflect and Adjust:

• Regular Check-ins: Schedule time to reflect on your progress. What's going well? What needs improvement?

• Course Correction: Don't be afraid to make adjustments along the way. If something isn't working, try a different approach. Be agile and adapt to new information.

Part 5 - When Things Don't Go as Planned: Strategies for Recalibration

Despite your best efforts and intentions, there's always a chance that a new role might not be the perfect fit. And that's perfectly okay! It's not a failure; it's a valuable learning opportunity. Here's how to navigate this situation:

1. Open Communication is Key:

• Talk to Your Manager: Be honest about your challenges and concerns. They might be able to offer support, suggest solutions, or help you adjust your role.

• Seek Feedback from Colleagues: Get their perspectives. They may offer insights into team dynamics or company culture that you haven't fully grasped.

2. Analyze and Diagnose:

• Identify the Root Cause: Try to pinpoint the underlying reasons why things aren't working out. Is it the role itself, the team dynamics, the company culture, or something else entirely?

• Reflect on Your Decision: Revisit the factors that led you to make the jump. What did you overlook or underestimate? This reflection will be invaluable for future career decisions.

3. Explore Your Options:

• Consider an Internal Transfer: If you like the company but not the specific role, explore an internal transfer. Perhaps you could return to a previous role or move to a different department.

• Reconnect with Your Previous Employer: If you maintained a good relationship with your previous company, reaching out to them might be an option.

• Explore External Opportunities: If internal options aren't feasible, start looking for new opportunities elsewhere.

4. Give it Time, But Set Limits:

• Allow for Adjustment: Give yourself and the role/team sufficient time before making any drastic decisions. It takes time to adapt, and initial challenges might resolve themselves.

• Define a Timeframe: If you decide to try and make it work, set a clear timeframe. This prevents you from staying in an unfulfilling situation indefinitely.

Remember: Careers are rarely linear. Be open to pivoting or re-pivoting as needed. Each experience, even the challenging ones, provides valuable insights that inform your future choices and bring you closer to your ideal career path.

Part 6 - Key Takeaways, Common Fears, and Resources

Career transitions can be intimidating, but they can also be incredibly rewarding. Here are some of the most important lessons we've learned along the way:

Key Takeaways:

• Embrace Change as an Opportunity: View change as a chance for growth, self-discovery, and finding a path that truly aligns with your goals.

• Preparation is Paramount: Thorough research and preparation significantly increase your chances of a successful transition.

• Lifelong Learning is Essential: The tech landscape is constantly evolving. Never stop learning, and continuously adapt to stay relevant.

• Pivoting is a Strength: Don't be afraid to change course if things aren't working out. It's a sign of strength and adaptability, not failure.

• Networking is Invaluable: Build and nurture your professional network. It can open doors to new opportunities and provide crucial support.

Common Fears & Misconceptions:

• "I'm not qualified": Imposter syndrome is common, but don't let it hold you back. Focus on your transferable skills, your willingness to learn, and your past successes.

• "It's too late to change": It's never too late to pursue a career path that aligns with your passions and goals.

• "I'll lose my seniority": While you might experience a temporary adjustment in title or compensation, your experience and skills remain valuable assets.

Resources That Helped Us:

• Sales & Negotiation (Anis):

  • "Never Split the Difference" by Chris Voss

  • "SPIN Selling" by Neil Rackham

  • “Cold Calling Sucks (And That's Why It Works)” by Armand Farrokh and Nick Cegelski

  • “Selling with” by Nate Nasralla

• Product Management (Mahdi):

  • "Inspired: How to Create Tech Products Customers Love” by Marty Cagan

  • "Monetizing Innovation: How Smart Companies Design the Product Around the Price" by Madhavan Ramanujam, Georg Tacke

  • Lenny’s Podcast and Lenny’s Newsletter by Lenny Rachitsky

• Mindset & Personal Development:

  • (Anis) "The Mamba Mentality" by Kobe Bryant

  • (Anis) "Next Play" by Coach K

  • (Mahdi) "Principles: Life and Work" by Ray Dalio

  • (Mahdi) "Range: Why Generalists Triumph in a Specialized World" by David Epstein

Conclusion:

Ultimately, your career journey is unique. Don't be afraid to forge your own path, embrace change, and seek out opportunities that excite and challenge you. We hope that sharing our experiences and the resources that helped us along the way will empower you to navigate your own career transition with confidence and achieve your professional goals. Remember that career paths are dynamic, not linear.

Best of luck,

Anis & Mahdi

We'd love to hear from you! Share your own experiences with career transitions in the comments below. What challenges did you face? What advice would you give to others?

Grab your espresso (or cappuccino if you’re feeling adventurous) and let’s dive into this month’s brew! We’re tackling the barriers to standardization in the data space, some personal wisdom I’ve picked up in my 20s (it’s been a ride!), and the 100-page gems of The Do Book Co. So, without further ado, let’s talk data engineering (and personal updates) while the espresso is still hot.

Barriers to standardization in the data space

Maxime Beauchemin, one of the pioneers (THE pioneer?) of the data engineering field, recently wrote about why data teams keep reinventing the wheel by building their transformation layer from scratch. In his article (which I highly recommend), he offers a proposal relying on “Parametric Pipelines” and “Unified Models” to build reusable and generic data components/assets that are also flexible enough to accommodate the inevitable specificities of every business.

Maxime’s article navigates the nuances of one key part of the data stack (transformation) and why it lacks standards, but I think he touched on a topic that applies to the stack as a whole: We're great at standardizing the tools (thanks dbt & Airflow), but everything else feels like the Wild West - which is a shame since we universally agree that all data teams do slightly different versions of the same job.

I think this lack of standardization outside of the tools themselves is mainly due to two barriers we still need to overcome:

• Every tool generates its own metadata (and doesn’t share it): Probably the thing that hurt data teams the most in the past five years is that every tool within the Modern Data Stack wanted to store its own metadata and build features on top of it. This meant that every tool knew a bit about the data and the pipelines, but no tool had the full story (not even the data catalog) - making it very difficult to centralize all of the stack’s metadata in one place and build insights on top of it (or even define standards for managing it). The ideal scenario in this area was to have a standard representation of metadata that tools within the stack can implement and then leverage to exchange metadata in an automated manner, allowing for the rise of standardized ways to manage the data itself. I still believe we’ll reach that state eventually, but the road ahead is, unfortunately, long.

• We don’t write enough YAML (seriously - hear me out): Although the “data engineers are in fact YAML engineers” reflection started out of (valid) frustration with YAML, I think we’re still paying a high price for not doing (most) things in code. Even though the Modern Data Stack (RIP) brought with it a lot of advancements, the focus remained on the UI, and many technical workflows (essential for automating and standardizing things) were simply ignored. By moving more things (dashboards, tooling configuration, metrics, data observability, etc.) to code/YAML, we dramatically shorten the path toward standardization and open new industry-wide collaboration doors. The “Post-Modern” Data Stack (PMDS?) is fortunately taking things in the right direction, and I believe we’ll overcome this barrier in the next few years.

Out of the comfort zone: 5 learnings from my twenties

This September I turned 30 and decided that it was a good moment to reflect on the biggest learnings of my twenties (which were, in a nutshell, a rollercoaster). The end result felt like something worth sharing (more on that below), but these are the five learnings I’m extremely grateful for:

• Figure out how you’re wired: What makes you you?

• It’s all about resilience (Luctor et Emergo): You’ll inevitably face an insurmountable mountain, and you need to be ready for it.

• Curiosity and open-mindedness are the path forward: The combination of these two characteristics, whether in a personal or professional context, ensures that you’re always on the right track toward becoming a better version of yourself.

• Take measured risks: Understand what risk is and figure out how much of it you want in your life.

• Learn from your losses, and celebrate your wins: As you progress in life, you’ll inevitably make many wrong turns and yet more right ones. Acknowledge the outcome of these turns and make them count.

Want to jump into the details? I wrote a full article about it! (Isn’t that what you do when you turn 30?)

Out of office: The 100-page gems of The Do Book Co

On a recent trip to Vienna (highly recommended!) I stumbled upon The Do Book Co’s “pocket guides” at a cool concept store called Calienna. These tiny books (around 100 pages long on average) are all written by subject matter experts and come in an ideal format: long enough to cover a complex topic and deep-dive into it, but not a big time commitment like a 400-page book.

The first two I read are:

• Do Deal by Richard Hoare & Andrew Gummer: Fantastic advice and insights that cover all the stages of negotiations (in any given context).

• Do Start by Dan Kieran: This one was so good that it deserved a Twitter/X post. If you're interested in working at startups or starting your own, this book is a must-read. Dan covers all the key areas of entrepreneurship while providing valuable insights and telling the inspiring story of Unbound.

They have a ton of other guides on all sorts of topics, so check them out!

If you enjoyed this issue of Data Espresso, feel free to recommend the newsletter to people in your entourage.

Your feedback is also very welcome, and I’d be happy to discuss one of this issue’s topics in detail and hear your thoughts on it.

Stay safe and caffeinated ☕

In this edition, we will discuss data modeling techniques that you can leverage in a post-Modern Data Stack world, Notion’s very interesting “reverse-Modern-Data-Stack” journey, and the tricky first steps of building a data team. So, without further ado, let’s talk data engineering while the espresso is still hot.

Data modeling techniques for the post-Modern Data Stack world

Over the past few years, as the Modern Data Stack (MDS) introduced new patterns and standards for moving, transforming, and interacting with data, dimensional data modeling gradually became a relic of the past. In its place, data teams relied on One-Big-Tables (OBT) and stacking layer upon layer of dbt models to tackle new use cases. However, these approaches led to unfortunate situations in which data teams became a cost center with unscalable processes. So, as we enter a “post-modern” data stack era, defined by the pursuit to reduce costs, tidy up data platforms, and limit model sprawl, data modeling is witnessing a resurrection.

This transition puts data teams in front of a dilemma: should we revert back to strict data modeling approaches that were defined decades ago for a completely different data ecosystem, or can we introduce new principles that are defined based on today’s technology and business problems?

In my opinion, the answer is (drumroll) somewhere in the middle. It's important to move away from model sprawl, but without impacting the delivery speed of new data products. How? In my latest article on Towards Data Science, I discuss different techniques to achieve this. (Spoiler: it starts with defining the right standards.)

Fresh off the press: Notion’s reverse-MDS journey

As data professionals, we’re all familiar with the data lake → Modern Data Stack (MDS) path: Data teams realize that they’re spending too much engineering time managing the data platform itself due to its complexity (thanks for the trauma, Hadoop) and decide to migrate to SaaS tools that offer a streamlined experience and minimal maintenance needs.

In a recent blog post, the Notion data team discussed their rather unique journey in the opposite direction. Instead of migrating from a data lake to an MDS platform, they replaced their shiny Snowflake- and Fivetran-based platform with a data lake architecture using Hudi, Spark, and Kafka — with very positive results.

We should note here that their use case is unique to their product since around 90% of their upserts are updates, which Snowflake hates while Hudi excels at. Their journey is, however, a great reminder that the data architecture should be defined first and foremost around your specific business needs, and the tooling that’s “cool” or “modern” is not always the best fit for your particular use cases.

Out of the comfort zone: The data team’s tricky first steps

As I transitioned from engineering to product (a journey that I’ll discuss in an upcoming blog post), I started listening to ’s fantastic podcast on a weekly basis (which I highly recommend even if you don’t work in product). In a recent episode, his guest was Jessica Lachs (VP of Analytics and Data Science at DoorDash). The whole episode was filled with valuable insights, but I personally appreciated the discussion around a very important yet overlooked topic: the data team’s “delicate” first steps.

Data teams have a rather awkward positioning in most companies: They don’t directly generate value, and so it’s easy to head in the wrong direction with just one wrong step. Decisions like building an unnecessarily complex platform early on, focusing on the wrong business goals, or adopting a reactive “data team as a support team” mindset will ultimately make it very difficult for the data team to scale and showcase its value.

Outside of the podcast episode, which covers many other topics like defining effective metrics and the ideal data team structure, Jessica published a blog post that covers this topic in particular. If you’re leading a data team in its early days, this is a must-read.

Out of office

Until we meet again for another espresso, and if you still have room for one addition to your summer reading list, I highly recommend The Art of Rest by Claudia Hammond. If, like myself, you feel “guilty” when resting, then this book will definitely change your perspective.

If you enjoyed this issue of Data Espresso, feel free to recommend the newsletter to people in your entourage.

Your feedback is also very welcome, and I’d be happy to discuss one of this issue’s topics in detail and hear your thoughts on it.

Stay safe and caffeinated ☕

Navigating your data platform’s growing pains: a path from data mess to data mesh

When working on software components, developers can leverage a wide range of frameworks, design patterns, and principles to scale their products and seamlessly adjust their architecture to support new use cases and handle increasing usage and complexity. This allows software engineering teams to ensure optimized performance and reliability as their platform (and its value) grows in scale.

Data teams, however, are not so fortunate. While the initial months of a data platform’s lifecycle are often marked by the excitement of tackling complex technical challenges and the joy of delivering a first wave of data products, what often follows is a daunting spiral of mounting complexity, rising costs, and diminishing returns.

Unlike other problems that we need to navigate as data teams, our scalability struggles are inherently different from the ones faced by software teams. In the data world, these struggles come in the form of unavoidable technical complexity (like mixing a multitude of patterns to move and transform data across an ever-growing list of systems) and the data platform’s unique positioning within the company (since eventually every business unit gets connected to it either directly or indirectly).

So, in this post-MDS world, where data teams are thoroughly scrutinized over their spending and continuously asked to showcase their value, it is more important than ever to define standards and tenets for successfully scaling a data platform.

In my latest article, I offer five principles (complete with actionable strategies) to navigate the tricky state between being a new data team and building a large-scale data platform that generates substantial business value.

Fresh off the press: Spotify’s data platform

Among the big tech behemoths, some companies stand out from the rest when it comes to data - think Airbnb, Netflix, Uber, and Spotify. These companies play a key role in driving the data field forward by constantly rethinking their approaches to data and, more importantly, by being open about their data work (whether by open-sourcing the tools they build or sharing the learnings of their journey).

I’m personally a big fan of how the Spotify data teams approach their data projects, whether it’s to build the largest Dataflow job ever or to scale and democratize both analytics engineering and data visualization. Last week, Spotify published the first article in a new series that will discuss the different aspects of its data platform and data endeavors. I recommend following along the journey since it would definitely present many valuable learnings that can be applied when working on data projects.

Out of the comfort zone: The post-Modern-Data-Stack world

In the past few months, one of the hottest topics in the data world has been the death of the modern data stack. I believe that enough ink has been spilled on this topic, but I also want to reflect on the Modern Data Stack era and how it transformed the data field.

When I started my career in early 2017, the Hadoop ecosystem was all the rage. Companies were spending enormous resources on building data lakes that had a terrible return on investment and were very taxing to maintain. The problem wasn’t the data itself but the systems, which were simply too complex and, in most cases, barely working.

Fast forward to 2024. Last week, while reading dbt Labs’ annual state of analytics engineering report, I was pleasantly surprised by the following chart:

The takeaway is that we reached a state where the biggest two challenges that data teams face (poor data quality and ambiguous ownership) are related to people and processes and not to technical complexity or the data platform itself. This is, in a way, a direct acknowledgment of the modern data stack’s role in solving one of the Hadoop era’s biggest problems: today’s data platform just works. Gone are the days of messy configuration, navigating chaotic error logs across different systems, and spending endless engineering time on building and maintaining the platform.

The Modern Data Stack, for all its flaws, solved the data platform’s technical hurdles. Now it’s time to solve the business ones.

If you enjoyed this issue of Data Espresso, feel free to recommend the newsletter to people in your entourage.

Your feedback is also very welcome, and I’d be happy to discuss one of this issue’s topics in detail and hear your thoughts on it.

Stay safe and caffeinated ☕

Writing design docs for data pipelines

Over the past few years, adopting software engineering best practices has become a common theme within the data engineering space. From dbt’s software-engineering-inspired capabilities to the rise of data observability, data engineers are getting increasingly accustomed to the software engineer’s toolset and principles.

This shift had a major impact on how we design and build data pipelines. It made data pipelines more robust (since we moved away from hard-coded business logic and complex SQL queries to modular dbt models and macros) and drastically lowered the number of “Hey can you check this table?” Slack messages (via automated data quality monitoring and alerting).

These changes helped us move the industry in the right direction — but there are still areas in which we still have much to learn from our software engineering counterparts.

Based on recent data released by dbt Labs, around 20% of dbt projects have more than 1,000 models (and 5% have more than 5,000). These numbers highlight one fundamental problem in our data pipelines: we’re not intentional enough with how we design them. We stack layers upon layers of ad-hoc models and use-case-specific transformations and end up with ten models that represent the same logical entity “with subtle differences”.

In my latest article, I discuss one artifact that can help us design (and build) robust foundations for our data platforms: design docs.

Fresh off the press: our dbt journey at Zendesk

At Zendesk, we started our dbt journey more than a year ago (and have been running it in production for over six months) - and today, more than half of our data pipelines are dbt jobs.

The most important pillar to which we owe the success of our approach is setting the right foundations from day one and defining standards that make sense within the context of our use cases and architecture.

Last month, I published an article in the Zendesk engineering blog that discusses the core foundations of our dbt setup (and more) - it’s intended to be the first article of a three-part series, so stay tuned for more technical articles on our dbt implementation.

Out of the comfort zone: What got you here won’t get you there.

Last year I participated in the fantastic Ignite tech lead mentoring program (which I highly recommend), and one of the key learnings that stuck with me is “What got you here won’t get you there.” This was a very elegant way to phrase a principle that I’ve been applying throughout my career, and I think it’s important to keep it in mind as you progress throughout yours.

Every vertical (or horizontal) transition in your career introduces you to a new job requiring a specific skill set that differs from your previous role. Senior, Staff, Tech Lead, or Engineering Manager are all very different hats that require some preparation before wearing them. Some companies merely rely on the “years of experience” metric to determine your readiness to change hats, but what really matters is how comfortable you are with the skill set of the new job.

As you think about your career goals, consider the role that you currently aim to achieve and the skills that it requires, then use this information as a map to define the areas on which you want to focus in your development and the experiences you want to gain.

A sound to code to

Ever since the Succession series finale, I’ve been mostly relying on Nicholas Britell’s beautiful, grandiose, and melancholic season 4 score as my coding playlist - with a focus on Andante Risoluto (unsurprisingly).

If you enjoyed this issue of Data Espresso, feel free to recommend the newsletter to people in your entourage.

Your feedback is also very welcome, and I’d be happy to discuss one of this issue’s topics in detail and hear your thoughts on it.

Stay safe and caffeinated ☕

Airflow vs. Dagster: it’s all about philosophy

Ever since it was open-sourced by Airbnb back in 2015, Apache Airflow established itself as the de-facto standard for orchestration within the data space. Thanks to its feature-rich User Interface (UI), its ability to manage a wide range of operations, and particularly its no-nonsense and intuitive approach to organizing workflows via DAGs and tasks, it quickly eclipsed existing orchestrators like Spotify’s Luigi (which was open-sourced in 2012) and the Hadoop ecosystem’s Oozie.

Airflow is used today by data engineering teams around the world for an ever-expanding list of use cases, supported by custom operators, in-house abstractions, and a myriad of hacks to leverage some of its aging features. On the other hand, the way we interact with data today is very different from how things were seven years ago:

• We no longer just want to run Spark jobs. Instead, we think about the quality, state, and lineage of our heterogeneous data assets.

• We can no longer tolerate weeks-long development cycles to generate new data assets. Instead, data pipelines should be written, tested, and deployed as efficiently and as fast as possible.

• We can no longer rely on a small centralized data engineering team that builds and maintains all the DAGs. Instead, we aim for self-service capabilities and automation that would allow a larger set of contributors to build data assets and push them to production.

• Finally, the data stack we built for the above is moving fast and changing old patterns. And so we no longer think about tasks, but we think about dbt models, Airbyte connectors, metrics, and a whole ecosystem of capabilities that are the modern incarnation of 2015’s Airflow operators we once had to write from scratch.

With the above in mind, it’s definitely time to ask the question: Is Airflow still the undisputed go-to orchestrator for data pipelines, or is Dagster, the new orchestrator that’s built with all the previous points in mind, the better option?

In my latest article, published on Restack’s blog, I go into the details of how Dagster differs from Airflow and whether data teams that have already invested a lot in Airflow should make the jump. But most importantly, I explain why “Airflow vs. Dagster” is not a technical question at all - it's just a matter of philosophy.

From binge-reading to binge-watching

I personally believe that the year’s most important Modern Data Stack conference isn’t Databricks’ Data + AI summit or Snowflake’s summit - instead, it’s dbt Labs’ Coalesce. This is not only because dbt is the tool that opened the door to the third wave of data technologies, but also because at Coalesce everyone within the data community belongs.

If you never attended Coalesce before, I totally recommend doing so this year. You’ll learn quite a lot about the Modern Data Stack and how fellow data practitioners are doing more with third-wave data technologies. You’ll see how fun and engaging the dbt Slack is. You’ll experience how welcoming and diverse the data community is. And most importantly, you’ll feel that you belong - because you do.

Throughout the next couple of weeks, let’s binge-watch Coalesce talks (there are way too many interesting ones) instead of binge-reading data content. And hey - it’s our yearly opportunity to tackle the endless debate: is Kimball's dimensional data modeling approach still relevant? (even though we all know that the only right answer is “mostly yes, but it depends”).

A sound to code to

After watching Cyberpunk Edgerunners, I’ve been listening to most of its soundtrack on repeat for the past couple of weeks (it’s really that good) - and I can confirm that it offers a productivity boost similar to Night City’s finest cyberware upgrades. My personal favorite is - unsurprisingly - “I Really Want to Stay At Your House”.

If you enjoyed this issue of Data Espresso, feel free to recommend the newsletter to people in your entourage.

Your feedback is also very welcome, and I’d be happy to discuss one of this issue’s topics in detail and hear your thoughts on it.

Stay safe and caffeinated ☕

Having trust issues with your data?

A statement that I keep running into variations of online is how “you can’t create value with ML if you don’t have good data to begin with”. (I.e., garbage in, garbage out).

I personally believe that neglecting data quality and building data products without first ensuring robust data validation is one of the key issues that eventually lead to failed data initiatives. But why did data quality only gain attention rather recently?

For many years (think 2010 to 2016) data engineers were building pipelines without software engineering best practices - the goal was to deliver as much data as possible, as fast as possible. This didn’t cause any immediate issues at the time, because “big data” was still a secondary factor for decision-making at most companies. But now that data is a first-class citizen everywhere and metrics are consumed by a wide range of users/teams, we frequently find ourselves trying to understand why two dashboards present different values for the same metric, or how a failure in one pipeline would impact our end-users. Now, we’re paying our overdue debts for building data pipelines without having data quality in mind. So how should you tackle data trust issues?

Within the Modern Data Stack, you’ll find dedicated companies that concentrate on solving this particular issue. While this may be the way to go if you don’t actually have any data engineers within your company (you should though!), I find that it adds unnecessary complexity and costs to solve a rather simple problem - for most cases.

Data quality, at its core, can be achieved by being able to answer three main questions:

• What types of checks do you want to implement for your pipelines? (schema checks, data checks, etc.)

• Which operations should you implement to ensure that your data meets the standards defined by answering the first question? (row count, handling nulls, deduplicating the data, recasting fields, ensuring default values, etc.)

• What action should happen when one of your checks fails? (raising a warning or an error based on the type of the check and the severity of the issue, sending alerts, etc.)

For example, if you have an architecture that relies on Spark-based pipelines, you can implement the checks and actions related to them as Spark-based nodes within your orchestrated pipelines, with the aim of running these checks as close to the source as possible (to minimize the impact on downstream processes). Or better yet, you can leverage a tool like Soda SQL or Great Expectations, which are open-source projects that will simplify your data quality checks (you’ll have plenty of pre-defined checks out of the box) without adding much complexity to your stack.

Data quality is a key pillar of having a successful data-driven strategy, and yet as a problem, it’s actually not that complex. Adding yet-one-more-vendor to your stack solely for data quality is unnecessary in most cases because there’s no hidden complexity behind the core problem.

Fresh off the press

A couple of weeks ago, the unbundle vs. rebundle debate took Data Twitter by storm.

First, Gorkem Yurtseven published The Unbundling of Airflow on Features & Labels’ blog, arguing that Airflow (and workflow engines in general) is being unbundled into separate tools each focusing on one part of the data stack, which would eventually make the orchestration engine redundant.

Then, Nick Schrock published Rebundling the Data Platform on Dagster’s blog, countering with the point that the “next thing” wouldn’t be giving up on workflow engines, but instead evolving them so that they can orchestrate software-defined assets - which Dagster now supports.

With the modern data stack being extremely fragmented in its current state, the debate about how to connect all these tools is just getting started.

Out of the comfort zone: navigating the endless stream of data content

A few weeks ago I came across a tweet about how it’s extremely hard to keep up with all the topics/discussions in the data community - and I immediately related to it:

This is an extremely beneficial and healthy environment for the field itself since it’s boosting innovation and opening new possibilities every single day - but it can also get very overwhelming for us humans of data.

Personally, I try to stick to two rules when it comes to navigating data content:

1. Streamline your sources: Multiple people in the data space try to post content on a daily basis - while this is done with good intentions, I genuinely believe that as humans we simply can’t deliver thoughtful insights every single day. If you’re following many data “influencers” on LinkedIn for example, you’ll find yourself frequently consuming short pieces of content throughout the day that are written mostly because the writer wants to share content, not because there’s a genuine thought or reflection. Consuming these “bites” of content drains your precious energy and points you in dozens of directions in one single scrolling session. Instead of merely following people who post “daily”, I recommend seeing what the community is discussing on Twitter once or twice per day (personally I usually try to avoid threads), where the 280 character limit enforces everyone to go straight to the point - and then you can make your own deep dives into topics that interest you.

2. Streamline your thoughts: This second rule is a direct result of the first one and the fact that data engineering is a vast and ever-evolving field. By limiting your sources and consuming long-form content only when the topic genuinely interests you, you’ll be able to build knowledge in areas that matter to you. Instead of knowing a bit of everything, the aim should be to know a bit of everything and a lot about a few things. This would allow you to formulate your own thoughts and opinions instead of only consuming content, and will help you get a better understanding of the bigger picture. If you’re interested in the concept of a “metrics layer” for example, you can spend time going through articles and white papers by major tech companies who built their own internal metrics platforms, to learn the “why”s, “how”s, and the lessons they learned along the way.

My point is that trying to stay up to date with everything happening within data engineering / the data stack is a futile effort that won’t give you long-term knowledge, whereas focusing on a few areas that interest you and building meaningful and deep knowledge in them is a path towards widening your expertise and potential.

But again, this is the approach that works for me - and it won’t necessarily work for everyone.

A sound to code to

During the last few weeks I’ve been the most productive when playing Polo & Pan’s latest album, Cyclorama. With Attrape-rêve as a definite standout.

If you enjoyed this issue of Data Espresso, feel free to recommend the newsletter to people in your entourage.

Your feedback is also very welcome, and I’d be happy to discuss one of this issue’s topics in detail and hear your thoughts on it.

Stay safe and caffeinated ☕

If this is your first time reading Data Espresso, I recommend going through the first two posts (post #0 and post #1) to get familiar with the newsletter’s concept and the motivation behind it.

As mentioned in the previous edition, the newsletter is still not in its finalized format and so certain sections can be changed in the future - but enough with the introductions, let’s talk data engineering while the espresso is still hot.

Why open table formats are a game-changer

One word that characterized the second wave of data systems (the Hadoop ecosystem and co.) is compromise - to have horizontal scalability in storage and compute we had to let go of certain features that were a given in existing systems, like ACID transactions and schema enforcement. And although most of the compromises were necessary (as explained by the CAP theorem for example), some of them weren’t.

The de-facto metadata store of the Hadoop ecosystem, the Hive metastore, is where most of the compromises happened. To manage distributed and partitioned file-based tables, the Hive metastore was designed in a way completely disconnected from the data itself - it tracks directories (for both tables and partitions) instead of data files, and so most of the features that are offered by a typical RDBMS are effectively unattainable since the metastore doesn’t directly manage the data files.

This design choice in the Hive table format meant that Hive tables were problematic to manage (since the schema isn’t verified for every data file), hard to trust (since there’s no table transaction log), and also inefficient when they reach a certain scale (since we can’t optimize the queries below the partition level and each query necessitates listing the data files).

Due to such issues, many companies struggled with file-based tables. Additionally, advancements in distributed query engines and the metadata management space were slowed down by the inefficiencies of the Hive metastore - pushing companies to opt for a scalable data warehouse for their analytics architecture instead of relying on a file-based system.

But that all changed in 2019. In one year, Databricks open-sourced its Delta project under the name Delta Lake, and both Uber and Netflix submitted their own table formats, Hudi and Iceberg respectively, to the Apache Software Foundation.

Each one of these new table formats, in its own way, solved most of the issues related to the Hive metastore with a set of features and key strengths that differentiate it from the other formats - but most importantly, their introduction meant that data lakes are no longer inferior to data warehouses when it comes to table management and query optimization. The data lake is dead, long live the lakehouse!

To familiarize yourself with open table formats, and determine which one suits your needs the most, the following blogs would be a great place to start:

• Iceberg:

  • Metadata Indexing in Iceberg by Ryan Blue (Iceberg creator)

  • A Short Introduction to Apache Iceberg by Christine Mathiesen (Expedia Group)

  • Apache Iceberg – An Architectural Look Under the Covers by Jason Hughes

• Delta Lake:

  • Massive Data Processing in Adobe Experience Platform Using DeltaLake by the Adobe Experience Platform team

  • Engagement Activity Delta Lake by the Salesforce Engineering team

  • Diving Into Delta Lake: Unpacking The Transaction Log on the Databricks blog

• Hudi:

  • Apache Hudi - The Data Lake Platform on the Apache Hudi blog

  • Data Lakehouse: Building the Next Generation of Data Lakes using Apache Hudi by Ryan D'Souza & Brandon Stanley

Fresh off the press

The Four Innovation Phases of Netflix’s Trillions Scale Real-time Data Infrastructure

Zhenzhong Xu, who led the Stream Processing Platform team at Netflix, dives deep into the different phases that the Netflix real-time data infrastructure went through, with their respective challenges and learnings.

Data to engineers ratio: A deep dive into 50 top European tech companies

A very interesting analysis by Mikkel Dengsøe, the head of data science at Monzo, in which he compares the data engineers ratio at 50 tech companies from different sectors.

Out of the comfort zone

From the outside, it’s easy to romanticize Silicon Valley and the idea of launching startups that turn into tech giants making the world a better place - but we all know that things don’t happen that way.

If you watched The Social Network then you probably already know that “you don't get to 500 million friends without making a few enemies”, and the power struggle that we witness in the movie isn’t an exception or something specific to Facebook (um, Meta), but it’s a story that’s rather familiar in Silicon Valley. Some of these stories that happen behind closed doors were documented by award-winning journalists via page-turner books. Below are my recommendations:

• Super Pumped: The Battle for Uber by Mike Isaac: this one is my personal favorite because it not only compellingly tells the story behind Uber, but also delves into the dark side of Silicon Valley unicorns and the “hustlin’” that happens behind the scenes.

• No Filter: The Inside Story of Instagram by Sarah Frier: even though Kevin Systrom and Mike Krieger (Instagram’s founders) agreed to sell the company to Facebook early on, life didn’t get any easier for them and their small team. Sarah Frier masterfully tells the story of a company within a company, and how the decisions of a small group of engineers can change the world we live in.

• An Ugly Truth: Inside Facebook's Battle for Domination by Sheera Frenkel and Cecilia Kang: Yes, it’s Facebook/Meta again, and this time you get to discover the dynamics between Mark Zuckerberg and Sheryl Sandberg, how the Cambridge Analytica scandal unfolded, and what led to the company’s fall from grace.

• Bad Blood: Secrets and Lies in a Silicon Valley Startup by John Carreyrou: Last but not least is the story of the unicorn that never was. Theranos was probably one of the biggest scams of the 21st century, and the story of Elizabeth Holmes’ startup is filled with lessons about what can go wrong when aiming to change the world - but most importantly, it shows that most people aren’t as smart as they think they are.

A sound to code to

Bonobo’s new album, Fragments, is all you need for a productive coding session.

If you enjoyed this issue of Data Espresso, feel free to recommend the newsletter to people in your entourage.

Your feedback is also very welcome, and I’d be happy to discuss one of this issue’s topics in detail and hear your thoughts on it.

Stay safe and caffeinated ☕

First of all, I’m glad that you’re joining me on this ride - hopefully this will be a mutual learning experience. The newsletter is still going through the first cycles of its life, so probably this is not the finalized format and certain sections can be changed. Throughout the first few issues, you’ll basically see the newsletter go through adolescence.

As stated in my initial post, my aim for this newsletter is that it will be more of an exchange of ideas, so you’re definitely encouraged to reply to the emails with your thoughts and feedback.

Let’s talk data engineering

Can open-source survive in the modern data stack?

If we look back to the previous wave of data technologies, the wave of the Hadoop ecosystem, NoSQL databases, and the term “Big Data”, we’ll notice that open-source was the standard. The data technologies realm was one of the first battles that open-source won, and this in turn was one of the most important factors that led to the rapid pace of innovation during the Hadoop era. The whole ecosystem thrived on the open-source model, with multiple teams of engineers from the biggest tech giants contributing to the same projects under the umbrella of open-source foundations.

In contrast, if we look at the ecosystem of the modern data stack, we’ll notice that things are not so similar. We’ve somehow taken a step back in a way and decided to rethink open-source-based business models. This is the result of multiple factors, one of which is cloud providers offering managed versions of open-source projects that effectively render the companies built on top of these projects redundant.

The consequence of this is either closed-source business models (which are back in style) or stricter licenses similar to what Elastic did in 2021 - which would offer open-source-based companies a fighting chance against cloud providers.

With the tweaked licensing and the continued support of the tech communities, open-source is still a viable option for data companies. The best proof of that is Airbyte, which disrupted the data integration space with its open-source approach and just recently announced a $150 Million series B funding round, at a $1.5 billion valuation. (after also opting for a stricter license)

So, yes, open-source will survive (and thrive) in this era, but we’ll potentially witness a slower rate of innovation due to the fragmented ecosystem and the closed-source model that multiple modern data stack companies opted for.

Fresh off the press

Life with dbt

When considering adding a new tool to your stack, nothing is more helpful than the feedback of other engineering teams that are already using it. Well, if you’re considering using dbt (and you should be), the data engineering team at Devoted published a very insightful piece on their first year using it and how powerful it can be when integrated with the different components of your stack. (the post is written by Adam Boscarino and Jason Brownstein.)

A look back at 2021

2021 was an eventful year when it comes to the Modern Data Stack, and in an excellent article published on Towards Data Science, Salma Bakouk (co-founder & CEO of Sifflet) goes through the trends that shaped it.

… And a look ahead to 2022

In an article also on Towards Data Science, Prukalpa Sankar (co-founder of Atlan) discusses six ideas that will continue to shape the Modern Data Stack in 2022.

Out of the comfort zone

One of the best books I’ve ever read is Range: Why Generalists Triumph in a Specialized World by David Epstein, for a very simple reason - it proved something that I always believed in: you should always start gathering knowledge, expertise, and skills horizontally before specializing in a particular subfield.

If you just graduated from university and you want to start a career in a technology field (not necessarily data engineering), it might be tempting to jump headfirst into learning the most in-demand technology/tool of that field. While technically that might indeed help you land your first job, your long-term strategy should be built around widening your knowledge and expertise horizontally as much as possible, before making vertical deep dives into a specific technology/trend.

The reasoning behind this is that by first familiarizing yourself with concepts and paradigms, you’ll be able to:

• Learn and master the technologies themselves faster

• Recognize patterns more easily and know when and how to leverage the right technology for a specific use-case

• Have the flexibility to switch between technologies without a significant learning curve

For example, if you want to get into data engineering, instead of starting with a deep dive into writing Apache Spark jobs, focus on learning concepts like distributed computing, massively parallel processing (MPP), MapReduce, ETL/ELT, and data modeling. This will first help you learn Spark in a shortened amount of time (because you’ll be comfortable with the concepts behind it), better understand how Spark implements specific concepts, and more importantly, recognize when Spark isn’t the right tool for a given problem. In the long term, this will give you one (very important) additional possibility throughout your career:

Yes, you’ll be able to pivot from one technology/field to another one without much trouble - and in a field that’s moving extremely fast like data engineering, this is a very valuable asset.

A great book that can help you understand a wide range of data-related concepts is Designing Data-Intensive Applications by Martin Kleppmann.

A sound to code to

Asking a developer whether they like to listen to music while coding is like asking someone whether they’re a cat person or a dog person: it’s something that feels like a personality tell, but you’re never quite sure what it actually means.

Personally, if I’m not working on a complex topic, I find myself more productive when listening to certain types of music - and the aim of this section is to share with you some of the records that I find the most suitable for coding sessions.

Since this is the first issue of the newsletter, my recommendation is actually the artist I listen to the most when working: Jamie xx (yes, the xx’s discreet DJ). I won’t burden you with unnecessary details - I’d just recommend that you give his 2015 album, In Colour, a listen during your next coding session. It’ll either turn into a dance session or you’ll finish what you’re working on in half the expected time: it’s a win-win.

If you enjoyed this issue of Data Espresso, feel free to recommend the newsletter to people in your entourage.

Your feedback is also very welcome, and I’d be happy to discuss one of this issue’s topics in detail and hear your thoughts on it.

Stay safe and caffeinated ☕

Welcome to Data Espresso, a bi-weekly newsletter in which I’ll discuss various topics related to data engineering and technology in general.

Who am I?

My name is Mahdi Karabiben, and I have been working with data engineering technologies on a daily basis since 2017. My (rather short) career has already allowed me to work with data in very different scenarios: I first started my career at Democracy International, where I worked on visualizing electoral data; after that, I joined the data-marketing firm Numberly to build data pipelines for AdTech use cases; I then transitioned into the financial sector, where I worked on data projects at CA-CIB and FactSet; and now I'm building data products at Zendesk and experimenting with open-source projects in my free time.

I’m someone who has always been very passionate about data, but I also like building human connections. And those are the two pillars of this newsletter.

Why am I starting a newsletter?

Before COVID, I used to take an afternoon coffee break with a colleague of mine during which we discussed topics that range from the latest news, trends, and advancements in the data engineering world to the latest TV show that one of us binge-watched. When I look back at those conversations now, it’s clear to me that those coffee breaks were one of the highlights of my typical workday.

My aim with Data Espresso is to create a similar experience for you as a subscriber. Each issue would consist of the following sections:

• 3 brief observations related to data engineering (trends, commentary on certain technologies, updates, etc.)

• The most interesting article that I came across during the past two weeks.

• A sound to code to: This is mostly the result of my 4 years as a member of the radio club back in university - Every issue will come with a song that I personally enjoy coding to.

• Out of the comfort zone: This is a section that won’t have anything to do with data engineering or technology as a whole.

As you can see, the idea consists of mixing technology with a human/personal aspect - like in an actual discussion between two people who happen to have the same job, while drinking an espresso.

Why subscribe?

Subscribe to get full access to the newsletter and website. Never miss an update.

Stay up-to-date

You won’t have to worry about missing anything. Every new edition of the newsletter goes directly to your inbox.

Join the crew

Be part of a community of people who share your interests.

To find out more about the company that provides the tech for this newsletter, visit Substack.com.

Subscribe now

Subscribe now

A guide to building an automated MLOps pipeline by leveraging the trusted DevOps toolset.

Continue reading on Towards Data Science »

A set of recommendations and starting points to efficiently run Superset at scale

Continue reading on Towards Data Science »