Introduction

It is going to be a rough couple of years for data teams.

I recently spoke to a friend who joined as the first data engineer at a company that raised $500m at a $10bn+ valuation. They were the sole engineer building the data platform. They worked 60+ hours, and always felt underwater. Their requests for an addditional team member were rejected.

AI is changing how Data Teams are built. There is more complexity to work through than ever before, expectations and demand are higher than ever before, but Data Team budgets are not growing in size.

Specifically, many assume tokens are a substitute for manpower, which may be true to a point, but does not scale.

In this article we’ll run through the top things you need to be doing to build and scale a data team to avoid your team burning out.

Why do you need consider burnout when building a data platform? AI.

The Data Team Bowtie

The bowtie above illustrates the extreme pressureon data teams.

On the one hand, companies that are larger have a greater demand for services. This is being exacerbated by AI, with more and more companies choosing to be like Mid-Market or Enteprise companies where everyone wants to do everything with AI.

AI is also increasing the need for a more robust, fully-featured data stack. This leads to an increase in demand while reducing supply, creating massive pressure on the central data team. Concretely, there are a few ways to think about this.

MDS Expectations > budget

Everyone wants context, lineage, observability, data quality, governance and self-service. These features are no longer treated as “nice to have”; they are expected from the data platform.

The budget rarely expands at the same rate. In dbt Labs’ 2025 survey, only 30% of respondents reported a larger data budget and 40% reported increased headcount. In other words, most teams were expected to deliver more without receiving more people or money. Read the report.

The result is predictable: engineers spend their time maintaining tooling, responding to alerts and explaining broken pipelines instead of building. Burnout is therefore not just a people problem. It is an architectural problem.

Answer: you need to pick a single Agentic Control Plane that offers more features and easier “quick wins” without having a completely modular ad absurdum stack. Options like Databricks and Orchestra win here.

Data volumes are growing faster than data teams

The number of people operating the platform might remain roughly constant, but the number of sources, models, pipelines and consumers does not.

IDC’s latest Global DataSphere forecast expects data creation and consumption to continue expanding, driven partly by generative AI, IoT devices and increasingly complex digital activity. View the IDC forecast.

More data means more infrastructure, more dependencies and more ways for something to fail. If operating the platform requires engineers to manually monitor pipelines, investigate every failed task and coordinate every rerun, the workload scales directly with the volume of data. Eventually, the humans become the bottleneck.

A scalable data platform must therefore scale operations as well as compute. Short story is — buying an old-school tool like Y42, Weld or one of those weird shitty data warehouses like Panoply or perhaps even Postgres won’t hold up to the simple volume of data you’ll probably need.

Conclusion: leverage platforms built for scaling Big Data. Data Volumes are increasing. People want power. Give it to them.

Bonus: serverless orchestration. You are going to need at least three engines (compute engine e.g. Snowflake, orchestration engine, harness engine (Where you run agents)). Don’t put these all on clusters. It’s too much work.

AI means everyone wants to automate everything

AI has made automation accessible to almost anyone. Product, finance, operations and marketing teams can now use tools such as n8n to connect applications, call models and build workflows without waiting for a traditional engineering project.

This is already happening at scale. Trendyol grew n8n from a local proof of concept to more than 1,000 users, 700 active workflows and roughly half a million executions per quarter. Its case study also describes non-technical employees beginning to think about their everyday work as automatable flows. Read the n8n case study. Pretty crazy stuff — I am not sure I would want to be managing that web of dependencies.

This is powerful, but every automation creates another dependency on data, infrastructure and APIs. Without central visibility, governance and observability, citizen automation does not remove work from the data team — it quietly creates more of it. The platform must make automation safe and operable without making engineers responsible for manually supervising everything.

How to build a data stack: the short answer

Figuring out where to spend time needs to go into 4 buckets.

• Architecture: you need to pick your architecture correctly for the type of company you are. That means deciding if you’re going Kubernetes-first or using a Partner like Orchestra

• Buy-in: assess the type of company you are. Your AI readiness is determined by your data readiness

• Budget: understanding if there is any budget for the team and how to cadence it is imperative

• Depth of solution: a data stack’s capability is infinite. You need to find a balance between what you need today, future scalability, and over-enginering

The Modern Data Stack today: where to focus

There are essentially 4 ways to think of this

Scenario 1: the over-zealous CTO

In this scenario you have an over-zealous, penny pinching CTO in a company with very little Data and AI Readiness. This could be a SAAS company where many people are new to the company and data standards are not well defined. In this case, you may assume you need a complex solution but in fact you do not. So you should proceed to Step 1.

Scenario 2: The early company

This is a great scenario to be in where the most important thing is getting buy-in from higher-ups to get more budget. How should you build a data stack in an early company? Focus on high leverage tools like Snowflake, Orchestra, Databricks, and Anthropic. For example, you can use AI to build end to end pipelines using python and dbt running on Orchestra with Snowflake as a data warehouse.

Build these very minimally and always caveat “The data may not be reliable”, until you make it reliable. This is very important, as without effective expectation setting you cannot succeed

Scenario 3: The late stage traditional company

We engineers like to think of ourselves as unique, but we are not. Most SAAS companies and many companies in traditional industries have pretty standard data requirements.

This means that the technologies and patterns we use could in theory be standardised pretty well across them.

In a late stage traditional company, the complexity of solution or uniqueness of solution is low, but the demand for Data and AI is high.

In this case, you should adopt a similar approach to Scenario 2 but with more bells and whistles like observability, metadata, cataloging, and perhaps applications etc. to enable self-serve to scale the company and relieve pressure on the bowtie.

Scenario 4: The late stage technology company

Generally speaking, later stage technology companies are much better at hiring engineers than companies in traditional industries like manufacturing. These people want to build and own more of the stack, and spend more money on engineers instead of SAAS.

However, these engineers are often very bad at internally speccing SAAS Apps. For example, recently I spoke to a team that wanted to build their own MCP atop their data.

Rather than look at Snowflake’s native offering, an existing data MCP like Orchestra, they decided to build their own. It started as a simple Fast API router ontop of Postgres, before it was changed to a Graph DB, before deciding to evaluate other solutions in parallel. 8 months passed, no MCP.

This happens all the time with orchestration too. What starts out as an own-managed Airflow instance becomes ungovernable, so platform teams add Okta Auth, they spin up more instances, they start maintaining clusters, and then they realise they need a simpler way for people to schedule stuff, so they build UIs ontop of Airflow. This leaves a gigantic multi-tenanted unservable mess where a SAAS option like Orchestra would be much easier.

Instead, draw a distinction between platform and functional teams. The best place to leverage platform teams is compute. This means using clusters to save money for the data-heavy bits. Use the cluster to displace warehouse and ingestion compute as these are normally the heaviest. Here’s an example of this below.

Conclusion: build for scale, not burnout

AI does not remove work from the data team. It makes building new things cheaper, which creates more pipelines, more automations and more dependencies for someone to operate.

This is the real bottleneck. Tokens may help an engineer write code faster, but they do not replace architecture, governance or operational ownership. If one engineer is responsible for maintaining clusters, monitoring pipelines, fixing data quality issues and supervising hundreds of citizen-built automations, the platform has already failed — regardless of how sophisticated it looks.

There is no single perfect Modern Data Stack. An early-stage company should prioritise quick wins and expectation-setting. A traditional enterprise needs self-service, observability and governance. A late-stage technology company should reserve its engineers for genuinely differentiated infrastructure instead of rebuilding authentication, orchestration and metadata systems that already exist.

The answer is relatively simple: understand your company’s actual data and AI readiness, budget honestly, choose infrastructure that can scale, and automate the operational layer before demand becomes unmanageable.

For many teams, that means scalable compute through Snowflake or Databricks, transformation through dbt and an Agentic Control Plane like Orchestra to coordinate everything and give less technical users a governed way to build stuff that doesn’t require an army of consultants to maintain.

Your data team should enable the company — not become the human safety net holding its entire AI strategy together.

Frequently Asked Questions

What is a data stack?

A data stack is the collection of technologies an organisation uses to collect, store, transform, orchestrate and analyse data. It might include ingestion tools, a cloud data warehouse, dbt, an orchestration platform such as Orchestra and a business intelligence tool.

What is a modern data stack?

A modern data stack is a cloud-based set of tools for moving, storing, transforming, orchestrating and analysing data. It typically combines specialised platforms rather than relying on one large, all-in-one system. The companies that coined it originally were Fivetran and dbt, but fast-followers like Orchestra (an end-to-end data pipeline automation plaform) swiftly became a big part of it. A Modern Data Stack also goes by the term Modular Data Architecture.

What is modern data stack architecture?

Modern data stack architecture usually includes data sources, ingestion tools like Estuary, cloud storage or a warehouse like Snowflake, transformation software like dbt, orchestration and data quality/observability in a tool like Orchestra, and finally business intelligence. These components work together to move raw data into reliable, analytics-ready datasets. More architecture examples are available here.

Introduction

On 26 August 2026 the team in Amsterdam at DuckLabs / DuckDB and AWS announced the Amazon had signed a definitive agreement to acquire DuckLabs. To paraphrase:

Hannes Mühleisen and Mark Raasveldt, who created DuckDB and co-founded DuckLabs, will continue leading the team and the open-source project’s technical direction as part of AWS. The DuckDB open-source project will also continue to be driven by the DuckLabs team, remain open source under the independent Foundation (the non-profit that oversees DuckDB), and available under the MIT license as it does today (see DuckLabs blog).

The move came as a surprise to much of the community, but was clearly well forecast for those in the know. As seen by the immediate release of Duck DB outgrowing its nest from Jordan (see here) and the general very professional PR job that came with the announcement.

There are many questions.

First — the future of DuckDB. DuckDB has quickly become a community favourite, offering lightning-fast, in memory SQL atop newer and rawer storage formats like iceberg and parquet. It has become embedded in different products; from Databricks to some of AWS’ analytics projects itself.

Typically we have seen open-source projects suffer significantly after acquisition, so you might think it’ll be interesting to see how that happens here.

Secondly — AWS. AWS Has long positioned itself as the “picks and shovels” for creating the modern data stack, with very many companies including Databricks and Snowflake continuing to commit more to AWS. Redshift or “Red****” has been unpopular for a long time. You may think that this signals the start of something different, for AWS.

And finally — what of Motherduck? The company just this week announced the intent to acquire tower.dev, a python runtime provider, which seemed odd given the runtime launch of “Motherduck Flights” not a few months ago. Motherduck has an agreement with DuckLabs, and many assumed the fate of the two was intertwined. Now, it would appear that the flock is less sure.

Let’s DIVE in.

The future of DuckLabs and DuckDB after AWS

The community have been very clear that Duck DB remains OS and company keeps the IP.

Hannes said:

Our team will remain together in Amsterdam, continuing our work on DuckDB, DuckLake, Quack, and the broader community. Joining AWS gives us the resources and reach to bring this technology to many more developers and organizations, and to pursue ideas at a scale that would have been difficult for us to reach alone.

Most importantly, the foundations of the project will remain firmly in place. DuckDB and the other open-source components of the “Duck Stack” will remain free and open source under the MIT license, with the nonprofit DuckDB Foundation continuing its stewardship of the projects.

Really this is not about technology and ownership but about growth.

At the end of the day Hannes loves developing and talking about DuckDB and doing cool engineering stuff. He is an out and out engineer, but is also a huge amount of fun (he lives on a boat).

The team said:

We also worried that scaling DuckLabs into a much larger sales, support, and operations organization would pull our attention away from the technical work and open-source community that made DuckDB successful in the first place.

And this could not be truer. The data and AI market is absolutely savage right now, it is cut-throat, and selling DuckDB-as-a-service when it is so foundational is extremely hard (think about what it would take for you to sell spark-as-a-service or kafka-as-a-service; this is even more low-level than that).

To date, DuckLabs received revenue from development commitments from large organisations around the world. While sustainable, it was likely not scalable. So to firm up the developer env around DuckDB, it made sense that the team would look to bring in another investor.

Terms were not disclosed — which counts too.

Will AWS Create a managed Duck DB and finally get back in to the data stack?

Personally I don’t think so. DuckDB isn’t just “the answer” to the query engine and making iceberg mainstream.

See migrating to a iceberg lakehouse by me

It will be more likely that DuckDB simply gives AWS strategic power over an incredible piece of technology that makes the other hyperscalers think twice about embedding it — that’s a great position to be in.

Adding a small hosted DuckDB Server with logging on stuff like EC2 instances would work well too.

There is a clear product roadmap progression where you could imagine duckdb running on an EC2, then perhaps on ECS, on Lambda, and eventually in a more distributed way (Kubernetes is not necessarily the answer here).

One thing is for sure — redshift is not going away anytime soon.

How does the DuckLabs acquisition impact Motherduck

There are a few important quotations from Motherduck’s blog post

It is possible that in another couple of years this post will look hopelessly naive, as in “I’m sure that inviting the wolves over for a dinner party will turn out fine.”

Some day, Amazon will likely release their own service based on DuckDB. After all, they’re not acquiring Duck Labs just because they love open source. We agree that DuckDB is a great engine for building a cloud data warehouse, and appreciate that Amazon sees that too. We welcome the competition.

They clearly are marketing their service as the enterprise offer for running a warehouse on duck db / running duck db at scale.

It’s an interesting position and one I greatly sympathise with. DuckDB is something that is loved by developers. It is loved by developers for its versatility and for its ability to run on one node.

This means that the scenario where you are a big duckdb lover but also want to actually scale workloads across multiple duck DB instanceson different or similar-sized nodes, and are willing to pay for this — at least as far as being a DuckDB developer goes, is fairly unlikely.

Take nothing away from Motherduck. The warehouse is an incredible product capable of some seriously awesome stuff. Some of their customers refuse to publicise their use of Motherduck because they maintain it is such a competitive advantage they don’t want others to know.

However if you were a developer looking to easily host DuckDB, you probably are more inclined to take it from Amazon Web Services than to force it into a warehouse-centric product.

The other thing is commercial leverage. If your core technology is now owned by a competitor this tends to not be good. To give an example — at Orchestra we built our own state-aware transformation engine called Sao Paolo rather than rely on Fusion as the license kept changing.

Conclusion

It was a crazy week this week with nvidia acquiring Hugging face for almost $13bn, AWS Acquiring DuckLabs and then motherduck acquiring tower.dev at the start of the week. Three acquisitions all with completely different levels of magnitude

What it demonstrates is that these mini-battles are being fought at all levels.

We’ll likely see DuckDB remain an incredible community project but the other hyperscalers will need to think twice about embedding a piece of technology one of their competitors owns. Remember when Oracle bought Java?

For now, as the mother duck would say, keep on swimming.

Sources

DuckLabs acquired by Amazon on Amazon news (link)

DuckLabs join AWS from DuckLabs (link)

AWS and DuckLabs: Building the Future of Analytics (link)

Read free here

Today I am super excited to finally formally announce our fundraise where we raised $3.3m bringing our total raised to $4.6m. The reason we are announcing this now is because we’re seeing a massive opportunity in the market and I’ll explain that to you now.

Firstly demand — we are at the tip of the iceberg in AI adoption. With AI the barrier for doing stuff is lower. So everybody is a builder now. That means A) increased demand but B) demand for things that have more guardrails built-in, as the new ICP is completely different to a traditional data engineer who has been studying Kubernetes handbooks for multiple years

Second — supply. The Modern Data Stack was a fun experiment and now we know that instead of just SSIS in the cloud there are lots of other things we want too. Therefore, there is more to build than ever before, and Data Teams feel that pressure.

In addition — tokens and AI. Data Teams have not doubled headcount, but token budgets have exploded. Now, there is more pressure on teams to deliver than ever. If you have a big token budget and you’re a data team, you have no excuse — even without any more headcount.

Finally — governance. When things get easier, they become less governed. This was the overriding symptom of the “data swamp” — it got easier to put data into object storage as everyone obsessed over “Big data”, and thus everyone began to do it without thinking about how, why, or when. This created these huge “data swamps” of useless data.

The same thing is happening today with AI. Teams are rapidly building hundreds of not thousands of workflows in un-governed tools like n8n, cursor workflows, and claude scheduled tasks. This creates an “AI Swamp” with agents running around, spending money, duplicating work, without supervision and in some cases — any knowledge of their existence.

The problems are big, and the solutions are even bigger. It’s going to be a great year!
HL

LONDON, August 25, 2026 — Orchestra, the Agentic Control Plane for enterprise data and AI workflows, today announced its formal launch following a year in which platform usage grew more than 10x. Backed by $4.6 million in funding to date, the AI-native platform brings pipelines and governed AI agents into one environment, giving data teams greater control over how they build and operate across their existing technology stacks.

AI has created unprecedented demand for working with data, but Data and AI infrastructure supporting that shift is complex and costly. Data teams struggle to scale the Data and AI Platform beyond the data team, and spend their time acting as a bottleneck instead of an enabler. Orchestra removes that bottleneck by seamlessly connecting a company’s existing data infrastructure to form a context-aware control plane anyone can use to reliably automate data and AI workflows.

“AI is creating enormous demand across the enterprise, but data and now AI teams are still managing infrastructure that was never designed to support it,” said Hugo Lu, co-founder and CEO of Orchestra. “Orchestra gives them the control they need to move beyond being a service desk and build the platform teams use to drive AI adoption across the business.”

The company’s Serverless engine can orchestrate millions of tasks and agents in parallel meaning engineers do not need to rely on being able to maintain large clusters. Engineers can build through an intuitive interface or with a few lines of code, then monitor everything from the same place, with end-to-end visibility that exposes the dependencies behind each workflow and helps teams resolve failures before they affect the wider organization. More than 100 core integrations allow organizations to adopt Orchestra without replacing the technology they already use, and companies can implement the platform in weeks rather than spend years assembling the underlying infrastructure.

Customers have cut pipeline runtime costs by up to 80 percent and turned migrations that once took years into projects completed in a couple of months. The platform also cuts pipeline development time by up to 95 percent, while automating hundreds of failure investigations every day.

“We’re moving our entire dbt estate from Airflow to Orchestra,” said Mike Holmes, Global Head of Data Engineering, Octopus Energy. “We’re excited about running agents alongside our data, giving them direct & secure access to the context they need to take action. A big plus is keeping our stack simple & easy to work with.”

Orchestra Runtime offers a secure environment for operating AI agents. Each agent runs in isolation with controlled access to the resources needed for its assignment. Companies can use Runtime to create pipelines, investigate production incidents and automate failure triage.

Runtime works with Orchestra’s Context Layer, which organizes information about an enterprise’s systems and workflows. This gives agents the relevant context to complete data engineering tasks accurately without opening the wider environment. Its model-agnostic design allows companies to choose the models that best meet their needs while using fewer tokens.

Together, Orchestra’s core orchestration platform and Runtime allow data engineers to support AI use across the enterprise without manually handling every request. Teams can provide a shared foundation that enables others to work with data and AI while preserving security and oversight.

The launch is supported by $4.6 million in total funding, including a $3.3 million seed round led by Differential Ventures and a $1.3 million pre-seed round led by Moonfire, which returned for the seed financing. Additional backing came from Breakers, Tokyo Black, and Contour Ventures. The company is using the capital to expand its engineering and go-to-market teams in London, advance product development, and support growth across the United States and Europe.

“The industry has only seen the tip of the iceberg of enterprise AI adoption and one of the most significant barriers to powerful AI systems deployed in production is reliable data pipeline management,” said Nick Adams, co-founder and managing partner at Differential Ventures. “Orchestra’s platform represents a major leap forward toward that objective by delivering a single data orchestration layer that customers love.”

ABOUT ORCHESTRA:

Orchestra is the Agentic Control Plane for enterprise data and AI. It brings pipelines and governed agents into one environment so organizations can work faster with greater reliability and oversight. Companies like Experian, Octopus Energy, and more are using Orchestra as the foundation for their operations. Headquartered in London, Orchestra serves customers across the United States and Europe. Learn more at getorchestra.io.

MEDIA CONTACT: Lauren Gill, MAG PR at lauren@mooringadvisorygroup.com.

What has always fascinated me is the difference in sales culture vs. product in the data industry.

Companies like Snowflake are very difficult to partner with, but in practice a bit easier as their ecosystem is missing core functionality that naturally lends itself to partnership.

Companies like Databricks commercially are easier to partner with, but in practice finding the niche is harder.

This week, we announced that Orchestra is officially a databricks partner (woo!). But it hasn’t been easy. Partnering with the megaton gorilla in the room is hard - especially when they are building everything under the sun.

There are a few interesting things to note here.

1. Outcomes matter

Companies like Sigma have done an exceptional job at partnering with organisations like Databricks, not least because they send a ton of compute down there. This is key, but something that is also key is outcomes. The BI Tools are able to drive outcomes and adoption metrics for Databricks where “Apps alone” doesn’t quite get the buy-in a single vendor might push for.

Similarly at Orchestra, we send a ton of compute via the pipelining and transformation layer. This is attractive at scale, because databricks workflows doesn’t cut the mustard or scale well across multiple domains

2. Partnerships must find a niche

Ultimately there must be a niche and a scale problem for people to partner with companies like Databricks. They almost begin to resemble Azure, offering everything without one thing being absolutely killer or standalone.

3. AI is critical

There are rumours spreading that Databricks recently signed its first $1bn dollar deal which is pretty incredible and almost certainly has AI as a critical part. Unity Catalog is not the perfect context store for data teams, and running all your AI on Databricks does not meet the definition of “Meeting people where they are”. Embedding Databricks Services as first party citizens in AI products will, I think, be an increasingly important part of how organisations partner with large companies in the data space going forward.

Anyway we are very excited about this. If you know folks managing multiple teams on Databricks send them this! Here is the full announcement.

Orchestra is Officially a Databricks Partner

Today, we are excited to announce Orchestra’s partnership with Databricks.

Databricks has become the platform on which many of the world’s largest organisations build their most important data and AI workloads. Orchestra gives those teams the control plane they need to build, run and monitor those workloads alongside the rest of their data stack.

Together, Databricks and Orchestra provide a powerful foundation for enterprise data and AI: Databricks supplies the scalable compute, storage, governance and intelligence layer, while Orchestra coordinates the end-to-end workflows that make everything work reliably in production.

“Databricks gives data teams an incredibly powerful platform for building data and AI products. The challenge is coordinating everything around it: dbt^TM, Lakeflow, ingestion, BI, cloud infrastructure and now AI agents,” says Orchestra CEO Hugo Lu. “Our partnership with Databricks gives enterprises a simple, scalable control plane for doing exactly that—without forcing them to build and maintain another orchestration platform themselves.”

The partnership formalises work Orchestra is already doing with some of the largest and most advanced data teams in the world.

Octopus Energy, the UK’s largest energy provider, uses Orchestra with Databricks to power a multi-tenanted analytics platform containing more than 5,000 dbt models across over 10 business units. Instead of maintaining a custom fork of Airflow, its teams can use Orchestra’s state-aware orchestration to run dbt intelligently on Databricks and give engineers a single place to understand the health of the wider platform.

Engineers can work in YAML and Git, while analysts and other data practitioners can use Orchestra’s visual interface. Platform teams retain control through environments, workspaces, RBAC, SSO and SCIM without forcing every contributor to become an Airflow expert.

That is exactly what this partnership is about: helping enterprises get more from Databricks while reducing the orchestration code, infrastructure and operational overhead required to run it.

What is Orchestra?

Orchestra is the agentic control plane for data and AI workflows.

It gives data teams a declarative, code-first and GUI-driven environment for building pipelines, running dbt, coordinating infrastructure and executing AI agents. Orchestration, lineage, observability, alerting, data quality and incident management are included in one platform.

Orchestra is not the data-processing engine. Workloads continue to run on the customer’s own compute—including Databricks—while Orchestra coordinates execution and collects the operational context teams need to understand what happened.

This serverless architecture is particularly valuable for enterprises using Databricks as part of a wider ecosystem. A pipeline may start with ingestion through Fivetran, Airbyte or a custom Python service, continue through Databricks and dbt, and finish in Power BI, Tableau, Sigma or an operational application. Orchestra coordinates the entire workflow rather than stopping at the boundary of a single platform.

Data teams therefore get the power of Databricks without having to force every workload into Databricks or maintain a separate orchestration framework around it.

How does Orchestra integrate with Databricks?

Orchestra provides a deep integration with Databricks that combines workflow execution with metadata, lineage, observability and data quality.

Teams can use Orchestra to coordinate Databricks SQL, notebooks, one-time runs and Lakeflow Jobs inside larger end-to-end pipelines. Runs, dependencies, logs, retries and alerts are available in one place, including the steps that execute outside Databricks.

This is not simply a thin API call wrapped in a generic task. Orchestra understands what the underlying operation is doing and brings the resulting operational context into the control plane.

The result is a much better developer and operator experience: less boilerplate, faster root-cause analysis and complete visibility across the data stack.

The best way to run dbt^TM Core on Databricks

Databricks is an exceptional compute engine for dbt, but operating a large dbt project still creates difficult scheduling and observability problems.

As projects grow, teams typically create more tags, selectors and scheduled jobs. Models are frequently rebuilt even when neither their code nor their upstream data has changed. Different teams create overlapping schedules, while engineers are left piecing together dbt artifacts, Databricks logs and orchestration metadata when something fails.

Orchestra’s state-aware orchestration solves this problem without requiring teams to replace dbt Core with a proprietary transformation engine.

Orchestra stores and understands the previous state of the dbt project. It can then rebuild a model when its code has changed or fresh upstream data is available, while safely reusing models that do not need to run again.

‍

This gives teams running dbt on Databricks several immediate benefits:

• Faster dbt runs because unchanged models can be reused

• Lower compute costs because Databricks does not process data unnecessarily

• Simpler scheduling without maintaining a growing collection of tags and selectors

• Clear lineage showing which models ran, which were reused and why

• A shared state across pipelines within an Orchestra workspace

Octopus Energy demonstrates what this looks like at genuine enterprise scale. Orchestra state-aware orchestration helps the company coordinate more than 5,000 dbt models on Databricks across a multi-team, multi-domain environment. This is not a small proof of concept; it is an operating model for one of Europe’s most ambitious technology businesses.

One control plane for Lakeflow, notebooks, SQL and dbt

Enterprises rarely standardise on a single way of building inside Databricks.

One team may use notebooks. Another may use dbt Core. A third may build batch or streaming workloads using Lakeflow Spark Declarative Pipelines. Platform teams may package jobs and pipelines through Declarative Automation Bundles, formerly known as Databricks Asset Bundles.

Orchestra provides a common control plane above them. Teams can coordinate native Databricks workloads alongside external ingestion, transformation, BI and activation services. They can define dependencies declaratively, pass outputs between tasks, trigger workflows from events or schedules, and rerun from the point of failure.

This allows teams to use the right Databricks capability for each workload while retaining one consistent approach to orchestration, monitoring and incident response.

Extending Unity Catalog as the Context Layer for AI

Unity Catalog provides a governed foundation for discovering and controlling data and AI assets in Databricks, but it lacks specific operational and business context often living outside of Databricks. Through the Context Layer, Orchestra provides this metadata which acts as a foundation for the Agent Runtime - a single place to run AI Agents leveraging Databricks.

Orchestra can track tables and views registered in Unity Catalog, including column metadata and data quality test activity. It then combines this asset context with pipeline runs, dbt lineage, logs, failures and dependencies from across the wider stack.

This matters because enterprise lineage does not begin and end inside a single compute platform. Source data may arrive through an external ingestion service, while the final asset may be consumed by a BI dashboard or operational workflow elsewhere.

This means that the Agent Runtime uses less tokens than claude, achieves its goals in a fraction of the time, and is able to debug and diagnose pipeline failures end-to-end with greater reliability.

Who uses Orchestra and Databricks?

Orchestra is designed for enterprises and fast-growing teams using Databricks for analytics engineering, data engineering, machine learning and AI.

It is especially valuable where:

• Multiple teams or business domains share a Databricks platform

• dbt projects have grown too large for simple cron-based scheduling

• Databricks workloads must be coordinated with tools outside the Lakehouse

• Teams need end-to-end lineage and observability without building it themselves

• An existing Airflow, Azure Data Factory or custom orchestration estate has become expensive to maintain

• Organisations want to introduce AI agents without sacrificing security, governance or control

For migration and transformation programmes, introducing a unified control plane early prevents orchestration logic, alerts and metadata from becoming fragmented across notebooks, scripts and individual tools.

‍

See Orchestra customer stories to learn how enterprise data teams are simplifying their platforms and delivering data products faster.

What is next for Orchestra and Databricks?

This partnership is the beginning of a much deeper collaboration. We will continue investing into providing greater visibility for data teams into other parts of the platform such as Lakeflow declarative pipelines, Databricks Apps, and Agents.

We will look to support models hosted in Unity Gateway via the Orchestra Agent Runtime. We will continue to invest into the Orchestra Context layer as a means of hydrating Unity Catalog to create a single place for engineering Context as well as pipelines.

Databricks gives enterprises the platform to build the future of data and AI. Orchestra gives them the control plane to run it. We’re very excited to continue this awesome partnership.

Find out more

• Explore Orchestra for Databricks

• Read the Databricks integration documentation

• Learn about state-aware orchestration for dbt Core

• Try Orchestra

• Book a conversation with the Orchestra team

For partnership, customer or press enquiries, contact legal@getorchestra.io.

An introduction

Recently we’ve seen a ton of movement in the market with consolidation and it almost begins to sound like a broken record. However — data quality, trust, speed of delivery, and value are bigger problems than ever. Data Teams are getting cut. More is being asked of Data Teams. Executives want data faster. Data Platform Teams are now AI and Data Platform Teams. So there is more work than ever to be done, and paradoxically, we seem to have less hands albeit more tokens to do it.

ELT, or specifically EL (extract and load) remains a critical part of the stack. Every enterprise data engineer I speak to is principally concerned with moving data, replicating databases, and replicating data across regions.

However, there is some light forming. API connectors are now so easy to build, you can just get AI to write the code for them, leaving infrastructure as the missing piece.

Database replication remains something that is gnarly. One of the final challenges or “ELT FINAL BOSS” might be cross-region replication across different buckets in different cloud providers in different regions; that’s really exciting stuff.

In this, I want to cover a bit about how setting up pipelines means you now need best-of-breed to really feel like an “AI-Enabled engineer”. What is best of breed? Well — certainly not an all-in-one platform.

There is a growing divide between those who “simply do everything in Fabric”.

These people are willing to accept that Microsoft has the biggest fist in the room, and consequently are happy to have their orchestration, monitoring, and ELT subsidised by their extortionately priced Power BI licenses.

Databricks to an extent are in the same category, though Databricks is a better product.

Snowflake want to be in that category, but put simply Snowflake Tasks, Snowflake dbt, Snowflake Horizon, OpenFlow etc. are inferior products to their Databricks counterparts, and far inferior to those things on the market specialising in their niche.

So perhaps it makes sense to have a complex architecture with Fivetran, dbt Cloud, Apache Airflow, Kubernetes, Datadog, Monte Carlo, Atlan, Cube.dev and Tableau?

Perhaps not — such a combination is rather expensive, and rather hard to stitch together. Let’s face it — there’s a lot of overlap between those tools that were all once heralded as “Best of breed” but they’re not anymore. Why not?

It is telling that companies like Informatica, that used to be best of breed, no longer are because they are “too clunky”. The same is said for companies like Matillion, though they have a new CEO and a ton of new hires, which should tell you something.

Fundamentally these companies offer three things

1. Data movement

2. Some form of data transformation

3. Some limited alerting and orchestration / scheduling

The biggest missing piece in all of this? An enterprise orchestrator. The biggest and #1 time-sink for companies without an orchestrator operating at scale is building, fixing, debugging, and answering questions stakeholders have of “where is my data” and “Why isn’t this fresh?” — the answer as you all know — is a lack of orchestration, and a lack of observability.

Best of breed is no longer the categories that the Modern Data Stack gave us. Best of breed for me at least, now encompasses a few smaller categories

1. Data Movement and Data Replication

2. Data Storage and Compute

3. Control Plane (Orchestration + Observability + Metadata + incident management + runtime for agents)

It is no surprise to see Fivetran and dbt pitch themselves as everything you need. They did this last year.

This was the same pitch as Informatica, Matillion and lo! The trend is coming back.

Apart from you us data engineers have memories, and they’re short but not that short!

Further examples and reading lifted from last week - skip to the end for conclusion

Example: How to build an ELT Pipeline end-to-end using AI with Estuary and Orchestra

Orchestra does can move data using python and it transforms data using dbt. It is a baby step in ELT; where you use Orchestra to run python to move data before graduating to a managed ELT tool.

Its main job is to decide whether the next thing should run, given the state of everything upstream of it. That framing sounds narrow until you notice that most of what upstream consists of is software we did not write and cannot inspect from the inside.

This means the interesting engineering in an orchestrator is almost entirely about integrations. Every one we ship is an answer to a specific version of the same question: how does a scheduled task find out whether the system it depends on is in a state worth depending on.

Daniel Pálma at Estuary recently built a pipeline that made this concrete enough to write about, and the result is a good illustration of why I think about integrations the way I do.

The setup: postgres CDC to Snowflake using Estuary

Postgres CDC into Snowflake through Estuary, running continuously. On top of that, an Orchestra pipeline running daily: dbt build, then four Snowflake data-quality tests. Seven tasks, scheduled.

The whole thing was authored by agents using two skill packs, ours and Estuary’s, against live accounts. It took nine runs to go green. Five of those nine runs failed at the same place, which is the part worth talking about.

The gap that best-of-breed opens up: streaming

Traditional orchestration assumes discrete work. A job starts, does something, exits with a code, and the orchestrator reads that code to decide what happens next. The model is clean because the boundaries are clean.

A streaming CDC pipeline has no boundaries of that kind. It is always running. There is no exit code, and there is no moment at which it declares itself finished. So when a daily dbt build wakes up and reads a warehouse table that a streaming pipeline has been writing to for the last twenty-four hours, it has no principled way of knowing whether that table is trustworthy.

This is the new best-of-breed. You need to basically just assume that data is getting streamed and replicated in real-time, especially if it’s db stuff. There is less stuff to orchestrate, but no less stuff to monitor.

Takeaway #1: streaming reduces the need for orchestration but increases the need for observability

In practice teams close this gap by writing a Python task that calls an API, parses a response and raises an exception. I have seen that task written many times. It is written slightly wrong most times, usually because the person writing it picked the first available signal rather than the right one, and because the failure semantics were decided in a hurry.

That pattern is exactly the shape of thing an integration should absorb. So ESTUARY_CHECK_FLOW takes four parameters:

check_capture:
integration: ESTUARY
integration_job: ESTUARY_CHECK_FLOW
parameters:
task: dani-demo/agent-demo/source-postgres
error_threshold: 0
warn_threshold: 0
latency_threshold: 1800
connection: ${{ ENV.ESTUARY_CONNECTION }}

It reads the Estuary task’s OpenMetrics endpoint and compares deltas against the previous check. New logged failures above threshold, or latency above threshold, and the task fails. No transaction progress, and it warns.

Read metrics, not liveness for observability

The first design rule I hold integrations to is that a check has to tell you something you could not have learned from a ping.

A liveness endpoint tells you a process is up. That is close to useless as a precondition, because the failure mode you actually care about is a pipeline that is running and not making progress. logged_failures_total, txn_count_total and publish latency are the signals that distinguish those two states, and reading deltas rather than absolutes is what makes them meaningful across scheduled runs.

There is a cost to this precision, and Dani found it. Our latency metric measures commit acknowledgement, so it reads in the hundreds of seconds even when Snowflake is nine seconds behind Postgres. A threshold set from a data freshness SLA will false-alarm continuously. That is on us to document better, and the practical advice is to tune the threshold against the metric you are actually reading.

Takeaway #2: observability and visibility is critical for streaming pipelines which are best-in breed, but it needs to be combined with visibility of batch pipelines

Three states, because infrastructure is mostly degraded

The second rule, and the one I would defend hardest: a health check needs three outcomes rather than two.

Binary checks force an unpleasant choice. Make them strict and they block good runs on transient noise, which teaches your team to ignore or remove them. Make them permissive and they let dbt build models on stale data, which is the failure they existed to prevent. Neither version survives contact with real infrastructure, because real infrastructure spends most of its life somewhere between fine and broken.

So WARNING allows downstream tasks to proceed and FAILED blocks them. A materialization syncing slowly warns and the batch layer runs. A capture that is throwing connection errors stops the line.

Dani’s run 4 is the clearest evidence that the semantics are right. Both Estuary checks came back WARNING, the pipeline continued, and dbt then failed on its own unrelated credential problem. The gate did not hide that failure and did not manufacture one.

The AI was able to understand all of this and explain it to me, because Orchestra had a lot of the metadata and context about the Estuary and dbt runs together anyway.

This leads to:

Takeaway #3: context and agents about pipelines are critical for building AI data Engineering resources

Preconditions belong in your control plane

Most people describe orchestration as running things in the right order. The framing I find more useful is that an orchestrator enforces conditions.

Data quality tests are postconditions. They run after a transform and tell you whether the output is acceptable, and everyone already accepts that they belong in the pipeline. Health checks are the same idea pointed the other way. They run before a transform and tell you whether the input is acceptable.

Both are cheap, both are declarative, and both should be visible in the same lineage view as the work they guard. When a precondition is buried inside a Python task, nobody can see why the pipeline stopped without reading code. When it is a task in its own right, the run history answers the question directly.

These are not separate concepts that require separate tools. They’re just things you should be able to do in a modern tool.

What agents changed about how we build APIs

Both skill packs in this build are agent-facing, and ours act through roughly 28 MCP tools rather than a CLI, because orchestration state lives behind an API rather than in files on disk.

The lesson I took from watching an agent use them was not about authoring. Agents write first-draft YAML badly. Across the two products Dani’s first drafts had six schema errors, including one of ours:

422: Invalid parameters for DBT_CORE_EXECUTE: commands - Input should be a
valid string, input_type=list

That error is the useful part. It names the field, states the expected type, reports what it received, and costs nothing to trigger, because validate_pipeline runs in seconds and writes nothing. An agent reads that and fixes it. A vaguer error, or an expensive one that only surfaces after publishing and waiting for a run, would have stalled the loop.

Which reframes something I used to think of as developer courtesy. Specific errors and free dry-run validation are now the difference between an API an agent can iterate against and one it cannot. If you are building anything with a public API right now, the strictness of your validation layer is a feature, provided the messages are precise about what went wrong.

The same holds for diagnosis. The agent found the bug in run 6 by pulling logs through three MCP calls, and the log filename gave it away before it read any content:

“1/dbt deps dbt build — select stg_orders daily_order_metrics”

One command where there should have been two. Reading that correctly is a more valuable capability than generating the YAML in the first place.

Takeaway #4: with AI, everything must be controlled as code or editable via API/CLI so they can be agent managed

Where this still needs a human

Worth being precise, since the story is partly a story about automation. Agents authored, published, diagnosed and fixed everything. A human was required for exactly three steps, all of them credential handling: creating the Orchestra connections, minting a GitHub read token, and supplying Snowflake credentials that Estuary encrypts and never hands back.

Connections being UI-only is the one I think about most. There is a real tension between making credentials scriptable and making them safe to hand to an agent, and I do not think the industry has settled it. Mapping existing connections to environment variables is already automatable through update_environment. Creating them is deliberately not.

The main takeaway is to go best-of-breed or suffer

The reason to run tools like Estuary and Orchestra together is that streaming health becomes a first-class precondition for batch work, expressed in four parameters and visible in the same lineage as everything it guards.

Five of nine runs in this build ended with the gate refusing to let dbt read the data. A demo that goes green on the first attempt tells you nothing about whether your preconditions are load-bearing. This one told us they are.

AI was used to generate everything and debug everything which was incredible to see. Everything was controlled as code. Everything you need for a best-in-class experience was handled with just three tools — Estuary, Orchestra and Snowflake.

Dani’s full technical writeup, including the gotcha catalogue and the demo repo: HERE

ALSO don’t forget — nice features like state-aware orchestration make everything REALLY easy to schedule. State means you can just run the same job over and over again — you don’t even need to have it in the same pipeline, since it won’t run anything erroneously that doesn’t have new source data; as the orchestrator catches the state.

This can help teams go incredibly far. It even scales to multiple teams, with both Estuary, Orchestra and Snowflake supporting multiple deployments for multiple teams across multiple regions.

The other point is scalability. I genuinely believe this is scalable as hell, since there is no infrastructure to maintain, no connectors to maintain, and everything can be automated on the maintenance side using AI. This is something I’ve never seen before as a data engineer, and it’s truly incredible to see with many shared customers like Trust&Will and Raylo leveraging Orchestra alongside Estuary to be best of breed.

Best-of-breed have to better and cheaper than the incumbents to make a business case. Those that can do it will be the best teams, with the least funding pressures. There aren’t many companies like that so when you find them act fast!

HL

I asked my friend Dani at Estuary to give our Agent a go at building an end-to-end data pipeline in Orchestra using Estuary and Orchestra’s State-Aware dbt. The premise is simple - someone of Dani’s calibre with the data engineering context should be able to use natural language and the Orchestra Runtime to build a pipeline that’s pretty complicated from scratch end to end. He wrote up the results, and it’s an awesome article on how to build data pipelines using AI - one of my favourites. Anyway, enjoy.

Estuary and Orchestra both offer agent skill packs. Estuary’s cover data movement, while Orchestra’s cover orchestration. Each is useful alone, and I wanted to know whether they compose, so I described a production-like agentic data pipeline in plain English and let agents build it against live infrastructure: continuous CDC with a scheduled delivery from Postgres to Snowflake, plus a daily dbt-and-tests layer gated on the health of that stream.

After chatting with my agent for a few rounds, I got the whole stack working in less than one hour; that includes data ingestion, transformations and orchestration. A year ago, this would have taken me at least a few days of trial and error.

Everything below executed against live Estuary, Snowflake, and Orchestra accounts. The error messages, statuses and row counts are copied from the runs.

In this article, “agentic” means that an AI coding agent can select skills, author pipeline specifications, call validation tools, inspect execution logs, and revise the pipeline based on the errors it receives.

Key results from the agentic pipeline test

Here’s what I built condensed into a few bullets:

• What was built: Postgres CDC into Snowflake via Estuary, then a scheduled Orchestra pipeline that gates dbt and a few data-quality tests on top of Estuary task health repoting.

• Who wrote it: Agents authored, published, diagnosed and fixed every spec. Human involvement was limited to three steps all having to deal with managing credentials.

• Key integration: Orchestra’s ESTUARY_CHECK_FLOW job reads an Estuary task’s OpenMetrics endpoint and turns failure count, latency and transaction progress into a pass, warn or fail.

• Random cool finding: flowctl catalog test and validate_pipeline both run in seconds, exercise real credentials, and write nothing. A cheap, useful habit!

Agentic data pipeline architecture

Alright, let’s take a look at what we’re building from a distance.

The division of labour carries the design. Estuary owns continuous movement, with seconds of latency for the extraction and a user-defined schedule for delivery. Orchestra owns the periodic transform-and-validate layer and gates it on Estuary’s health. Neither tool duplicates the other’s job, and the gate in-between is what makes this one system instead of two pipelines sharing a warehouse.

Set up the Estuary and Orchestra agent skills

As a first step, let’s take a look at how we can enable our agents to do the actual work. Both packs are installed as Claude Code plugins.

# Estuary
/plugin marketplace add estuary/agent-skills
/plugin install estuary-captures@estuary
/plugin install estuary-materializations@estuary
/plugin install estuary-operations@estuary

# Orchestra
/plugin marketplace add orchestra-hq/orchestra-skills
/plugin install orchestra@orchestra-marketplace

Orchestra’s skills are MCP-native. They act through roughly 28 tools instead of a CLI, so the server has to be registered too.

claude mcp add orchestra https://mcp.getorchestra.io/orchestra \
  --transport http \
  --header “Authorization: Bearer <ORCHESTRA_API_KEY>”

Restart the client afterwards, because plugins and MCP tools load at startup. Mine were unavailable until a reload, which forced an entertaining detour covered at the end. If you are scripting this headlessly, claude plugin marketplace add … and claude plugin install … are the CLI equivalents of the slash commands.

What each agent skill pack covers

Both agent skill packages can do a lot, and this demo is only going to use a friction of those capabilities.

With that said, let’s start building!

Building the PostgreSQL CDC to Snowflake Pipeline with Estuary

For the demo purposes, we’ll be using a Postgres with wal_level=logical in Docker, a generator that continuously inserts rows, advances them through pending → shipped → delivered and deletes some, so the capture sees all three CDC operations rather than an insert-only stream, and ngrok tcp 5432 to expose it to Estuary’s cloud.

Then four prompts.

1. Check that flowctl is installed and authenticated.

Routes to estuary-flowctl-setup. The skill thankfully knows one trap that costs people time: flowctl catalog list --prefix “” returns PermissionDenied: user is not authorized to access prefix or name ‘’, which reads like an auth failure and is only prefix scoping. The real answer comes from flowctl auth roles list, which showed admin on dani-demo/, the tenant I’ll be using for this demo.

2. Capture my Postgres orders table into Estuary.

Routes to capture-postgres-create. The skill declined to hand-write a schema. It wrote a spec containing only the endpoint block and bindings: [], then ran discovery.

flowctl discover --source discover.flow.yaml --flat

Discovery found public.orders, excluded flow_watermarks as connector plumbing, smartly inferred key: [/order_id], and emitted four files: the capture spec, a write schema, a read schema, and a sops-encrypted config.

That matters more than it sounds, because a hand-written guess and a discovered spec differ in ways that bite later.

Documents landing in the collection carry their CDC metadata:

{
   "_meta":{
      "op":"c",
      "source":{
         "schema":"public",
         "snapshot":true,
         "table":"orders"
      }
   },
   "amount":"129.48",
   "customer_name":"Joseph Garner",
   "status":"pending",
   "order_id":"0034c6c6-…",
   "event_ts":"2026-07-29T10:50:19.173331Z"
}

amount arrives as a string, and that’s fine. Postgres numeric and timestamptz both land as JSON strings, which is the reason a dbt staging model exists downstream. If you want to verify, head over to the Estuary dashboard and take a look at the collection details. It should look like this:

3. Materialize the orders collection into Snowflake.

This prompt routes to materialize-snowflake-create. Before publishing, the skill validated against the real warehouse without writing anything.

flowctl catalog test --source mat.flow.yaml

# Tests successful ✅

flowctl catalog test is the underrated command in this build. It exercises real credentials and connectivity and commits nothing. Every credential problem I hit surfaced here, in seconds, rather than minutes after a publish. Definitely helpful.

After publishing, logs showed storedHistory: [247, 1011, 7, 5, 5]: two backfill batches of 247 and 1,011 rows, then incremental commits of 7, 5 and 5. Snowflake was 7,272 rows in and 9 seconds behind Postgres. But that’s okay, the pipeline quickly caught up and new changes started streaming in 5 minute internals (which I configured via the sync schedule)

4. Verify pipeline health and data movement

Routes to estuary-task-health, estuary-task-stats and estuary-logs.

dani-demo/agent-demo/source-postgres        Streaming CDC Events

dani-demo/agent-demo/public/orders          Ok

dani-demo/agent-demo/destination-snowflake  Running

When exploring task health, one flag came back: recentFailureCount: 14. Log analysis grouped all fourteen as unable to query ‘wal_level’: failed to connect to user=flow_capture, dating from an ngrok tunnel restart, even though nothing was currently failing.

Those fourteen historical failures are about to turn an Orchestra task red, which is a good demo of the quality gates!

Orchestrate dbt and Snowflake data-quality tests

My prompt here was: “Create an Orchestra pipeline that first checks my Estuary capture and materialization are healthy, then runs dbt build, then runs Snowflake data-quality tests, daily.”

This is routed to the skill called create-orchestra-pipeline.

The skill produced three task groups chained with depends_on, then called validate_pipeline before publishing anything. Validation caught two small errors in the first draft.

422: Invalid parameters for DBT_CORE_EXECUTE: commands — Input should be a

     valid string, input_type=list

422: alerts.0.statuses.1 — Input should be ‘ANY_COMPLETED’, ‘CANCELLED’,

     ‘FAILED’, ‘SUCCEEDED’, ‘WARNING’ or ‘SKIPPED’

The agent managed to fix and revalidate both in a few seconds: Pipeline schema is valid, published as daily_orders_analytics_agent_demo with 7 tasks at v1. validate_pipeline is free and fast, so publishing unvalidated has no upside.

Second prompt: “Add Snowflake quality tests on the marts: freshness within 24h, no null or negative revenue, status limited to the known enum, and no future-dated events.”

Routes to the skill write-snowflake-dq-tests.

The pattern it used is worth a few sentences to explain: Every SNOWFLAKE_RUN_TEST statement is written so that returning rows means something is wrong, and error_threshold_expression: ‘> 0’ turns any violating row into a failure. Freshness needs a HAVING clause so the query emits exactly one row when data is stale and zero rows otherwise.

select max(updated_at) as newest
from ESTUARY_DB.AGENT_DEMO_ANALYTICS.STG_ORDERS
having max(updated_at) < dateadd('hour', -24, current_timestamp())

The skill’s reference doc is opinionated in a way I have started repeating to people. It insists on profiling every candidate column before writing a test, and it draws the design rule from semantic type rather than SQL type: a VARCHAR holding a status code and a VARCHAR holding a free-text note need entirely different tests.

Gate dbt on Estuary task health in Orchestra

This is the part most likely to be new, and it is what makes the two tools compose really well. Orchestra ships a native ESTUARY integration. Everything is defined as .yml, which is nice.

check_capture:

  integration: ESTUARY

  integration_job: ESTUARY_CHECK_FLOW

  parameters:

    task: dani-demo/agent-demo/source-postgres

    error_threshold: 0        # any new error fails the run

    warn_threshold: 0

    latency_threshold: 1800   # seconds since the task last published

  connection: ${{ ENV.ESTUARY_CONNECTION }}

The job reads the Estuary task’s OpenMetrics endpoint and compares deltas against the previous check.

Real messages collected across the nine runs, which show it inspecting the stream rather than pinging a liveness endpoint:

FAILED     Task reported 10 new failure(s). logged_failures_total delta > 0.
FAILED     Latency 1665s exceeds threshold 900s.
WARNING    No new transactions (txn_count_total delta = 0).
SUCCEEDED  txn delta 3150
SUCCEEDED  txn delta 1; latency 694s

WARNING allows downstream tasks to run and FAILED blocks them. That distinction is critical: a slow-syncing materialization warns and lets the batch layer proceed, while a broken capture stops the line.

The connection itself is a refresh token plus a default Estuary task name, which Orchestra uses to build the metrics URL and verify the token.

Nine test runs: from failed validation to a green pipeline

To give you an idea of the minimal trial and error steps the agent worked through while setting up the pipeline, here’s a summary.

Each run failed differently, and each failure taught something specific.

Run 4 convinced me the semantics were correct, although both checks warned, downstream ran anyway, and dbt failed on its own merits.

Around run 8 I was convinced the gate was working well. A 945-second latency reading against a 900-second threshold, and five downstream tasks skipped rather than run against data Orchestra could not vouch for.

And finally… drumroll please… run 9:

[SUCCEEDED] Check Postgres capture           ESTUARY     txn delta 173
[SUCCEEDED] Check Snowflake materialization  ESTUARY     txn delta 1; latency 694s
[SUCCEEDED] dbt build                        DBT_CORE    All dbt commands finished
[SUCCEEDED] Orders are fresh                 SNOWFLAKE   Test passed - returned 0 rows
[SUCCEEDED] Revenue is non-negative          SNOWFLAKE   Test passed - returned 0 rows
[SUCCEEDED] Status values are known          SNOWFLAKE   Test passed - returned 0 rows
[SUCCEEDED] No future-dated orders           SNOWFLAKE   Test passed - returned 0 rows

How the agent diagnosed and fixed pipeline failures

This is where the agent earned its keep, by reading logs rather than by writing YAML. identify-pipeline-error routes into fix-orchestra-pipeline, and the whole loop runs over MCP.

list_task_runs_for_pipeline_run  → which task failed, and its integration
list_task_run_logs               → what log files exist
download_task_run_log            → the actual stderr

For run 6, the log filename gave the bug away before reading a byte of content.

“1/dbt deps dbt build --select stg_orders daily_order_metrics”

One file, one command. The newline in my commands string had been flattened and the content confirmed it.

[DBT CORE] Installing dbt dependencies          ← Orchestra runs dbt deps itself

[DBT CORE] Parsed dbt command(s): dbt deps dbt build --select stg_orders …

[DBT CORE] Running command #1: ‘dbt deps dbt build --select stg_orders …’

Usage: dbt deps [OPTIONS]

Error: No such option ‘--select’.

[DBT CORE] exited with code 2

Two findings came out of one log: the flattening, and the fact that dbt deps was redundant because Orchestra already installs packages.yml dependencies when it clones the repo. The fix was to delete a command.

Where human intervention was still required

“The agent built my pipeline” is a claim that deserves a boundary. Agents did the authoring, publishing, diagnosing and fixing. I was required for exactly two things, all related to credential handling. Looks like that is something platforms currently don’t want to let agents handle. Fair.

Everything downstream of those two was the agent’s work, including mapping the connections to environment variables, which update_environment handles with type integration_credential even though creating the connections does not.

The dbt Core connection illustrates the boundary well. Pasting the repo, token and profiles.yml is mine. Everything about what runs against it belongs to the agent.

Production lessons and configuration gotchas

The full catalogue from this build runs to 25 items, annotated inline in the demo repo. These are the ones I would have paid money to know in advance.

DBT_CORE_EXECUTE.commands takes a single string holding a single command. Validation rejects a list, and newlines then get silently flattened into one shell invocation, so “dbt deps\ndbt build” becomes dbt deps dbt build … and exits 2. dbt deps is also unnecessary, since Orchestra installs packages.yml for you.

ESTUARY_CHECK_FLOW’s latency metric tracks commit acknowledgement. It says nothing about row freshness. I measured 694 to 945 seconds while Snowflake sat 9 seconds behind Postgres. latency_threshold: 900 false-alarms even on a 5-minute sync, and 1800 is a realistic floor. Tune this against the observed metric rather than against your data SLA.

The first check has no baseline, so it reports the task’s entire failure history as new. Looks like run one is always noisy, so judge from run two onward.

Keep flowctl current. Flowctl gets updated frequently, make sure you have the latest version to avoid any backwards incompatibilities.

A handful of smaller ones:

• The Snowflake materialization connector creates tables and leaves schemas alone. Create the target schema first, or get Schema ‘ESTUARY_DB.AGENT_DEMO’ does not exist or not authorized.

• Field names are snake_case: private_key rather than privateKey. timestamp_type is required, and warehouse is required unless the user has a default.

• Estuary config changes wait for the current transaction. A connector on a 30m sync finishes its acknowledgement delay first. The symptom is logs still showing the old table name and delay: 29m56s long after a successful publish. Disable and re-enable the shards.

• Orchestra cron expressions have six fields starting with minutes, as in 0 6 ? * * *. The ? resembles Quartz syntax, where the first field is seconds, so a copied Quartz expression shifts everything by one field.

• Pipeline aliases accept letters, numbers and underscores. Hyphens fail.

• A SLACK alert destination will not validate without a default Slack connection, safe to comment out for the demo.

• Your dbt project has to live in a Git repo, since Orchestra clones it and cannot read a local directory. It needs requirements.txt or pyproject.toml at the root. The Git token needs Contents: Read and Metadata: Read.

• project_dir is relative to the repo root, so . when dbt_project.yml sits at the top level.

• profiles.yml gets pasted into the connection and never committed, and its profile name has to match dbt_project.yml. Skip +schema in dbt_project.yml when the profile already targets the right schema, because dbt concatenates them and yields ANALYTICS_analytics.

One process gotcha applies regardless of product: verify credentials before wiring anything. dbt debug and flowctl catalog test each take seconds and eliminate whole classes of later confusion. The exact profiles.yml I handed Orchestra was proven locally first, with dbt debug returning All checks passed and dbt build returning 12/12 PASS.

Sidebar: an MCP server is an HTTP endpoint

MCP tools load at client startup, so the freshly added Orchestra server was not callable in-session. Rather than stall, the agent spoke JSON-RPC to the streamable-HTTP endpoint directly, walking initialize, notifications/initialized, tools/list and tools/call, shelling out to curl and tracking Mcp-Session-Id by hand.

As a proper solution, it’s always better to add the MCP server before kicking off an agentic session, but as it turns out, MCP is HTTP with a session header, which is handy to know when a client has not reloaded.

Measured Postgres-to-Snowflake pipeline results

Data movement half:

Transformation half: dbt build returned PASS=12 WARN=0 ERROR=0 SKIP=0.

The mart across three successive Orchestra-triggered builds, which is the proof it transformed live streamed data rather than a snapshot:

All four DQ tests returned 0 rows. Hooray! :fire: :D

What this agentic data engineering test proves, and what it does not

The headline is not that AI writes YAML, because it writes YAML pretty badly on the first attempt: six schema errors across two tools in the first drafts alone.

The useful result is that the loop closes. The live APIs are strict enough to reject bad specs with specific, actionable errors (422: commands — Input should be a valid string, Error: No such option ‘--select’, MAC mismatch), and specific errors are something an agent can act on. validate_pipeline and flowctl catalog test both run in seconds and write nothing, which makes iteration cheap and that combination is what turned nine runs into a short afternoon instead of a week.

The narrower result is more interesting to me: streaming health can be a first-class precondition for batch work. ESTUARY_CHECK_FLOW is the concrete answer to why you would run these two products together. Gating dbt on ingestion freshness used to mean writing a Python task that queried an API and raised an exception. There are now four parameters, and it distinguishes degraded from broken, so a slow sync warns while a failing capture stops the line.

Five consecutive runs ended with the gate refusing to let dbt read the data. That behaviour is the thing being demonstrated.

Try it yourself

• dbt project used here: danthelion/agent-demo-dbt

• Full demo, including Docker Postgres and generator, Snowflake setup SQL, Estuary capture and materialization specs, dbt models, and the Orchestra pipeline YAML: https://github.com/estuary/examples/‎estuary-orchestra-agent-pipeline

• Estuary skills: /plugin marketplace add estuary/agent-skills

• Orchestra skills: /plugin marketplace add orchestra-hq/orchestra-skills

1. Skills are like functions, but they are easier to define

2. Simple things can be functions, you may want a complex thing to be a skill

3. Functions are bad at doing things badly expressed in code, like running a bash script in a python script or running web search

And then, in turn:

• Claude Code runs on your desktop. It has minimal governance, no centralisation, the UI is limited to VScode (or similar) or I guess Claude Desktop (which is a different product with different features)

• Claude uses its own model, and always will

• Codex providea a centralisation and operationalisation layer for running agents (more infra other than your computer and an LLM inference engine hosted in Anthropic’s Cloud)

So there are a few things a centralised agent management system can provide that claude code cant

1. Multi agent harness and cost reduction. If LLM costs are getting to you, this is how you solve it. Model arbitrage

2. Centralisation, governance, logging, consistency - every LLM-based workflow going through one point

3. Orchestration, Monitoring and Scheduling (sounds like Orchestra)

Where do Skills come in? Skills are the resuable components agents get. They’re the logic of how to do something. It’s becoming increasingly apparent to me that skills are going to need A) a standard and B) centralisation.

It’s also interesting that C) Skills are basically natural language represntations of flow-charts and loops.

This is not to be confused with an agent-writing framework like Langchain, which governs how an agent moves from one task to another operationally, rather than how to move between tasks from a logical perspective.

This “Loop” is not the same thing as the “loop” in a decision tree, e.g.

• Spec task

• Plan task

• Execute steps in task

• Observe results

• Task complete? If not go back to step 2

Which is what you’d see in a skill.

So a few takeaways for data teams

1. We’re going to be running agents centrally somewhere at some point for things that arent’ just generating code at home

2. Agents have loops to operate, but when we give them skills we’re basically defining flow charts which are themselves representations of functions

3. We are going to need to get good, and efficient, at writing good, efficient flowcharts. After all, a badly written flowchart is like a badly written function

4. There is some value in non-model provider AI companies as they’ll give harnesses that allow you to pick between models and therefore save on costs

5. Ephemeral infrastructure is basically critical for closing the loop as it helps with the observe results step above. Without that, your loop is screwed as your agent cannot observe or check anything.

We open-sourced a bunch of our skills at Orchestra today and a lot of them have these types of loops, if you would care to look:

Open-sourcing Orchestra Skills

Today I'm super excited to share that we're open-sourcing our Orchestra Skills repo 🔥

With AI Skills it's like having a data engineer in your pocket. In Christian Chance's words "This is a game changer" "It's making me second guess hiring a new data engineer"!

AI Skills allow you to do things on autopilot that would have taken hours if not days:

🔃 Easily retry broken pipelines from the point of failure with fixes applied in new branches, integrated slack alerting to let you know what is going on

⚒️ Build Orchestration pipelines e2e. One yml file and you are done. Alerts, monitoring, observability included

🧪 This is a fun one: the skills can also scan your entire DB and write all your data quality tests for you. Supported skill for Snowflake Databricks ClickHouse and BigQuery

And lots more. Huge thanks to Gary James for the big lift on these. There are some nice additions like the plugin to allow these to be used for not just claude but any AI e.g. Cursor

Check it out here! https://lnkd.in/edDh8uWC

#orchestra #claude #ai #skills

It’s no surprise that the largest companies in the world are building agents to create autonomous architecture. Genie Zero Ops was the worst-kept secret in Databricks Land, but it’s a really exciting development.

Genie Ops has context of everything in Unity Catalog. It is, weirdly, always running. It has access to sandboxes, and tests data and can hotfix it, even waiting for a human in the loop to interact with the results (github, Jira ticket etc.) to allow the loop to be closed.

It’s a very cool piece of kit, but wanting in many ways.

1. Credential Management is not clearly defined

2. It is always on, and therefore control is limited

3. Not obvious how the Skills are defined

4. lacks context of anything outside of Unity - tying you to Unity Catalog “Everything is fine if you have the context” “It’s easy when the data is clean reliable and fresh”

5. Lacks a proper orchestrator / event-based orchestration

Concretely, there are a few barriers to achieving the nirvana of self-healing data architecture and I don’t think we are quite there yet. I wrote these up below. The Databricks announcements corroborate this article quite nicely.

Introduction

For many data engineers, AI examples of Data Engineering revolve around one thing: fixing pipeline. An engineer opens up claude code, pastes some logs, and a pull request is made.

Semantics are fundamental here. Because when people say “Self-Healing” what they mean is “Self-managing”. The key to success in AI is not defined by manual intervention and interaction — but the absence of it.

The dream for Data Teams is a system whereby Data Pipelines and workflows generally succeed without any human intervention at all. However, there are barriers that lie in between us and this golden future.

Agents require context — fixing a pipeline may be due to a transient error, upstream schema change, or something uncontrollable entirely like a human dropping a table. Experience provides engineering teams with the know-how of how to fix these; context agents are missing.

A shift in mindset will also be apparent. The old pattern of “New branch, merge, re-run” is distinctly slow and not agenty. Unless we are to change our patterns and allow agents to merge PRs as well, this seems like a large mindset shift is required.

Finally, data does not “branch” well. Projects like Lake FS promised to make “Git for data” mainstream, but it is not. I have been writing about zero-copy cloning for years, but it is still not widely used. The distinctions between code and data are not obvious.

In this article, we’ll cover 7 barriers in between the typical Data Stacks of today and the nirvana of self-healing data pipelines / autonomous data pipelines.

Let’s dive in!

Barrier 1 | Context and failure recall

Pipelines can fail for a plethora of reasons, and being able to fix pipelines period is a requirement for an AI system. We can categorise failures into a few broad types:

• Infrastructure issues

• Code issues

• Data Issues

• Transient or Third Party issues

Generally, the manner of fixing data requires knowledge of the system. For example, Acme’s Kubernetes Cluster may only be accessible by Mr. Bob, who is the only person who has access to Bob’s special access key hidden in AWS Secrets Manager with a non-standard header. AI doesn’t know about Bob’s key, so won’t be able to fix the cluster.

Similarly, Analyst Sophie may know that the right thing to do in Widgets Incorporated is to simply gloss over the fact that sales are reported in multiple currencies, and to manipulate the numbers to be 10% higher than the ones yesterday. AI doesn’t know how to treat the numbers.

AI may also not know that to failure handle the internal API, you simply need to try it again between 2.47am and 3.12am.

These are ridiculous examples, but they illustrate the point that the knowledge to fix these different types of errors often exists within individuals’ heads. It is not enough to speak about “Metadata context”. While gathering lineage, logs, code, documentation, and other written-down context is undoubtedly imperative, AI is actually pretty good at just working it out.

As Data Folks, we’ve all been in a situation where we (Or perhaps someone we’ve spoken to) has thought:

“How on earth could I have known that?”

At the end of the day, only humans know where the bodies are buried.

Barrier 2 | Elastic infrastructure

Considering issues of the infrastructure type specifically, I am coining a term “Elastic” infrastructure. “Elastic Infrastructure” does not just scale, but also has an API to manage it.

An EC2 instance would not be elastic, as it does not scale beyond a certain point.

A Kubernetes cluster on a locked-down machine would not be elastic w.r.t cloud as there would be no API to be managed.

The reason is that AI will require access to Infrastructure in order to recover failures from it.

SaaS providers should relish this opportunity. SAAS providers necessarily take the management burden of infrastructure from Data Teams away, for a fee. This is a very AI-friendly approach, but falls down in respect of Barrier 6, which we will get to.

Barrier 3 | Operational Agents and Quality Data

Pete in Finance has overwritten the Supply and Operations Planning Google Sheet for the US again. The international forecasts are broken, and your pipeline is failing. There are 0 rows in us_forecast_dec_v1 and forecasts_agg is stale.

AI is telling you the connectors are fine but there was no data. It can’t do anything.

What is the solution here? Let’s play a quiz. I’ll give you some ideas, and you pick the right answer.

• Option 1: let AI hallucinate the forecasts

• Option 2: let AI hallucinate the forecasts in your data warehouse, and re-run the Google Sheet Pipeline Later

• Option 3: AI Tells Pete to upload the damn forecasts!

• Option 4: there is a warm pool of rented humans. When this type of pipeline fails, the AI instructs the warm pool to bother Pete in person until he fixes the pipeline himself, by hand

Of course, there is no right answer! All options are not great, ranging from bad to ludicrous. In fact, Option 4 doesn’t really require AI at all, but something called Teamwork.

Quality Data is as ever, the most important thing for a data engineer. Data Teams should ask this question when they interview more “How good is your data?”. It is such a determinant of Quality of Life, it is surprising not to get more of a mention.

That is not to say that Operational Agents have no place — for example, genuine fat finger errors could easily be corrected by an operational agent. For example, let’s say there is a new deal for $10m — perhaps the correct number is $1m. An agent with a Salesforce API Key could easily amend the data, and restart a pipeline.

Barrier 4 | Git for Data

The previous example raises an important question, which is “Should AI Agents edit production?”

If you’ve experienced multiple Saleforce environments in your career — I hear your pain. But the feature is designed to avoid the situation above. You see, perhaps the account executive has landed a whale deal and it is worth $10m. In that case, surely much better for the agent to edit the staging Salesforce instance rather than the Production one?

The above is a high-level rendering of what the process using a git-for-data like approach would work. There is a simple version below.

In both cases, AI needs a new branch to do its work. That branch needs zero copy clones of the data, it needs a git for data approach, and you need to be able to efficiently “switch in” the data at the end.

Without this structure in place, I struggle to see how AI will be trusted to reliably fix things, without creating a governance nightmare whereby it has write access to production data.

In respect of this, companies like Snowflake are well-positioned as they have supported features like zero-copy cloning for a long time. Motherduck also supports this feature. The clearest winner, though — is iceberg.

Iceberg supports time travel, rollback, and git for data. Companies like Bauplan have built compute engines around iceberg, which make for a nice, AI-friendly experience. AI should be a huge catalyst for iceberg.

Barrier 5 | Pervasion through the industry

Self-Healing Architecture hits a problem when we talk about interoperability.

Fivetran and dbt made a big fuss about open data infrastructure in 2025 — it is not the same thing as open source data infrastructure, but rather refers to an approach I think is better called the Modular Data Architecture, whereby different functions get different tools.

Another example below.

There is no point having a self-healing architecture if the underlying components do not support it. Underlying service providers most provide relevant APIs that support all the tenets of this paper, as well as self-healing functionality themselves for patterns to work.

For example, suppose there is a silent failure in an ELT Provider, whereby the sub-schema changes; the columns and types remain the same, but the values change. Perhaps now there are currencies reported in Yen, as well as in USD, but the two columns currency and local_value remain.

The right thing to do may be to amend the ELT Job in its staging environment, verify the rest of the pipeline from that staging data, switch out the data that is now correct, and then finally switch over the erroneously succeeding ELT job.

Many ELT tools simply do not provide the APIs to get this functionality. However if you were doing this with a python script you controlled yourself — no problem. This will create massive pressure on the ETL players of today to change their structures or die.

This is a massive barrier in between the modular systems of today and true self-healing autonomous architecture. The only other examples would be for the systems themselves to all become independently self-healing, as you would hope that if all parts of a system are self-healing, then so too is the whole.

Barrier 6 | Agent Sandboxes and New Orchestrators

The logical place to run Agents that fix things is within an Orchestration Tool.

This is because the Orchestration Tool has a few things the agent needs.

1. The ability to run any code, and to replay any DAG with any sets of arbitrary parameters

2. The connections to the different parts of the system the agent may need (remember, an orchestrator orchestrates, so it has access to things)

3. Alerting built-in, with monitoring, recovery, and scalable infrastructure

However there is one huge enormous problem — and that is security.

Companies like Cloudflare have built agent sandboxes. This is because models like Fable (which was recently banned) need sandboxes, as they can break out. This is especially the case when under attack from prompt injection.

Legacy Orchestration Tools are simply not made to handle agents in this way. The security risks are immense. Not to mention AI workloads could tread on the toes of data ones!

It is quite clear agents will require access to orchestration frameworks. Whether that is Open AI and Anthropic providing an orchestrator, new age orchestrators with agent sandboxes, or some form of interoperability between the two — something has to give here. Because security.

Barrier 7 | Standards for Proxy Servers and Agent Definition

One approach to security is to set-up a Proxy Sservice for Agents. Rather than install the secrets in the sandbox, the Agent has access to a given number of tools / MCPs.

The Proxy Service is then the only thing that has access to external systems. This means that even if the Agent becomes a victim of a prompt injection attack, all it could do is limited by the endpoints in the MCPs it has access to.

What this proxy service needs to look like is not obvious. MCP is big. Cloudflare released Code Mode. If you need to access multiple different endpoints, how the MCP Servers ought to be configured is not straightforward or obvious.

Open Standards should prevail — any agent looking to interact securely with multiple systems would benefit, from a security perspective, from interactive with a proxy service. These exist today, but in private SaaS tools like Foundry.

Frameworks for designing agents would also need to emerge. In the example above, a single agent requiring integration to hundreds of systems may not be feasible, as the context required to access hundreds of MCPs may be too large.

Putting all together | Single Pane of Glass for AI

Collectively, achieving the above would allow data teams to build out a single pane of glass for AI.

• Context: provides the agents with the information to solve any problem

• Elastic infrastructure: provides the foundation for fixing pipelines

• Quality Data: eradicates the human side of the data inputs

• Git for Data: creates reliability and trust in AI

• Mass Adoption: prevents industry collapse

• Agent Sandboxes and New Orchestrators: remove legacy architecture

• Proxy Servers: do their best to guaranteee security

This single pane of glass would allow AI Agents to operate in a secure way. They would execute when they needed to, and would have the context to achieve what they needed to as well.

Core data primitives like git for data, elastic infrastructure, and support throughout the ecosystem would turn this from a theoretical idea into a practical reality.

Data Teams looking to implement Autonomous architecture will impose significant pressure on existing vendors to support interoperability.

This will exacerbate consolidation, as traditional walled-gardens like Salesforce, SAP, and ServiceNow roll-out their own agentic products and data studios, capable of controlling the end-to-end without providing interoperability.

It is probably about time you tried Orchestra.

Introduction

A pet peeve of mine has always been the lack of real-time alerting and information provided by out of the box dbt logs.

The journey of a dbt practitioner typically goes like this.

• Stage 1: basic failure Yes/No

• Stage 2: after everything has compiled, what failed?

• Stage 3: after everything has compiled, what failed and why?

• Stage 4: after everything has compiled, what failed that I care about and why?

dbt live logs

This maps clearly to a functional requirement.

Stage 1: parse the exit code from the program

Stage 2: clean the manifest / run_results.json and after flattening it, send an alert

Stage 3: parse data from the flattened run_results.json and put it in an alert

Stage 4: Add different levels of severity in the dbt project, parse this via the manifest.json and put it into the alert

Tools like Elementary and dbt Cloud took years to get to Stage 4, but there are additional Stages we would like to get to at scale.

Stage 5: live logs. Send an alert at model level as soon as it fails

Stage 6: Live anomalies. Send an alert at model level when a model starts to take longer than normal

Stage 7: Live sophisticated anomalies. Send an alert at model level when a model satisfies an arbitrary condition.

In this article, we will show you how to set-up the final Stages 5–7 and therefore achieve dbt monitoring nirvana.

Step 1: set-up Live dbt Logs

The first step is to ensure you run

dbt build --log-level debug --log-format json

This allows dbt to spit out as much data that is structured as possible. While this does write a logfile, we want the live logs, so we look at stdout.

Step 2: set-up the sidecar process

The sidecar process essentially needs to run in parallel on the same machine as the dbt process. It would read from stdout and contains the relevant async + alerting logic to send live alerts. The code looks like this.

“”“Stream dbt task logs from Orchestra and send Slack alerts on test failures.”“”

import json
import os
import subprocess
import sys
import time

import httpx

BASE_URL = “https://app.getorchestra.io/api/engine/public”
ORCHESTRA_API_KEY = os.environ.get(”ORCHESTRA_API_KEY”, “”)
SLACK_WEBHOOK = os.environ.get(”SLACK_WEBHOOK”, “”)

MAX_RETRIES = 3
RETRY_DELAY_SECONDS = 3
HTTP_TIMEOUT_SECONDS = 10


def timestamp() -> str:
    return time.strftime(”%Y-%m-%d %H:%M:%S”)


def send_slack_message(payload: dict) -> None:
    if not SLACK_WEBHOOK:
        print(”SLACK_WEBHOOK not set; skipping alert”)
        return
    try:
        httpx.post(SLACK_WEBHOOK, json=payload, timeout=HTTP_TIMEOUT_SECONDS)
    except httpx.HTTPError as exc:
        print(f”Failed to send Slack message: {exc}”)


def fetch_task_run(task_run_id: str) -> dict:
    “”“Fetch task run details from Orchestra, retrying with exponential backoff.”“”
    for attempt in range(MAX_RETRIES):
        try:
            response = httpx.get(
                f”{BASE_URL}/task_runs”,
                headers={”Authorization”: f”Bearer {ORCHESTRA_API_KEY}”},
                params={”task_run_ids”: task_run_id},
                timeout=HTTP_TIMEOUT_SECONDS,
            )
            response.raise_for_status()
            return response.json()
        except httpx.HTTPError:
            if attempt == MAX_RETRIES - 1:
                raise
            time.sleep(1)
    raise RuntimeError(”unreachable”)


def wait_for_log_stream(task_run_id: str, dbt_command: str) -> subprocess.Popen:
    “”“Spawn orchestra-cli repeatedly until the log stream is available.”“”
    while True:
        process = subprocess.Popen(
            [
                “orchestra-cli”,
                “task”,
                “logs”,
                “--task-run-id”,
                task_run_id,
                “--filename”,
                f”1/{dbt_command}”,
            ],
            stdout=subprocess.PIPE,
            stderr=subprocess.STDOUT,
            text=True,
            env=os.environ.copy(),
        )

        time.sleep(1)

        # Process died immediately -> logs not ready yet.
        if process.poll() is None:
            line = process.stdout.readline()
            if line and “Download task log failed” not in line:
                print(line, end=”“)
                return process
            process.terminate()

        time.sleep(RETRY_DELAY_SECONDS)


def parse_test_identifiers(data: dict) -> tuple[str, str, str]:
    “”“Split a dbt test node name into (model, test_name, column).”“”
    model = data.get(”attached_node”, “”).split(”.”)[-1]
    base = data.get(”name”, “”).rsplit(”__”, 1)[0]
    test_name, _, remainder = base.partition(f”_{model}”)
    # A “__” prefix is meaningful: strip only one underscore to keep the rest.
    column = remainder[1:] if remainder.startswith(”__”) else remainder.lstrip(”_”)
    return model, test_name, column


def build_alert_message(data: dict) -> dict:
    “”“Build a Slack Block Kit payload styled as a test-failure card.”“”
    model, test_name, column = parse_test_identifiers(data)
    meta = data.get(”node_info”, {}).get(”meta”, {})
    owner = meta.get(”owner”, “Unknown”)
    subscribers = meta.get(”subscribers”, [])
    tags = meta.get(”tags”, [])
    failures = data.get(”num_failures”, “unknown”)
    status = data.get(”status”, “fail”).upper()
    failed_at = data.get(”node_info”, {}).get(”node_finished_at”, “”)
    alerted_at = timestamp()

    blocks = [
        {
            “type”: “header”,
            “text”: {
                “type”: “plain_text”,
                “text”: “\U0001f6a8 dbt Test Failure”,
                “emoji”: True,
            },
        },
        {
            “type”: “section”,
            “text”: {”type”: “mrkdwn”, “text”: f”*Model*\n`{model}`”},
        },
        {
            “type”: “section”,
            “fields”: [
                {”type”: “mrkdwn”, “text”: f”*Owner*\n{owner}”},
                {”type”: “mrkdwn”, “text”: f”*Channel*\n`{meta.get(’channel’, ‘’)}`”},
                {”type”: “mrkdwn”, “text”: f”*Column*\n{column}”},
                {”type”: “mrkdwn”, “text”: f”*Test name*\n{test_name}”},
                {
                    “type”: “mrkdwn”,
                    “text”: f”*Subscribers*\n{’, ‘.join(subscribers) if subscribers else ‘None’}”,
                },
                {
                    “type”: “mrkdwn”,
                    “text”: f”*Tags*\n{’, ‘.join(tags) if tags else ‘None’}”,
                },
            ],
        },
        {
            “type”: “section”,
            “text”: {
                “type”: “mrkdwn”,
                “text”: f”\U0001f50e *Result*\n```Got {failures} failures, status: {status}```”,
            },
        },
        {
            “type”: “context”,
            “elements”: [
                {
                    “type”: “mrkdwn”,
                    “text”: f”Failed at {failed_at}  \u2022  Alerted at {alerted_at}”,
                },
            ],
        },
    ]

    # Fallback text for notifications and clients that don’t render blocks.
    fallback = (
        f”dbt test failure: {test_name} on {model}.{column} ({failures} failures)”
    )

    # Blocks live inside the attachment so the red sidebar spans the whole card.
    return {
        “text”: fallback,
        “attachments”: [{”color”: “#E01E5A”, “blocks”: blocks}],
    }


def is_test_failure(log: dict) -> bool:
    info = log.get(”info”, {})
    data = log.get(”data”, {})
    channel = data.get(”node_info”, {}).get(”meta”, {}).get(”channel”, “”)
    return (
        info.get(”name”) == “LogTestResult”
        and data.get(”status”, “”).lower() in (”fail”, “warn”)
        and channel != “”
    )


def watch_logs(process: subprocess.Popen) -> None:
    “”“Stream log lines, alerting on test failures, until run stats appear.”“”
    print(”Looping:”, timestamp())

    for line in process.stdout:
        print(”Line:”, timestamp(), line)
        try:
            log = json.loads(line)
        except json.JSONDecodeError:
            continue

        data = log.get(”data”, {})

        if “stats” in data:
            print(”breaking”, data)
            return

        if is_test_failure(log):
            print(”Data:”, data)
            send_slack_message(build_alert_message(data))

    print(”Process exited without ‘stats’ in data”)


def main() -> None:
    if len(sys.argv) < 2:
        sys.exit(f”Usage: {sys.argv[0]} <task_run_id>”)

    task_run_id = sys.argv[1]
    print(”Logging for Task Run ID:”, task_run_id)
    print(”Starting:”, timestamp())

    task_run = fetch_task_run(task_run_id)
    result = task_run.get(”results”, [{}])[0]
    dbt_command = result.get(”taskParameters”, {}).get(”commands”)
    print(”dbt Task Status:”, result.get(”status”, “”))

    process = wait_for_log_stream(task_run_id, dbt_command)
    try:
        watch_logs(process)
    finally:
        process.wait()


if __name__ == “__main__”:
    main()

In our case, we use Orchestra as a way to run dbt core. This means the logs are streamed live to a bucket, which in turn is exposed via API, allowing those logs to be streamed from any external system. This means we run a pipeline with two tasks; the dbt task and the python task in parallel, which fetches those logs as the process goes and streams live logs.

dbt Live Logs output

The output is similar to a regular alert, but received as soon as something fails.

Real-time dbt alert failures

This is extremely helpful when:

• You have a big dbt project that takes a long time

• You have a lot of people that need to be alerted

• You have multiple teams using dbt whose data depends on each other

This is what running the pipeline looks like in Orchestra.

version: v1
name: ‘dbt Databricks Simple #dbt #databricks’
pipeline:
  f16f81cf-a1cf-41af-8545-6b5dd4254600:
    tasks:
      e5aaa2cd-d1a4-4f0f-b721-11a0c78bafd8:
        integration: DBT_CORE
        integration_job: DBT_CORE_EXECUTE
        parameters:
          commands: dbt build --log-level debug --log-format json
          package_manager: PIP
          python_version: ‘3.12’
          use_state_orchestration: false
          branch: live-alerts-dbt
          project_dir: dbt_projects/databricks
          shallow_clone_dirs: dbt_projects/databricks
        depends_on: []
        name: dbt Build
        connection: dbt_databricks_demo_54814
      2094af7f-66e8-4a7c-ba13-752ebeec4d40:
        integration: PYTHON
        integration_job: PYTHON_EXECUTE_SCRIPT
        parameters:
          package_manager: PIP
          python_version: ‘3.12’
          build_command: pip install httpx orchestra-cli git+https://github.com/orchestra-hq/orchestra-cli.git@main
          set_outputs: false
          source: GIT
          command: python live-dbt-alerts.py ${{ ORCHESTRA.PIPELINE_RUN_TASKS[’e5aaa2cd-d1a4-4f0f-b721-11a0c78bafd8’].TASK_RUN_ID
            }}
          project_dir: python
          shallow_clone_dirs: python
        depends_on: []
        name: Live Alerts
        connection: python__product__dbt_bigquery__79436
    depends_on: []
    name: group
webhook:
  enabled: false

dbt Live Logs Extensions

You could edit the script above to store the average task runs for the models and also flag operations in a RUNNING state. This would allow for real-time detection of anomalies / models that are running too long.

This would prevent cost overflow as well as make data teams even more proactive instead of reactive.

Below is some code that Claude wrote for me. I do not expect you to copy it, but more include it to demonstrate how straightforward it is to generate these sorts of sophisticated systems from scratch.

#!/usr/bin/env python3
“”“Detect slow / long-running Orchestra operations and send Slack anomaly alerts.

Sibling to ``live_dbt_alerts.py``. Where that script streams a single dbt task
run and alerts on *test failures*, this script looks across *all* operations
Orchestra has recorded recently and alerts on *duration anomalies*:

A) For every operation that carries timing data, it fetches the operation
   “artifacts” from Orchestra’s public API and builds a baseline by averaging
   the durations of the last ``BASELINE_ITERATIONS`` (default 7) completed
   iterations of that operation (keyed on its stable ``externalId``).

B) An arbitrary multiplier threshold (``ANOMALY_THRESHOLD``, default 2.0x)
   defines how far above baseline counts as an anomaly. Operations that
   *completed* far slower than their baseline raise an alert.

C) Operations that are still *running* are compared against the same baseline
   using their elapsed wall-clock time, so a job stuck mid-flight is flagged
   before it ever finishes.

Run it on a schedule (e.g. an Orchestra Python task every N minutes).
“”“

import os
import sys
import time
from collections import defaultdict
from datetime import datetime, timedelta, timezone

import httpx

BASE_URL = “https://app.getorchestra.io/api/engine/public”
ORCHESTRA_API_KEY = os.environ.get(”ORCHESTRA_API_KEY”, “”)
SLACK_WEBHOOK = os.environ.get(”SLACK_WEBHOOK”, “”)

# Baseline / anomaly configuration (all overridable via environment variables).
BASELINE_ITERATIONS = int(os.environ.get(”BASELINE_ITERATIONS”, “7”))
ANOMALY_THRESHOLD = float(os.environ.get(”ANOMALY_THRESHOLD”, “2.0”))
LOOKBACK_DAYS = int(os.environ.get(”LOOKBACK_DAYS”, “7”))
# Only alert on completions that finished within this many minutes, so a single
# scheduled run does not re-alert on the same historical anomaly forever.
ALERT_WINDOW_MINUTES = int(os.environ.get(”ALERT_WINDOW_MINUTES”, “60”))
# Ignore operations whose baseline is below this many seconds: a model that
# normally runs in 0.3s will trip a 2x threshold on pure scheduling jitter.
MIN_BASELINE_SECONDS = float(os.environ.get(”MIN_BASELINE_SECONDS”, “5.0”))
# Optional comma-separated operation_type filter (e.g. “MATERIALISATION,TEST”).
OPERATION_TYPES = os.environ.get(”OPERATION_TYPES”, “”)

MAX_RETRIES = 3
HTTP_TIMEOUT_SECONDS = 30
PAGE_SIZE_GUARD = 100  # safety cap on pages fetched

# Statuses that represent a genuine execution with a meaningful duration.
# REUSED / SKIPPED nodes report no timing and must not pollute the baseline.
COMPLETED_STATUSES = {”SUCCEEDED”, “WARNING”, “FAILED”}


def timestamp() -> str:
    return time.strftime(”%Y-%m-%d %H:%M:%S”)


def parse_dt(value: str) -> datetime:
    “”“Parse an Orchestra ISO 8601 timestamp into an aware datetime (UTC).”“”
    dt = datetime.fromisoformat(value)
    if dt.tzinfo is None:
        dt = dt.replace(tzinfo=timezone.utc)
    return dt


def send_slack_message(payload: dict) -> None:
    if not SLACK_WEBHOOK:
        print(”SLACK_WEBHOOK not set; skipping alert”)
        return
    try:
        httpx.post(SLACK_WEBHOOK, json=payload, timeout=HTTP_TIMEOUT_SECONDS)
    except httpx.HTTPError as exc:
        print(f”Failed to send Slack message: {exc}”)


def _get(path: str, params: dict) -> dict:
    “”“GET an Orchestra endpoint, retrying transient HTTP errors.”“”
    for attempt in range(MAX_RETRIES):
        try:
            response = httpx.get(
                f”{BASE_URL}/{path}”,
                headers={”Authorization”: f”Bearer {ORCHESTRA_API_KEY}”},
                params=params,
                timeout=HTTP_TIMEOUT_SECONDS,
            )
            response.raise_for_status()
            return response.json()
        except httpx.HTTPError:
            if attempt == MAX_RETRIES - 1:
                raise
            time.sleep(1)
    raise RuntimeError(”unreachable”)


def fetch_operations() -> list[dict]:
    “”“Fetch all operations Orchestra recorded in the lookback window.

    The public API caps a single query at 168 hours (7 days), so the lookback
    is clamped accordingly and paged through in full.
    “”“
    now = datetime.now(timezone.utc)
    lookback = min(LOOKBACK_DAYS, 7)
    time_from = (now - timedelta(days=lookback)).strftime(”%Y-%m-%dT%H:%M:%SZ”)
    time_to = now.strftime(”%Y-%m-%dT%H:%M:%SZ”)

    base_params = {”time_from”: time_from, “time_to”: time_to}
    if OPERATION_TYPES:
        base_params[”operation_type”] = OPERATION_TYPES

    operations: list[dict] = []
    page = 1
    while page <= PAGE_SIZE_GUARD:
        payload = _get(”operations”, {**base_params, “page”: page})
        page_results = payload.get(”results”, [])
        if not page_results:
            break
        operations.extend(page_results)
        if len(operations) >= payload.get(”total”, 0):
            break
        page += 1

    print(f”Fetched {len(operations)} operations from the last {lookback} day(s)”)
    return operations


def is_running(op: dict) -> bool:
    “”“An operation is in-flight if it started but has not yet completed.”“”
    if op.get(”operationStatus”) == “RUNNING”:
        return True
    return bool(op.get(”startedAt”)) and not op.get(”completedAt”)


def operation_duration(op: dict) -> float | None:
    “”“Return the operation’s recorded duration in seconds, if available.”“”
    duration = op.get(”operationDuration”)
    if duration is not None:
        return float(duration)
    started, completed = op.get(”startedAt”), op.get(”completedAt”)
    if started and completed:
        return (parse_dt(completed) - parse_dt(started)).total_seconds()
    return None


def build_baselines(operations: list[dict]) -> dict[str, dict]:
    “”“Average the last N completed iterations of each operation (by externalId).

    Returns ``{external_id: {”avg”: seconds, “count”: n, “name”: ..., “type”: ...}}``.
    “”“
    history: dict[str, list[dict]] = defaultdict(list)
    for op in operations:
        key = op.get(”externalId”)
        if not key or op.get(”operationStatus”) not in COMPLETED_STATUSES:
            continue
        if operation_duration(op) is None or not op.get(”completedAt”):
            continue
        history[key].append(op)

    baselines: dict[str, dict] = {}
    for key, runs in history.items():
        # Most recent completions first, then take the last N iterations.
        runs.sort(key=lambda o: parse_dt(o[”completedAt”]), reverse=True)
        sample = runs[:BASELINE_ITERATIONS]
        durations = [operation_duration(o) for o in sample]
        avg = sum(durations) / len(durations)
        latest = sample[0]
        baselines[key] = {
            “avg”: avg,
            “count”: len(sample),
            “name”: latest.get(”operationName”) or key,
            “type”: latest.get(”operationType”) or “”,
        }
    print(f”Built baselines for {len(baselines)} distinct operations”)
    return baselines


def detect_anomalies(
    operations: list[dict], baselines: dict[str, dict]
) -> list[dict]:
    “”“Flag completed-too-slow and running-too-long operations against baseline.”“”
    now = datetime.now(timezone.utc)
    anomalies: list[dict] = []

    for op in operations:
        key = op.get(”externalId”)
        baseline = baselines.get(key)
        if not baseline or baseline[”avg”] < MIN_BASELINE_SECONDS:
            continue

        limit = baseline[”avg”] * ANOMALY_THRESHOLD

        if is_running(op):
            started = op.get(”startedAt”)
            if not started:
                continue
            elapsed = (now - parse_dt(started)).total_seconds()
            if elapsed > limit:
                anomalies.append(
                    _anomaly(op, baseline, elapsed, “RUNNING”, “Running too long”)
                )
            continue

        # Completed operations: only alert on recent finishes to avoid
        # re-alerting on historical anomalies on every scheduled run.
        completed = op.get(”completedAt”)
        if not completed or op.get(”operationStatus”) not in COMPLETED_STATUSES:
            continue
        age = (now - parse_dt(completed)).total_seconds()
        if age > ALERT_WINDOW_MINUTES * 60:
            continue
        duration = operation_duration(op)
        if duration is not None and duration > limit:
            anomalies.append(
                _anomaly(
                    op, baseline, duration, op[”operationStatus”], “Completed slower than baseline”
                )
            )

    return anomalies


def _anomaly(
    op: dict, baseline: dict, duration: float, status: str, reason: str
) -> dict:
    return {
        “operation_name”: op.get(”operationName”) or op.get(”externalId”),
        “operation_type”: op.get(”operationType”) or baseline[”type”],
        “external_id”: op.get(”externalId”),
        “integration”: op.get(”integration”, “”),
        “pipeline_run_id”: op.get(”pipelineRunId”, “”),
        “task_run_id”: op.get(”taskRunId”, “”),
        “status”: status,
        “duration”: duration,
        “baseline”: baseline[”avg”],
        “baseline_iterations”: baseline[”count”],
        “ratio”: duration / baseline[”avg”] if baseline[”avg”] else 0.0,
        “reason”: reason,
    }


def build_alert_message(anomaly: dict) -> dict:
    “”“Build a Slack Block Kit payload styled as a duration-anomaly card.”“”
    duration = round(anomaly[”duration”], 2)
    baseline = round(anomaly[”baseline”], 2)
    ratio = round(anomaly[”ratio”], 1)
    running = anomaly[”status”] == “RUNNING”
    headline = (
        “⏳ Operation Running Too Long” if running else “\U0001f40c Slow Operation”
    )

    blocks = [
        {
            “type”: “header”,
            “text”: {”type”: “plain_text”, “text”: headline, “emoji”: True},
        },
        {
            “type”: “section”,
            “text”: {”type”: “mrkdwn”, “text”: f”*Operation*\n`{anomaly[’operation_name’]}`”},
        },
        {
            “type”: “section”,
            “fields”: [
                {”type”: “mrkdwn”, “text”: f”*Type*\n{anomaly[’operation_type’]}”},
                {”type”: “mrkdwn”, “text”: f”*Integration*\n{anomaly[’integration’]}”},
                {”type”: “mrkdwn”, “text”: f”*Status*\n{anomaly[’status’]}”},
                {”type”: “mrkdwn”, “text”: f”*Reason*\n{anomaly[’reason’]}”},
                {
                    “type”: “mrkdwn”,
                    “text”: f”*{’Elapsed’ if running else ‘Duration’}*\n{duration}s”,
                },
                {
                    “type”: “mrkdwn”,
                    “text”: f”*Baseline (avg of {anomaly[’baseline_iterations’]})*\n{baseline}s”,
                },
            ],
        },
        {
            “type”: “section”,
            “text”: {
                “type”: “mrkdwn”,
                “text”: (
                    f”\U0001f50e *Anomaly*\n```{ratio}x the baseline “
                    f”(threshold {ANOMALY_THRESHOLD}x)```”
                ),
            },
        },
        {
            “type”: “context”,
            “elements”: [
                {
                    “type”: “mrkdwn”,
                    “text”: (
                        f”Pipeline run `{anomaly[’pipeline_run_id’]}`  •  “
                        f”Task run `{anomaly[’task_run_id’]}`  •  “
                        f”Detected at {timestamp()}”
                    ),
                },
            ],
        },
    ]

    fallback = (
        f”Operation anomaly: {anomaly[’operation_name’]} took {duration}s “
        f”vs baseline {baseline}s ({ratio}x) — {anomaly[’reason’]}”
    )

    # Amber for in-flight warnings, red for confirmed slow completions.
    color = “#ECB22E” if running else “#E01E5A”
    return {”text”: fallback, “attachments”: [{”color”: color, “blocks”: blocks}]}


def main() -> None:
    if not ORCHESTRA_API_KEY:
        sys.exit(”ORCHESTRA_API_KEY not set”)

    print(”Starting operation anomaly scan:”, timestamp())
    print(
        f”Config: baseline={BASELINE_ITERATIONS} iters, “
        f”threshold={ANOMALY_THRESHOLD}x, lookback={LOOKBACK_DAYS}d”
    )

    operations = fetch_operations()
    if not operations:
        print(”No operations returned; nothing to analyse.”)
        return

    baselines = build_baselines(operations)
    anomalies = detect_anomalies(operations, baselines)

    if not anomalies:
        print(”No anomalies detected.”)
        return

    print(f”Detected {len(anomalies)} anomalies:”)
    for anomaly in anomalies:
        print(
            f”  {anomaly[’status’]:>9}  {anomaly[’operation_name’]}  “
            f”{round(anomaly[’duration’], 2)}s vs {round(anomaly[’baseline’], 2)}s “
            f”({round(anomaly[’ratio’], 1)}x)  - {anomaly[’reason’]}”
        )
        send_slack_message(build_alert_message(anomaly))


if __name__ == “__main__”:
    main()

Claude Skill commentary

# Python workers

`python/` contains Python modules used by this repo’s demos.

Most subfolders are intended to be executed by Orchestra via Python task
integrations (i.e., Orchestra launches the Python code as part of a pipeline
run).

Platform interaction patterns live under `patterns/`.

## Contents

- Worker modules for integrations and data movement.
- Subfolders with targeted examples (for example `integration_a/`,
  `integration_b/`, and `claude_agent/`).
- Shared dependencies in `requirements.txt`.

## Live dbt Alerts

`live_dbt_alerts.py` contains a short Python script, wrapping the [Orchestra CLI](https://docs.getorchestra.io/docs/git-control-and-ci-cd/orchestra-cli), that utilises an [Orchestra Python Task](https://docs.getorchestra.io/docs/integrations/python) and dbt’s logging to send real time Slack alerts for dbt runs.

### Usage

In order to run live dbt alerting for a dbt task:

* Create Slack incoming webhook(s) - Full [docs](https://docs.slack.dev/messaging/sending-messages-using-incoming-webhooks/#getting_started)
  - Create per-channel webhook to send alerts to (just 1 in this example, but can be extended)
  - Navigate (in Slack dashboard) to *Your Slack App* > *Incoming Webhooks* > Copy the Webhook URL for later

* Create Orchestra Python connection:
  - Connect to repo containing `live_dbt_alerts.py`
  - Store `SLACK_WEBHOOK` in secrets containing the Webhook URL retrieved above

* Configure pipeline tasks
  - In your dbt pipeline of choice, add `--log-level debug --log-format json` to the dbt command you are running
  - Create a Python task to run in parallel:
    - Use the connection set up above
    - Set Python command to `python live-dbt-alerts.py ${{ ORCHESTRA.PIPELINE_RUN_TASKS[’<dbt task ID>’].TASK_RUN_ID }}` or wherever your alerting script is located
    - Add build command of `pip install httpx orchestra-cli git+https://github.com/orchestra-hq/orchestra-cli.git@main` or latest `orchestra-cli`

* Add alert config to dbt tests, field structure is as follows:
```yaml
- <test_name>:
    config:
      meta:
        tags:
          - “Super Urgent Data Tests”
        owner: “gary@getorchestra.io”
        channel: “#dbt-live-alerts”
        subscribers:
          - “spyro@getorchestra.io”
```

That’s it! On the next run of your dbt pipeline, Orchestra will spin up a monitoring Python task to parse your dbt failures and alert you in Slack!

## Operation Anomaly Alerts

`operation_anomaly_alerts.py` is a sibling of `live_dbt_alerts.py`. Instead of
streaming a single task run and alerting on test *failures*, it scans every
operation Orchestra has recorded recently (via the public `operations` API) and
alerts on *duration anomalies*:

- **A — Baselines from artifacts:** for each operation (keyed on its stable
  `externalId`), it averages the durations of the last `BASELINE_ITERATIONS`
  (default 7) completed iterations. `REUSED`/`SKIPPED` nodes carry no timing and
  are excluded.
- **B — Threshold:** `ANOMALY_THRESHOLD` (default `2.0`x) sets how far above
  baseline counts as an anomaly. Recently *completed* operations that exceed it
  raise a red Slack alert.
- **C — Long-running operations:** operations still in flight (a `startedAt`
  with no `completedAt`) are compared against the same baseline using elapsed
  wall-clock time, raising an amber “running too long” alert before they finish.

### Usage

Run it on a schedule — e.g. an Orchestra Python task firing every few minutes:

* Reuse the same Orchestra Python connection and `SLACK_WEBHOOK` secret as the
  live dbt alerts setup above.
* Set the Python command to `python operation_anomaly_alerts.py`.
* Add a build command of `pip install httpx`.

Tunable environment variables (all optional):

| Variable | Default | Purpose |
| --- | --- | --- |
| `ANOMALY_THRESHOLD` | `2.0` | Multiple of baseline that triggers an alert |
| `BASELINE_ITERATIONS` | `7` | Completed iterations averaged into the baseline |
| `LOOKBACK_DAYS` | `7` | History window to scan (API caps at 7 days) |
| `ALERT_WINDOW_MINUTES` | `60` | Only alert on completions newer than this |
| `MIN_BASELINE_SECONDS` | `5.0` | Skip fast operations where jitter dominates |
| `OPERATION_TYPES` | _(all)_ | Comma-separated filter, e.g. `MATERIALISATION,TEST` |

Summary | Don’t settle for less than live logs

I was so happy to see our Head of AI create a video demonstrating how live dbt logs could be created. Needless to say, Tasks running in Orchestra are poll-able and run-able and therefore support this pattern — however dbt logs have always been a thorn in the side.

With this simple sidecar process, we demonstrate how data teams can easily generate better, faster, more helpful logs without breaking the architecture bank by purchasing another tool.

Key to this is metadata. By having the metadata of historical runs for specific assets, it is straightforward to make alerts more technical and targeted, avoiding alert fatigue. In the case of Orchestra, Operational Data for Data Assets is gathered and exposed via API anyway, proving that the combination of Orchestrator + Observability + dbt makes for a meaningful improvement in alert speed, quality, and implementation.

Check-out the Claude Skill above as well for writing a sidecar process!

Follow me on Linkedin if this interests you. If you’re keen to try it out, everyone gets free compute in Orchestra on the Free Tier.

Introduction

It’s been a lot of fun adding state to dbt. Something I didn’t think was possible, our engineering team managed to find a way. It’s now used by some the most mature data teams I’ve worked with like Octopus Energy and Experian.

State of course is nothing new - data teams have been tracking state in various places for donkeys years, and more recently projects like SQLMesh added state too.

dbt Fusion added state - but state was proprietary and not public. The license, which wasn’t Apache, then subsequently became Apache (forever? Maybe). In between which, we released Sao Paolo into the wild, albeit not for free.

Over time I suspect the projects will converge. For example there are a bunch of features we have that dbt’s does not:

• Asset-Aware orchestration

• Some nice optimisations around how state is fetched

• Some nice optimisations around how models are selected

• Support for some DBs not in dbt Cloud e.g. Motherduck

There are some things dbt’s has we do not

• Support for some other databases

• The rust compiler / the rest of the fusion engine

We are laser focussed on helping data teams so hopefully this is a way to keep fusion honest as time goes by. They may of course, diverge, if we revert to the scenario we were in 6 months ago where core vs. Fusion was not an obvious decision due to the proprietary nature of it.

Furthermore, dbt state is still charged if you use it in Fivetran. Nothing is free.

Indeed, if you want to use State via Fivetran:

• Hugos’s Friend 8:33 PM

  Have you looked at dbt State pricing? In classic Fivetran fashion, it is Daily Active Target Tables (9.4 cents per). It was free 2 weeks ago for dbt Enterprise customers but now it is charged separately.

  https://getdbt.slack.com/archives/C0B7E4W8E7M/p1781188337125049

So if you have 1,0000 models then storing state is $90 a pop, per day. I’m not sure how much you know about state, but basically every time you run dbt you store state. And the vibe is sort of like, because you store state you can run it however frequently you want, say every 5 minutes, which would create 288*1000*0.09 instances of state…so thank god it’s active daily..?

In Orchestra, running dbt is $0.033/min to reflect the cluster and monitoring infrastructure we use to run it. No extra charge for state. Just enable it

In any case It feels great to be giving back to the community so please enjoy and let us know what you think.

pip install dbt-orchestra

We’re Open-Sourcing State-Aware Orchestration (Sao Paolo) for dbt core

Sao Paolo by Orchestra is going Open-Source.

State-Aware Orchestration on dbt core is available under Apache 2.0

https://github.com/orchestra-hq/sao-paolo

There are a few reasons we like this approach:

✅ Easier Scheduling: Orchestra SAO (State Aware Orchestration) means you don’t need to manually tag models, you just need to say when the models should be updated and Orchestra SAO handles the dependencies.

✅Save cost: Orchestra SAO detects when there is new data and only updates models and their downstream deps if there is new data, saving money and reducing time.

✅Works out of the box: no need to upgrade dbt versions to take advantage of Orchestra SAO

Being part of the dbt community was one of the things that originally brought me to data engineering back when I was working at JUUL, so it feels pretty awesome to finally contribute something back!

For those of you wondering how this compares to Fusion — we launched SAO in our proprietary solution a couple months back when the dbt Fusion license was still Elastic 2.0 and when the state APIs were proprietary.

The two projects are not currently identical, there are a couple of differences such as a nice optimisation around build_after configurations propagating up the entire DAG in Orchestra SAO for example.

Orchestra SAO will allow these two standards to converge over time to a better dbt experience with state-aware orchestration, asset-aware scheduling, and a rocket of a compiler — all available to the dbt community under Apache 2.0.

There is no requirements to use this in Orchestra. It works with your dbt repo, just requires you to configure where state is stored.

Getting Started with Orchestra state-aware orchestration

• The complete guide to SAO (read)

• Configuring build_after configurations (read)

• Get your project AI Ready with Orchestra Skills (github)

• Clone the repo

Introduction

Orchestra is at Summit and it’s incredible to see what’s been announced. The overarching theme relates to ease-of-use. Rather than re-create the entire data stack Snowflake are finally architecting something that is potentially capable of making developers’ lives easier.

However there is a dangerous trap — integration. Building a closed-end system allows you to do a lot of awesome stuff. However, it ropes you in to using a closed-end system. The new trend “tokenmaxxing” has a Snowflake version too; “Snowmaxxing”.

That said the “elastic data warehouse” is alive and well. I said in 2024, when Shridhar joined Snowflake:

The trade-off, of course, is ease of use. Using Snowflake has and will continue to be easy. It is their krpytonite to Databricks’ nuclear arsenal of capability.

Who is Sridhar Ramaswamy and what does this mean for Snowflake?

Frank Slootman: Thank you, and goodnight

medium.com

It is honestly pretty incredible to see the pace of integration it’s taken to action this. AI and Context in Snowflake is extraordinarily powerful. The feedback from the floor is intense — the AI is powerful, and easy to use, as billed.

Let’s dive in to what was announced.

Snowflake Datastream

Somehow without making an acquisition, they finally have their streaming solution, just like I predicted.

Snowflake also introduced Snowflake Datastream6, a new fully managed streaming service for Apache Kafka®7 that brings real-time data and AI together in a single, governed platform, giving organizations an easier way to power AI apps and agents with fresh, continuously flowing data.

Predicated in March 2025

Snowflake Cowork

See here.

Snowflake CoWork is Snowflake’s AI-powered work assistant designed to help employees interact with enterprise data and applications using natural language. Rather than functioning as a simple chatbot, CoWork acts as an agent that can understand business context, retrieve information from multiple systems, perform analysis, and take action on a user’s behalf.

One of its core features is Cortex Sense, a semantic layer that provides AI agents with a shared understanding of business concepts, metrics, and organizational knowledge. This allows CoWork to deliver responses that are grounded in company-specific definitions rather than generic AI reasoning.

CoWork also includes Deep Research capabilities, enabling it to conduct multi-step investigations across both structured data in Snowflake and unstructured content such as documents, emails, and knowledge bases. Users can ask complex business questions and receive synthesized insights without manually querying multiple systems.

To make interactions more personalized, CoWork introduces User Memory, which remembers user preferences, recurring tasks, and working patterns. Over time, the assistant becomes more tailored to an individual’s role and workflows.

Another key feature is User Skills, which allows users and organizations to package frequently repeated processes into reusable workflows. These skills can be invoked through natural language, reducing the need for manual execution of routine business tasks.

CoWork is also designed to connect with a broad ecosystem of enterprise tools, including productivity, CRM, collaboration, and file-management platforms. Through these integrations, it can not only retrieve information but also perform actions such as updating records, creating tasks, sharing documents, or triggering workflows.

Finally, CoWork includes Artifacts, a feature that enables users to create and share outputs generated through AI interactions. These outputs can include reports, analyses, summaries, and collaborative work products that can be distributed across teams and embedded into existing business processes.

Together, these features position Snowflake CoWork as a unified AI workspace that combines enterprise search, business intelligence, workflow automation, and action-oriented AI within the Snowflake platform.

Snowflake Managed Agents

We knew this was coming.

Snowflake Managed Agents is a framework that allows organizations to deploy and operate AI agents directly within the Snowflake platform without having to manage the underlying infrastructure, orchestration, or lifecycle of those agents. The service is designed to help enterprises move beyond conversational AI and build agents that can reason over data, execute tasks, and automate business processes while remaining governed by Snowflake’s security and data controls.

A key feature of Managed Agents is its ability to access and reason across enterprise data stored in Snowflake. Agents can work with structured warehouse data, unstructured documents, and business metadata, enabling them to answer complex questions and perform multi-step analyses using a unified data foundation.

Managed Agents also provide built-in orchestration and planning capabilities. Rather than responding to a single prompt, agents can break a request into multiple steps, determine which tools or datasets are required, execute those steps in sequence, and return a consolidated result. This allows them to handle more sophisticated workflows such as investigations, reporting, forecasting, and operational decision support.

Another important capability is tool and application integration. Managed Agents can connect to enterprise systems and external services, allowing them to retrieve information, trigger actions, update records, and coordinate work across business applications. This enables agents to move from generating insights to executing real business tasks.

The platform includes governance, security, and observability features that align with Snowflake’s enterprise architecture. Organizations can control which data sources an agent can access, apply existing permissions and policies, monitor agent activity, and maintain auditability for compliance requirements.

Managed Agents are also designed to work with Snowflake’s broader AI ecosystem, including Snowflake Cortex, semantic models, and business context layers. This helps ensure that agents understand company-specific terminology, metrics, and business rules, reducing the risk of inaccurate or inconsistent outputs.

To accelerate adoption, Snowflake provides a fully managed operational environment. Organizations do not need to provision infrastructure, manage scaling, monitor agent runtimes, or handle updates to underlying AI services. Snowflake manages these operational concerns, allowing teams to focus on designing agent behavior and business outcomes rather than platform engineering.

It is worth noting that for this to work, you need to be happy with agents running in your data warehouse or data platform. Oftentimes, folks may want this to be elsewhere for connectivity reasons or simply for architectural reasons.

Snowflake Horizon Context

Snowflake Horizon Context: The Governed Context Layer for AI, BI and Apps

Discover Horizon Context, a new capability within Horizon Catalog that delivers a connected, governed semantic…

www.snowflake.com

New connectors. This is what we get wth Horizon and Select *

Snowflake Migrations

Big push from CK on using COCO to migrate things like Teradata. Worth a note.

Snowflake SpaceX Models

Pretty interesting to see Grok getting made available in Snowflake. They are operating at an interesting part of the stack where they can provide models and distirbute them. This is very unique, bar the hyperscalers of course.

Agentic search (Neeva) — precise analytical results from unstructured data

Not hugely shouted about, but if you see here it is essentially a lot of the heavy-lifting going on behind the scenes when you ask CoCo to do something.

It’s another incentive to move towards putting all your data into Snowflake as well as a very nice dovetail into things in iceberg which may naturally be more unstructured.

Without this, the assistant would be incapable. So very foundational and interesting work.

Code Bundles — just execute code in Snowflake. Or Schedule it.

As discussed previously the ability to run any code in Snowflake, with some slightly better devex (see more). Again — do we really think Snowflake is going to rip out the hyperscalers when they are committed to spending $6bn with AWS over the next 5 years — probably not.

But it’s a nice feature for sure.

c.f. Daytona

Visual Pipeline Editor — JB and noone else

Snowflake have announced a Visual Pipeline Editor. “Who’s excited for a Visual Pipeline Editor in Snowflake?!” was greeted by crickets bar someone referred to as “JB”.

As we know, the orchestration of end-to-end pipelines should be simple but at some point, most large companies look to add some form of orchestration. Now Snowflake have two orchestrators — Snowflake Tasks and the Visual pipeline editor, which is presumably a wrapper around Tasks.

To understand how modern platforms like Orchestra outperform frameworks like Airflow and Snowflake Tasks, read more here.

Simplify the Entire Development Experience — Snowmaxxing…

CoCo for excel, Claude Code and Microsot VSCode

Here you go.

CoCo in Excel, Claude Code and VSCode.

Pretty awesome, love this as a release. Bringing Snowflake to the people. LFG.

CoCo Desktop

like Claude Code, but for CoCo. See here.

COCO Desktop announced at Snowflake Summit

It’s very interesting here to see Snowflake follow in the AI Labs’ stead by building a desktop App. Given the number of Business Users who want to interact with data I think this is a smart move.

Why would you use Claude Code Desktop when you could simply use Snowflake’s? Unclear. It feels like a 50/50 — but at this point, if you never try, you never know!

Fundamentally the game is to position Snowflake as an AI Infra company and it is paying off.

Conclusion: Sridhar Ramaswamy’s investments are coming off

When Sridhar joined Snowflake I wrote this in March 2024:

With the share price dwindling vs. the post IPO days, Ramaswamy has a bit of wiggle room to build. The Data and AI Arms Race is just beginning. There is no doubt, no question in my mind that Ramaswamy’s biggest priority is to build a nuclear arsenal of AI functionality. The question for us a s data practitioners will be whether the cost of using it exceeds the benefit. 🚀 (Link)

The stock is up. The AI Features are here. And unstructured search is here too — the vision of wedging what Sridhar did at Neeva in between the users and the data is the at the crux of the Snowflake AI Platform.

In addition, the company looks like it will successfully imitate Anthropic with a suite of AI products leveraging CoCo, thereby displacing Anthropic from the Data Engineering Space, using them only for LLMs in a commoditised way.

Whether or not this extends to the Snowflake Platform itself will determine the irony. After all, with companies like Clickhouse committing to “Headless” and “Serverless” architecture, Snowflake is decidedly doing the opposite — building a suite of features around Snowflake at the expense of their partner system.

It would be ironic if those partners disintermediated Snowflake somehow by commoditising compute. For all the furor around duckdb, iceberg, and some start-ups like Bauplan and Greybeam, commoditised, multi-engine compute is yet to blow into the mainstream.

The company is growing revenue of c.$5bn at c.35% y-o-y. Sridhar Ramswamy’s investments are paying off. Companies are snowmaxxing harder than ever. CoCo appears to be the closest the company has come to executing on its AI strategy.

It’s time for a new journey for Snowflake as something more than just a data warehousing solution.

Want to see how teams are SnowMaxxing wiht Orchestra and Snowflake harder than ever? check-out the video below on how Experian are building a stack that is easier and more powerful than ever before.

Full Snowflake Summit Announcements video

Anthropic and Dario Amodei are arguably the biggest thing to happen to the corporate world in 2026. After the Citrini report heralded the SAASpocalypse, companies like Atlassian and Block cut employees, citing AI productivity gains. blue-chip SAAS stocks are down 30% this year.

Salesforce share price from Google

But it is not all rosy. Altman and Amodei have both recently discovered the joys of being backed by venture. Rather than stick to the bread and butter of LLM model development, a fool’s errand due to the Great Token Conspiracy, they have branched out to seek new, greener pastures.

Source: MUFG Americas. Note Open AI’s Compute Forecast by 2030 is $600bn, the total of hyperscales combined.

One such pasture is the world of design. A couple of weeks ago, Anthropic released Claude Design, an interface where an AI helps you generate designs. These designs are code-based, and therefore port automatically to other code-based paradigms such as websites. Figma stock is down c.80% this year.

Figma Share Price from Google.

Open AI meanwhile seem more focussed on the consumer market. Open AI recently released GPT5.5 — a more capable model for human, individual tasks and automations.

The warning signs are already here. Anysphere, the company behind Cursor, recently sold the right to be acquired for $60bn for an undisclosed sum. In any event, should Elon Musk not exercise that right, they will be paid $10bn. At a X $bn valuation previously, the Founders would have realised that lacking their own proprietary model (though they have their “Own” based on Kimi), they lacked the “edge” required to succeed in the Great Token War.

Token costs over time. Source

With even companies like Cursor taking evasive action the only people left are Open AI and Anthropic, and these companies will do absolutely anything to sustain their valuations. The question is — who will they target first?

For all their ambitions to defeat the CRMs, ERPs, and Design Software tools, there is one industry they cannot touch. This area of engineering, often described as a “subset” of software engineering actually contains an enormous amount of context AI struggles to understand.

In this article we will dive into the particulars of this industry and why Anthropic will not be able to break into it without making an acquisition.

Historic(al?) Vendor Relationships

The Data industry is governed by historical relationships. These vary by vertical and company type, but can roughly be split as follows:

• Big Tech: do not buy that much data engineering software, and leverage AWS or GCP

• “Laggards” and “Late Adopters”: Microsoft

• Government in the US: palantir

• GCP

• Everyone else: Databricks and/or Snowflake

What this means is the decision-makers for buying Data Engineering software already have their favourite steakhouses and their favourite reps. There is simply a level of loyalty within the C-Suite and the IT Exec that no amount of tokens can break through.

Sunk Cost Fallacy and existing architecture

I routinely speak to enterprises that are on their 4th iteration of their data platform or later. In large enterprises, bodies are buried, and lots of them. Architecture is complicated — there are many tools already procured.

This means if Anthropic were to break into the data engineering space, they woudl have two options

1) Option 1: wipe the floor clean and start-over. This is not feasible as it would require Anthropic to develop a Palantir-Foundry equivalent data solution, which we do not think is likely

2) Option 2: build something interoperable. What could this be? Perhaps an Observability tool? The data stack is already crowded, which means there is not much space for a tool like Anthropic’s

Option 1 is not likely because of sunk cost fallacy. At every enterprise there are decision-makers that have made decisions to implement data architectures that are still in progress. They are simply not aligned to any value proposition put forward by Anthropic. The wheels are in motion already, don’t stop the train!

Size of opportunity

Anthropic is a big company. It already has 47$bn of revenue. That is 8–10x times as large as Databricks.

To meaningfully impact their revenue, the company would need to go for at least $1bn revenue opportunities. There simply are not that many companies like that in the data space. Apart from Agentic BI, where companies like Tableau and Looker hit revenue valuations of multiple hundreds of millions, the only companies at the scale of interest are Snowflake and Databricks.

One interesting area for Anthropic to look at breaking into would actually be agentic BI. Companies like Lightdash, TextQL, Omni, and Hex are all building agentic BI layers.

Snowflake, GCP and Databricks are all providing these natively. Microsoft’s is distinctly lacking. This strikes me as an area Anthropic could break into, but let’s call that “analytics” and not “data engineering” (for now).

Only Human People know where the bodies are buried

Like it or not, data engineering is a multi-disciplinary domain that is generally incredibly poorly-documented. Tribal knowledge of architecture, building pipelines, and fixing data quality errors is rife. Often there is a “way” of doing things that simply exists in peoples minds.

I wrote about this at length in the complete guide to implementing AI for CDAOs. There we identified that enterprises with a large amount of tech or knowledge debt would have a greater cost of implementing AI. It would also take them a longer time.

A worked example of increasing productivity with AI. How long would this take?

For the same reason, if Anthropic produced a data engineering product they would struggle for it to become meaningful without context. For example, Snowflake recently released agents but they struggle to debug anything outside of Snowflake.

At Orchestra, we orchestrate everything and gather all the metadata, including lineage. This means Orchestra agents can fix things like broken pipelines, no matter where they fail.

This is all it takes — fixing pipelines using agents and orchestra

version: v1
name: Fix everything
pipeline:
  e1470cdc-cefe-43a2-a5c5-f449a0c361cd:
    tasks:
      fdc77498-4023-45a4-a4c0-bda95e88358f:
        integration: ORCHESTRA
        integration_job: RUN_AGENT
        parameters:
          agent_id: Fix Pipeline
          prompt: ‘Using the Skill and the relevant Pipeline data below, fix the pipeline.


            ${{ inputs.pipeline }}’
        depends_on: []
        name: Fix everything
    depends_on: []
    name: group
trigger_events:
- type: pipeline
  pipeline_id: ‘*’
  statuses:
  - FAILED
  run_inputs:
    pipeline: ${{ outputs.pipeline_run }}
webhook:
  enabled: false
inputs:
  pipeline:
    type: string
    default: ‘[]’

For an Anthropic Data Engineering product to takeoff, it would need access to all the human documentation that individuals have about the business. While documenting this in totality is not that far fetched (AI writes great docs) whether or not the privilege of implementing Anthropic would provide a great enough incentive remains to be seen.

What Claude is good at: writing code

Undeniably, as a data engineer, Claude, Copilot and Codex are all incredibly good at writing code. Whether that is a dbt model, an orchestra pipeline, or python code to move data from an API, agents know what they are doing.

While it means we are all customers of Anthropic, I would not say it really constitutes a move into the data engineering market. After all, I might use Claude to generate my code, Codex to write my documentation and another model for a chatbot to serve analytics requests — does this mean AI is taking over Data Enginering?

Far from it. In Data Engineering, the effect of agents and LLMs is to speed-up AI-augmented workflows, rather than replace them. While there is some talk of agentic analytics completely replacing enterprise data architectures, the need for faster, more efficient, and more reliable answers inevitably will prevail.

Moreover, in a world where the human cost of writing code is close to nil, wherein do Anthropic win bar tokens? After all, data engineers have their patterns already — be it running code in managed services like Orchestra, deploying custom on a Kubernetes cluster or perhaps even another provider like Modal — Anthropic does not drive its own compute. it does not own hardware — despite its size, it is not a neocloud just yet.

The Ghost in the Shell | How Nebius is taking on Databricks, Snowflake, and Everyone else

The Rise of AI Clouds hinges on talent, ambition, streaming, and a few strategically placed bets

medium.com

Analytics Engineers are not safe

I believe there are two breeds of Analytics Engineer. There are those that excel at answering questions for the business and those that simply want to tweak dbt models, happy to have finally found a well-paying semi-technical job.

Create a simple hand drawn image of two types of analytics engineer to illustrate the below: I believe there are two breeds of Analytics Engineer. There are those that excel at answering questions for the business and those that simply want to tweak dbt models, happy to have finally found a well-paying semi-technical job.

AI makes managing a dbt project very easy. Agents can be integrated with CI/CD that automatically catch errors. Documentation becomes automatic. Writing incremental models a given. State Aware Orchestration exists in both dbt Cloud and Orchestra, and who knows where else, which makes scheduling even simpler.

The cost of having a well-organised dbt project has already become extremely low. The price however, of speaking to Stakeholders, understanding requirements, reading between the lines, and knowing how to translate these into SQL is higher than ever.

In fact, that knowledge must be documented for AI insight generators to have any meaning at all.

Analytics engineers capable of driving real change should be safe from AI. They will act as a translator between the business and the AI. They will provide an important part of the feedback loop for anyone building agentic analytics. Technical AEs — not so much.

Conclusion: Anthropic should look to agentic BI

We have seen there are numerous compelling reasons for Anthropic to avoid trying to break into the data engineering market. Aside from go-to-market considerations and the huge level of inertia and tech debt inherent in IT organisations, data engineering itself is not a practice that lends itsef well to AI. This is because reliable data engineering depends on a huge amount of context that may not be systematically documented.

Despite this, AI is disrupting data engineering by reducing the cost of writing and maintaining good code. In tandem, developments like the “codifying” of all pipelines and state-aware orchestration make it easier than ever to build effective systems quickly.

AI has the ability to severely disrupt analytics engineers. Those that have specialised in a niche area of writing code (dbt) are no longer required, as AI probably knows more dbt than we do. It has completed its dbt certification.

If Anthropic are to break into the data space, a serious cost benefit analysis should be done. However for the sake of this article, we can give them one for free.

Figma is a $1.4bn revenue business. While Claude Design is pretty, it lacks the maturity of Figma, that essentially acts as a content-management system for marketing assets.

Furthermore, the editability of Figma makes it an attractive option for designers seeking for patterns where AI may not “one-shot” designs consistently. Therefore, a company like Figma has strong barriers to entry, as well as a technological moat in the form of the content CMS Anthropic appears to have overlooked.

By contrast, the Analytics Space is a $34bn business. The fundamental product is a multi-tenanted system with a number of integrations to data warehouses and other systems with query engines. The core functionality includes the ability to generate charts, reports, and dashboards.

However, the secret sauce for many taking off are AI insights assistants. These are on-demand agents that understand your data and can easily and consistently answer questions about your data.

Furthermore, Analytics Products occupy an interesting space in the IT budget. Where tools like Figma may fall within their relevant domain’s budet (Marketing), BI tools are typically reserved for the entire organisation (after all, who needs data — everyone).

For this reason, companies like Tableau, Looker, Omni, Sigma and Hex have all managed to grow revenues comparatively easily, by selling not just to IT, but the business.

Anthropic sells to the C-Suite, and is therefore ideally poised to have this conversation. “You don’t need to spend $400k on Tableau, why not take some of your $1m Anthropic budget for the year and put it towards our AI Insights agent?”. This is a conversation you can see happening.

Claude is not replacing Data Engineers anytime soon. There is definitely an element of guarding the guardians. Data infrastructure must exist. Infrastructure must exist. Even with the most powerful model cannot replace a cluster. That is why Anthropic cannot touch Data Engineering.

Hugo Lu is the CEO of Orchestra, the AI and Data Infrastructure Company.

Introduction

Something shifted in the data stack this year, and the signals are hard to ignore.

Hex hit a milestone on March 10: agents now create more cells on the platform than humans do. Agent messages per user doubled from about 5.5 to 12 per week in just three months, and agent weekly active usage is compounding at over 10% week over week. Barry McCardel, Hex’s CEO, called it “the fastest usage takeoff for any data tool, ever.”

On the same timeline, MotherDuck published a piece titled “Don’t Fear the Agents” that opens with 1,000 agents running queries on a data lakehouse, this is a future that’s not too far and is a workload pattern they’re preparing for. Tools like Fabi AI and Zenlytic are putting natural language interfaces directly on top of warehouses, turning your average business user, who doesn’t even know what SQL stands for, into someone who can wrangle data confidently.

This is the trend: AI agents are becoming the primary consumers of your data infrastructure. Not your data analysts. Not your analytics engineers. Not your dbt jobs. Agents, running discovery queries, writing SQL, validating results, iterating, and drilling down.

For data leaders, this changes the calculus on your data stack in ways that optimization alone can’t fix.

The Cost Trajectory No Leader Wants to Defend

The math on agentic workloads is brutal for per-credit billing models.

A single user asking “why did revenue drop in APAC last week?” doesn’t trigger one query. The agent behind that question fires 6 to 12 queries: metadata discovery, schema inspection, candidate SQL generation, execution, validation, follow-ups, drill-downs. All before the user sees an answer.

Data warehouses like Snowflake has billing mechanics that make this particularly painful. There’s a 60-second minimum billing window. If the warehouse auto-suspends between bursts, each burst of short queries triggers a new billing window. You pay for 60 seconds of compute to run 3 seconds of work, repeatedly.

Scale that to 50 users asking questions throughout the day, and your warehouse bill grows disproportionately to the value generated. Not because the queries are hard, but because there are so many of them. These are small, read-heavy aggregations and lookups. The kind of queries that don’t need distributed compute.

Now imagine defending that line item to your CFO two quarters from now, after agentic adoption has compounded across product, ops, and executive teams.

Why “Optimize Harder” and “Migrate Off” Both Fail

Picking on Snowflake here, but this pattern tends to happen when a data leader sees rising costs.

Instinct one: optimize Snowflake.

Right-size warehouses, update auto-suspend timers, consolidate workloads, reduce refresh cadence. All good moves. Every one of them has a ceiling. The floor is your cost per credit, and no amount of optimization can change that.

Instinct two: migrate to another warehouse or query engine.

DuckDB is fast, cheap, and handles analytical queries beautifully. But DuckDB is a query engine, not a platform. No RBAC. No governance UI. No data sharing. No marketplace integrations, no Snowpark, no ecosystem your team has built workflows around. To use DuckDB as a primary engine, you’d need to build all of that infrastructure yourself. On the flipside, migrating to another warehouse is a multi-quarter migration, and there’s no guarantee it’ll be cheaper.

Neither path gives a data leader what they truly want: meaningful cost savings at scale.

Multi-Engine Is the AI Architectural Bet Leaders Are Making Now

Multi-engine is not “replace Snowflake with DuckDB.” It’s the ability to use the right tool for the job, or in this case, use the best engine for each query. Data teams should keep the great developer experience they’re familiar with on Snowflake while leveraging the capabilities of DuckDB, Clickhouse, StarRocks, or any other engine.

Three things made this practical instead of theoretical:

Open table formats crossed the adoption threshold. Apache Iceberg is no longer just conference material. Fivetran has devoted significant resources to moving customers to Iceberg-backed data lakes. Snowflake managed Iceberg tables let you store data in open formats while using Snowflake’s catalog. When your tables are Iceberg, any engine can read them.

The workload data supports it. In Fivetran’s analysis of Snowflake and Redshift usage, the 99.9th-percentile query scanned about 300 GB. In MotherDuck’s analysis of the Redset benchmark, roughly 99% of queries scanned under 100 GB. Agent-generated queries are exactly this profile: small, read-heavy, analytical. The vast majority never needed a distributed warehouse.

Routing infrastructure matured. Companies like Greybeam are building the plumbing to make multi-engine seamless: proxy layers, SQL dialect translation, catalog resolution, result compatibility. You don’t need your team to build it from scratch.

This is the decision in front of data leaders: agent-generated traffic could 5x your query volume. On a single engine, your bill scales with it. On multi-engine, it doesn’t.

Furthermore, AI is making agent query patterns far more bursty and unpredictable.

If agents need to suddenly send 10 times as many queries, not only will token costs potentially run rampant, but where they’re running becomes more important too. Especially if multiple people are asking similar questions, a caching layer and the ability to help agents route queries to the best engine becomes imperative.

What Multi-Engine Should Look Like in Practice

Here’s the bar: if your team has to think about which engine runs a query, multi-engine isn’t working.

Data teams shouldn’t need to learn a new SQL dialect. They shouldn’t need to validate that results match across engines. They shouldn’t need to spin up DuckDB clusters on AWS, manage infrastructure, or figure out which queries are “eligible” for routing. The whole point of multi-engine is that the complexity lives in the routing layer.

What data folks should expect from a multi-engine setup:

• No new SQL to learn. Your team writes Snowflake SQL. The system handles dialect translation automatically.

• No manual routing decisions. The system knows which queries benefit from a cheaper engine and which should stay on Snowflake. Writes, large scans, and engine-specific syntax stay where they belong.

• No infrastructure to manage. Your team isn’t provisioning clusters, tuning memory, or debugging container orchestration. Compute is fully managed.

• No result discrepancies to chase. Query results are identical regardless of which engine executed them. Downstream dashboards, reports, and models don’t know or care.

• Performance gets better, not worse. Single-node analytical engines like DuckDB are often faster than a distributed warehouse for the small, read-heavy queries that make up the bulk of agent-driven traffic.

Early adopters of this approach are already seeing results. Headset, a cannabis data and analytics company serving 2,500+ embedded analytics users, reduced Snowflake costs by 92% after routing eligible queries to DuckDB through Greybeam.

The window for Multiple Query Engines is now

Multi-engine is not a future architecture. The pieces are here: Iceberg for storage, DuckDB and others for cheap analytical compute, and routing infrastructure like Greybeam to make it seamless.

The teams moving early aren’t doing it because multi-engine is interesting. They’re doing it because the cost and performance gap between running everything on one engine versus routing to the right engine is too large to ignore.

See more here

Your data stack is about to serve a lot more queries from a lot more agents. The question for data leaders is whether you set up the architecture now — while it’s a proactive decision — or retrofit later, when costs are sky high and the board is asking why.

We just wrote a comprehensive guide on 10 core checks for every Snowflake account. The fastest way to cut 30-60% of your Snowflake costs. Message us here if you enjoyed this to see it!

Substack note

Yes I am excited to report that agentic data engineering is actually here.

You see, with agents you should be able to go from 0 to 100 without lifting a finger.

For that, you kind of need all the scaffolding around alerting, orchestration, CI/CD and environments set-up already.

It is easy to use claude to generate some python, run it, and land it in a dev database. But what happens next?

This demo takes it further. We also generate the staging environment and the scaffolding to run the pipeline securely in staging and promote it from staging to prod using zero copy clones which is more efficient and faster

rather than run this from a git action post PR creation, we basically mock that by just getting claude to do it. In reality, you would already have a git action, Claude would then create a PR, give you a link to it, and then everything would be the same

We are closely getting to the point where you can just get this to work. Super cool. Skip the presentation to see the demo.

Introduction

I recently ran a webinar showing something I think is fundamentally different from most “AI for data engineering” demos.

A lot of demos today follow the same pattern:

Claude generates some Python, everyone claps, and the session ends

That’s interesting — but it’s not production engineering.

In this webinar with MotherDuck and Orchestra, I wanted to show what happens when AI is connected to an actual operational stack:

• orchestration

• deployments

• branching

• serverless compute

• database promotion workflows

• sandboxing

• observability

• BI

The result was a fully automated ELT pipeline built end-to-end by AI — safely deployed into production infrastructure.

And honestly, this feels like the direction modern data engineering is heading.

The Goal — AI generating a proper ELT workflow with production capability

The challenge was simple:

Use AI to create and deploy a production-grade pipeline from scratch.

Specifically, we wanted AI to:

1. Create a staging environment in MotherDuck

2. Generate Python ingestion code

3. Build an orchestration DAG

4. Deploy the pipeline through Orchestra

5. Execute it serverlessly

6. Create analytics automatically

7. Keep the entire workflow sandboxed and safe

No manual DAG writing.
No manually provisioning infra.
No hand-written deployment steps.

Just AI operating against a well-defined engineering system.

Step 1: Defining the Claude Skill

The key to making this work was not prompting.

It was defining a skill.

The skill acts as a structured operational framework for the AI agent. Instead of vaguely asking Claude to “build a pipeline,” we give it rules, conventions, deployment instructions, and safety guarantees.

One of the first things the skill enforces is:

Every load happens in staging first.

This matters enormously.

Without staging, an AI agent could accidentally overwrite production datasets or mutate live systems unexpectedly. That’s unacceptable in a real data platform.

So before any ingestion code is generated, the workflow creates a sandboxed MotherDuck staging database.

This is where MotherDuck Snapshots become incredibly powerful.

Snapshots make it trivial to create isolated, reproducible environments without copying huge datasets around manually.

That single capability changes how safe AI-driven engineering can be. The skill is included at the bottom.

Step 2: AI Generates the Ingestion Layer

Once the environment existed, the agent generated the ingestion logic.

In the demo, the pipeline moved data from Linear into MotherDuck.

But importantly, the AI wasn’t just generating arbitrary Python.

The skill told the model:

• where secrets live

• how functions should be structured

• how environment variables should work

• how deployments are parameterised

• how orchestration integrates with execution

This is the difference between “AI code generation” and “AI software systems.”

The generated pipeline already understood operational concerns before it even ran.

We actually used dlt for this and the source didn’t even exist, and now it does:

“”“Run the Linear cycles -> MotherDuck dlt pipeline.

Environment variables consumed:
    LINEAR_API_KEY                       Linear personal API key.
    MOTHERDUCK_API_TOKEN                 MotherDuck service token (alias of MOTHERDUCK_TOKEN).
    LINEAR_CYCLES_SINCE_DAYS             Lookback window in days for ``updatedAt`` (default 7).
    LINEAR_CYCLES_MD_STAGING_DATABASE    MotherDuck staging database (default ``my_db_staging``).
    LINEAR_CYCLES_MD_DATASET             Target schema / dlt dataset_name (default ``dlt_linear_cycles``).
“”“

import os

import dlt
from dlt.destinations import motherduck

from linear_cycles import linear_cycles_source


def run_pipeline(since_days: int = 7) -> None:
    api_key = os.environ.get(”LINEAR_API_KEY”)
    if not api_key:
        raise EnvironmentError(”LINEAR_API_KEY must be set.”)

    token = os.environ.get(”MOTHERDUCK_API_TOKEN”) or os.environ.get(
        “MOTHERDUCK_TOKEN”
    )
    if not token:
        raise EnvironmentError(
            “MOTHERDUCK_API_TOKEN (or MOTHERDUCK_TOKEN) must be set.”
        )

    database = os.environ.get(”LINEAR_CYCLES_MD_STAGING_DATABASE”, “my_db_staging”)
    dataset = os.environ.get(”LINEAR_CYCLES_MD_DATASET”, “dlt_linear_cycles”)

    destination = motherduck(
        credentials={”database”: database, “password”: token},
    )

    pipeline = dlt.pipeline(
        pipeline_name=”linear_cycles”,
        dataset_name=dataset,
        destination=destination,
    )

    source = linear_cycles_source(api_key=api_key, since_days=since_days)
    info = pipeline.run(source)
    print(info)


if __name__ == “__main__”:
    days = int(os.getenv(”LINEAR_CYCLES_SINCE_DAYS”, “7”))
    run_pipeline(since_days=days)
  

“”“Linear cycles source for dlt.

Fetches cycles from the Linear GraphQL API. The ``cycles`` resource is filtered
server-side by ``updatedAt`` so callers can request only a recent window
(e.g. the last 7 days) without scanning the entire workspace.
“”“

from datetime import datetime, timedelta, timezone
from typing import Any, Dict, Iterator, Optional

import dlt
from dlt.sources.helpers import requests


LINEAR_API_URL = “https://api.linear.app/graphql”

CYCLES_QUERY = “”“
query Cycles($filter: CycleFilter, $after: String) {
  cycles(filter: $filter, first: 100, after: $after) {
    pageInfo { hasNextPage endCursor }
    nodes {
      id
      number
      name
      description
      startsAt
      endsAt
      completedAt
      autoArchivedAt
      archivedAt
      createdAt
      updatedAt
      progress
      scopeHistory
      completedScopeHistory
      inProgressScopeHistory
      issueCountHistory
      completedIssueCountHistory
      team { id key name }
    }
  }
}
“”“


def _post(api_key: str, query: str, variables: Dict[str, Any]) -> Dict[str, Any]:
    response = requests.post(
        LINEAR_API_URL,
        json={”query”: query, “variables”: variables},
        headers={”Authorization”: api_key, “Content-Type”: “application/json”},
        timeout=60,
    )
    response.raise_for_status()
    payload = response.json()
    if payload.get(”errors”):
        raise RuntimeError(f”Linear API error: {payload[’errors’]}”)
    return payload[”data”]


@dlt.source(name=”linear_cycles”)
def linear_cycles_source(
    api_key: str = dlt.secrets.value,
    since_days: Optional[int] = 7,
):
    “”“Linear cycles dlt source.

    Args:
        api_key: Linear personal API key (header: ``Authorization: <key>``).
        since_days: Only emit cycles whose ``updatedAt`` is within this many
            days. Pass ``None`` to fetch the full history.
    “”“

    @dlt.resource(
        name=”cycles”,
        write_disposition=”merge”,
        primary_key=”id”,
    )
    def cycles() -> Iterator[Dict[str, Any]]:
        variables: Dict[str, Any] = {”after”: None}
        if since_days is not None:
            since = datetime.now(timezone.utc) - timedelta(days=since_days)
            variables[”filter”] = {”updatedAt”: {”gte”: since.isoformat()}}

        while True:
            data = _post(api_key, CYCLES_QUERY, variables)
            page = data[”cycles”]
            for node in page[”nodes”]:
                yield node
            if not page[”pageInfo”][”hasNextPage”]:
                break
            variables[”after”] = page[”pageInfo”][”endCursor”]

    return cycles

Step 3: AI Builds the Orchestration DAG

This is the point where most demos usually stop.

Instead, we continued all the way through deployment.

The agent generated an Orchestra pipeline YAML file describing the workflow end-to-end.

That YAML defined:

• the Python execution step

• environment variables

• dependencies

• MotherDuck promotion logic

• staging → production movement

• snapshot creation

• orchestration configuration

See here:

version: v1
name: ‘Linear cycles to MotherDuck #dlt #motherduck #linear’
pipeline:
  3a1f8b40-77c2-4d18-9e2a-6b0c5d7f8a11:
    tasks:
      6c4d9e22-8a3b-4f51-bd07-2e1a4f6b9c23:
        integration: PYTHON
        integration_job: PYTHON_EXECUTE_SCRIPT
        parameters:
          package_manager: PIP
          python_version: ‘3.11’
          build_command: pip install -r requirements.txt
          source: GIT
          branch: ${{ inputs.branch }}
          command: python -m linear_cycles_pipeline
          project_dir: python/dlt
          shallow_clone_dirs: python/dlt
          environment_variables: ‘{ “LINEAR_CYCLES_SINCE_DAYS”: “${{ inputs.since_days
            }}”, “LINEAR_CYCLES_MD_STAGING_DATABASE”: “${{ inputs.md_staging_database
            }}”, “LINEAR_CYCLES_MD_PROD_DATABASE”:    “${{ inputs.md_prod_database
            }}”, “LINEAR_CYCLES_MD_DATASET”:          “${{ inputs.md_dataset }}” }’
          set_outputs: true
        depends_on: []
        name: Run linear cycles dlt
        connection: python__production__blueprints__19239
    depends_on: []
    name: ‘’
  5e9a3c14-2b6d-4f7e-8c91-1a3b5d7e9f02:
    tasks:
      a1b2c3d4-e5f6-7a8b-9c0d-1e2f3a4b5c6d:
        integration: MOTHERDUCK
        integration_job: MOTHERDUCK_EXECUTE_QUERY
        parameters:
          query: CREATE SNAPSHOT ${{ inputs.snapshot_name }} OF ${{ inputs.md_staging_database
            }}
          set_outputs: false
        depends_on: []
        name: Create snapshot of prod
    depends_on:
    - d15d7c8a-e4b8-4cbc-a19f-b9be813a782d
    name: ‘’
  8d2f5a91-4b7c-4e83-9d1a-3f6b8c2e7a45:
    tasks:
      b2d3c4e5-6f7a-8b9c-0d1e-2f3a4b5c6d7e:
        integration: MOTHERDUCK
        integration_job: MOTHERDUCK_EXECUTE_QUERY
        parameters:
          query: COPY FROM DATABASE ${{ inputs.md_staging_database }}  (OVERWRITE,
            SNAPSHOT_NAME ${{ inputs.snapshot_name }}, OVERWRITE) TO ${{ inputs.md_prod_database
            }}
          set_outputs: false
        depends_on: []
        name: Promote staging to prod
    depends_on:
    - 5e9a3c14-2b6d-4f7e-8c91-1a3b5d7e9f02
    name: ‘’
  d15d7c8a-e4b8-4cbc-a19f-b9be813a782d:
    tasks:
      9c8bb4c1-352f-48e9-a31e-5192c57fc6e1:
        integration: DBT_CORE
        integration_job: DBT_CORE_EXECUTE
        parameters:
          commands: dbt build --select models_i_just_did
          package_manager: PIP
          python_version: ‘3.12’
        depends_on: []
        name: dbt modelling
    depends_on:
    - 3a1f8b40-77c2-4d18-9e2a-6b0c5d7f8a11
    name: group
webhook:
  enabled: false
inputs:
  branch:
    type: string
    default: feat/dlt-linear-cycles
  since_days:
    type: string
    default: ‘7’
  md_staging_database:
    type: string
    default: my_db_staging
  md_prod_database:
    type: string
    default: my_db
  md_dataset:
    type: string
    default: dlt_linear_cycles
  snapshot_name:
    type: string
    default: dlt_linear_cycles

Because Orchestra pipelines are declarative YAML, they become highly compatible with AI systems. The model can reason about dependencies, execution order, parameters, and deployment states in a way that’s much harder with imperative orchestration code.

The Orchestra CLI is used to push this code to Orchestra and run it.

Step 4: Everything Happens on a Branch

Another critical detail:

The AI opens a feature branch automatically.

This matters more than people realise.

The branch becomes the isolated deployment target for the generated pipeline. Orchestra can execute directly against that branch before anything reaches main.

That means:

• safe experimentation

• isolated testing

• reproducible deployments

• rollback protection

• production-grade CI/CD patterns

Again — this is the difference between a toy demo and a real engineering workflow. You can see when it’s running below the branch we are running too

Step 5: AI Deploys the Pipeline

Once the code and orchestration existed, Claude used the Orchestra CLI to register the pipeline automatically.

This was one of the most exciting parts of the demo.

The workflow looked like this:

1. AI generates Python

2. AI generates orchestration YAML

3. AI commits to Git

4. AI registers the pipeline through Orchestra

5. Orchestra executes the pipeline serverlessly

The deployment process itself became programmable. That’s a major shift. This we ran from Claude, but in practice you could run this from a git action too.

Why Orchestra Serverless Matters

A huge reason this works smoothly is Orchestra Serverless.

When Orchestra executes Python code, it:

• connects to the Git repository

• pulls the branch

• installs dependencies

• runs the pipeline on ephemeral infrastructure

No Kubernetes setup. No Airflow workers. No infrastructure management.

This removes one of the biggest blockers preventing AI-generated systems from becoming operationally viable as infrastructure requirements go away. Very important for autonomous engineering workflows.

Why MotherDuck Fits So Well

MotherDuck also played a huge role in making the experience smooth.

Writing data into MotherDuck is extremely lightweight. There’s no complicated warehouse provisioning process. No heavyweight setup. No operational burden.

Combined with snapshots, you get something very powerful:

• isolated staging environments

• instant cloning

• rollback capability

• sandboxed AI experimentation

This creates a surprisingly elegant foundation for AI-native data engineering. Personally-speaking (not the AI), I had quite a lot of things to figure out:

1. motherduck MCP Server: no issues here. Easy to query docs and work out how to clone things

2. Snapshots and overrides: again, clear docs, commands all “just work”

3. Checking things with system tables: easy to see snapshot history, query history

Everything genuinely seemless, as evidenced by the smooth demo!

Some refs:

https://motherduck.com/docs/sql-reference/motherduck-sql-reference/create-snapshot

https://motherduck.com/docs/sql-reference/motherduck-sql-reference/copy-database-overwrite/

Extending the System Further

One thing I emphasised during the webinar is that this architecture is extensible. The ingestion pipeline was only the beginning.

For example:

• the agent could also generate dbt models

• AI could create tests automatically

• downstream transformations could be orchestrated

• documentation could be generated

• lineage could be updated

• monitoring rules could be authored

Once the deployment substrate exists, AI can operate across the entire analytics lifecycle. We decided to do this in the demo for Dives, a code-first BI capability that exists in Motherduck

Introducing Dives: BI as Code

The final part of the demo focused on something I’m personally very excited about:

Dives.

A Dive is essentially “BI as code.”

Instead of manually editing dashboards, the dashboard itself becomes programmable and parameterised.

In the demo, AI automatically updated the Dive with:

• the latest data refresh timestamp

• a direct link back to Orchestra lineage

Sounds simple, but it solves a real operational problem. Anyone viewing the dashboard can instantly answer:

• Is this data fresh?

• When did the pipeline last run?

• What happened upstream?

• Where can I inspect failures?

So very fun to easily get Claude to build these gnarly little things.

Conclusion | Embrace change!

The interesting part of the webinar wasn’t really that Claude could generate Python. We’ve all seen that before. What felt more useful was seeing what happens when you connect code generation to actual deployment infrastructure.

Most AI demos stop before the operational parts begin. There’s no branching strategy, no staging environment, no orchestration layer, no deployment process. The generated code exists in isolation.

In this workflow, the important thing was the surrounding system:

• staging databases in MotherDuck

• orchestration through Orchestra

• feature branches

• snapshots

• declarative pipelines

• serverless execution

• lineage and observability

Those things sound boring compared to “AI”, but they’re actually the parts that make the workflow usable. Without them, you mostly end up with generated scripts that still need a human to operationalise everything around them.

With them, you can start treating generated code like a normal engineering workflow: branch it, deploy it, run it safely, inspect it, roll it back if needed.

Final Recap

In the demo, the agent created a staging database in MotherDuck, wrote the ingestion code, generated the orchestration pipeline, opened a branch, deployed the workflow through Orchestra, and ran the pipeline serverlessly.

After the load completed, we used a Dive to inspect the data and expose some operational metadata like freshness timestamps and links back to the pipeline lineage.

It’s cool to see AI maturing a bit and making data engineering easier. Definitely a win for the Seniors. As for the juniors..what juniors?!

The Skill

Note — it is way too long and way too verbose. Some work needed.

---
name: dlt-source-via-orchestra
description: Build a new dlt source that lands API data into MotherDuck, then ship it through Orchestra. Use this when the user wants to “pull X from <API> into MotherDuck” or “add a new dlt source and run it on Orchestra”. Walks the full lifecycle on a feature branch — author the source, register an Orchestra pipeline that invokes it, trigger the pipeline against that branch, and poll until done.
---

# Build & ship a dlt → MotherDuck source via Orchestra

Use this end-to-end when adding a new API → MotherDuck source. The dlt code goes
in `python/dlt/`; the Orchestra pipeline that runs it goes in `orchestra/`. The
pipeline must be runnable against the feature branch before it lands on `main`,
so every step is parameterised by the branch created in step 1.

## Steps

### 1. Create a feature branch
From `main`, cut a branch named `feat/dlt-<source>` (e.g. `feat/dlt-linear`).
Save the branch name verbatim — you will pass it to `start_pipeline` in step 7
and substitute it into the Python task in step 4. Step 8 promotes the staged
load into prod, so make sure the prod DB name set in step 3 is the one the user
actually wants overwritten.

```bash
git checkout main && git pull
git checkout -b feat/dlt-<source>
```

**If the source already exists on `main`** (i.e. you’re “rebuilding” or
re-running a previously merged pipeline): cut a `-v2`/`-rerun` branch and
**keep the existing `python/dlt/<source>/`, `<source>_pipeline.py`, and
`orchestra/<source>_to_motherduck.yml` files as-is** — skip ahead to step 5.
“Rebuild” in this context means re-register and re-trigger on a fresh branch,
not delete and re-author working code. Only rewrite files if the user
explicitly says the existing code is broken or that they want it replaced.

### 2. Identify the source endpoints
Read the source’s API docs (or call `WebFetch` on them) to determine:
- Which endpoint(s) yield the entities the user asked for.
- Pagination style: cursor (`pageInfo.endCursor`), offset, link headers, or `next` URLs.
- Server-side filtering for the requested window (e.g. `updatedAt >= now-7d`). Do
  the filter in the API request, not client-side — keeps the load small.
- Auth mechanism: header name (`Authorization: <token>` vs `Bearer <token>`),
  query-string token, OAuth.

Write these down before coding — guessing pagination or filter syntax wastes a
full dlt run.

### 3. Locate credentials and set destination env vars
Both the source API and MotherDuck must be reachable from env vars (Orchestra
injects them via the Python integration’s `environment_variables`). Probe for
likely names before writing code:

```powershell
[Environment]::GetEnvironmentVariables(’User’).Keys |
  Where-Object { $_ -match ‘<source>|motherduck|md_’ }
```

**Credential env vars (must already exist — never hardcode):**
- MotherDuck: `MOTHERDUCK_API_TOKEN` (also accepted as `MOTHERDUCK_TOKEN`).
- Source API: e.g. `LINEAR_API_KEY`, `HUBSPOT_API_KEY`.

If a credential is missing, stop and ask the user — do not hardcode.

**Destination env vars (set defaults if unset, then promote to the Process env
so the local validation in step 7 and the Orchestra task config pick them up).
The pattern is: dlt lands into a **staging** database, prod is overwritten by
the step-8 COPY only after the load looks clean:**

| Var | Purpose | Default |
| --- | ------- | ------- |
| `<SOURCE>_MD_PROD_DATABASE`    | Final destination (overwritten by step 8) | `my_db` |
| `<SOURCE>_MD_STAGING_DATABASE` | Where dlt lands first | `<prod>_staging` (e.g. `my_db_staging`) |
| `<SOURCE>_MD_DATASET`          | Target schema (dlt `dataset_name`) | `dlt_<source>` |
| `<SOURCE>_MD_SNAPSHOT_NAME`          | Name fo the snapshot of the staging database  | e.g. `linear_19052026` |

PowerShell — set only if not already present:

```powershell
if (-not $env:<SOURCE>_MD_PROD_DATABASE)    { $env:<SOURCE>_MD_PROD_DATABASE    = ‘my_db’ }
if (-not $env:<SOURCE>_MD_STAGING_DATABASE) { $env:<SOURCE>_MD_STAGING_DATABASE = “$($env:<SOURCE>_MD_PROD_DATABASE)_staging” }
if (-not $env:<SOURCE>_MD_DATASET)          { $env:<SOURCE>_MD_DATASET          = ‘dlt_<source>’ }
if (-not $env:<SOURCE>_MD_SNAPSHOT_NAME)    { $env:<SOURCE>_MD_SNAPSHOT_NAME    = ‘dlt_<source>_pre_<YYYYMMDD_HHMMSS>’ }
```

These three env vars flow through all the later steps unchanged — step 6
snapshots `<SOURCE>_MD_PROD_DATABASE`, the Python script in step 4a writes to
`<SOURCE>_MD_STAGING_DATABASE` to create a snapshot called $env:<SOURCE>_MD_SNAPSHOT_NAME,
the Orchestra task in step 4c passes all three
through `environment_variables`, and step 8 reads both database names when
building the `COPY FROM DATABASE` SQL. Keep the names consistent across local
shell, Python script, YAML task, and MCP calls.

### 4. Author the dlt source AND the Orchestra YAML

**4a. Code under `python/dlt/<source>/`:**

- `python/dlt/<source>/__init__.py` — `@dlt.source` and `@dlt.resource` definitions,
  using `dlt.sources.helpers.requests` for HTTP. Resource should be
  `write_disposition=”merge”` with the API’s stable primary key, and accept a
  `since_days` argument that translates into the server-side filter.
- `python/dlt/<source>_pipeline.py` — wires env vars and runs the pipeline:
  - Pulls `MOTHERDUCK_API_TOKEN` and the source token from `os.environ`.
  - Reads destination from env:
    `database = os.environ.get(”<SOURCE>_MD_STAGING_DATABASE”, “my_db_staging”)`,
    `dataset  = os.environ.get(”<SOURCE>_MD_DATASET”, “dlt_<source>”)`.
  - Sets `DESTINATION__MOTHERDUCK__CREDENTIALS=md:<staging_database>?motherduck_token=<token>`.
  - Calls `dlt.pipeline(pipeline_name=”<source>”, dataset_name=dataset, destination=”motherduck”)`.
  - `LINEAR_SINCE_DAYS`-style env var for the window (default 7 in `__main__`).

The script writes to **staging** only — it must never read `<SOURCE>_MD_PROD_DATABASE`.
Prod is touched exclusively by the step-8 COPY task; keeping that separation
means a half-failed load only corrupts staging, and step 6’s snapshot of prod
is a clean rollback point.

**4b. Add any new packages to `python/dlt/requirements.txt`:**

Orchestra’s Python integration runs `pip install -r requirements.txt` in the
checked-out repo. The dlt MotherDuck extra is mandatory for this pattern and is
not pulled in by `dlt[bigquery]`:

```
dlt[motherduck]==1.16.0
```

Pin the version to the same line as the other `dlt[...]` extras so the resolver
picks one consistent dlt build. If the source needs a vendor SDK (e.g.
`hubspot-api-client`), add it here too.

**4c. Orchestra pipeline at `orchestra/<source>_to_motherduck.yml`:**

The pipeline has a single PYTHON_EXECUTE_SCRIPT task. The two non-obvious bits:

1. **Top-level `inputs.branch.default: main`** declares a branch override that
   defaults to `main`, so a scheduled run on `main` keeps working.
2. **`branch: ${{ inputs.branch }}`** on the Python task wires that override
   into the integration. At trigger time, `start_pipeline(branch=<feature>)`
   replaces it.

Skeleton (let the `orchestra-pipeline` skill generate the full YAML; this is
just the shape). Two stages: Python load → MotherDuck promote. The promote is
wired into the pipeline as a `MOTHERDUCK_EXECUTE_QUERY` task that depends on
the Python stage, so prod is updated atomically as part of each run and there
is no separate manual MCP `COPY FROM DATABASE` step. CHECK to see if the `COPY FROM DATABASE` syntax is correct for your version of MotherDuck — the example below uses `OVERWRITE` to copy data from staging snapshot into the prod database. If not, update the orchestra pipeline.yml.

```yaml
version: v1
name: ‘Linear cycles to MotherDuck #dlt #motherduck #linear’
pipeline:
  3a1f8b40-77c2-4d18-9e2a-6b0c5d7f8a11:
    tasks:
      6c4d9e22-8a3b-4f51-bd07-2e1a4f6b9c23:
        integration: PYTHON
        integration_job: PYTHON_EXECUTE_SCRIPT
        parameters:
          package_manager: PIP
          python_version: ‘3.11’
          build_command: pip install -r requirements.txt
          source: GIT
          branch: ${{ inputs.branch }}
          command: python -m linear_cycles_pipeline
          project_dir: python/dlt
          shallow_clone_dirs: python/dlt
          environment_variables: ‘{
            “LINEAR_CYCLES_SINCE_DAYS”: “${{ inputs.since_days }}”,
            “LINEAR_CYCLES_MD_STAGING_DATABASE”: “${{ inputs.md_staging_database }}”,
            “LINEAR_CYCLES_MD_PROD_DATABASE”:    “${{ inputs.md_prod_database }}”,
            “LINEAR_CYCLES_MD_DATASET”:          “${{ inputs.md_dataset }}”
            }’
          set_outputs: true
        depends_on: []
        name: Run linear cycles dlt
        connection: python__production__blueprints__19239
    depends_on: []
    name: ‘’
  5e9a3c14-2b6d-4f7e-8c91-1a3b5d7e9f02:
    tasks:
      a1b2c3d4-e5f6-7a8b-9c0d-1e2f3a4b5c6d:
        integration: MOTHERDUCK
        integration_job: MOTHERDUCK_EXECUTE_QUERY
        parameters:
          query: ‘CREATE SNAPSHOT ${{ inputs.snapshot_name }} OF ${{ inputs.md_staging_database }}’
          set_outputs: false
        depends_on: []
        name: Create snapshot of prod
    depends_on:
    - 3a1f8b40-77c2-4d18-9e2a-6b0c5d7f8a11
    name: ‘’
  8d2f5a91-4b7c-4e83-9d1a-3f6b8c2e7a45:
    tasks:
      b2d3c4e5-6f7a-8b9c-0d1e-2f3a4b5c6d7e:
        integration: MOTHERDUCK
        integration_job: MOTHERDUCK_EXECUTE_QUERY
        parameters:
          query: “COPY FROM DATABASE ${{ inputs.md_staging_database }}  (OVERWRITE, SNAPSHOT_NAME ${{ inputs.snapshot_name }}, OVERWRITE) TO ${{ inputs.md_prod_database }}”
          set_outputs: false
        depends_on: []
        name: Promote staging to prod
    depends_on:
    - 5e9a3c14-2b6d-4f7e-8c91-1a3b5d7e9f02
    name: ‘’
inputs:
  branch:
    type: string
    default: feat/dlt-linear-cycles
  since_days:
    type: string
    default: ‘7’
  md_staging_database:
    type: string
    default: my_db_staging
  md_prod_database:
    type: string
    default: my_db
  md_dataset:
    type: string
    default: dlt_linear_cycles
  snapshot_name:
    type: string
    default: dlt_linear_cycles

```

Tokens (`MOTHERDUCK_API_TOKEN`, source key) come from the connection’s
environment, not `environment_variables` in the YAML — never inline secrets.

### 5. Register the pipeline with Orchestra
Commit the YAML + Python code to the feature branch and push. Then use the
Orchestra MCP `import_pipeline` tool with the YAML contents to register it. The
tool returns the pipeline UUID — save it (you need it for step 7 and for any
future re-runs).

```
mcp__orchestramcp__import_pipeline(yaml=”<contents of orchestra/<source>_to_motherduck.yml>”)
```

If the pipeline already exists at the same alias, this updates it in place.

### 6. Prepare staging and snapshot prod
Two MotherDuck MCP `query_rw` calls before the pipeline runs:

```
mcp__MotherDuck__query_rw(
  sql=”CREATE DATABASE IF NOT EXISTS <md_staging_database>”
)


```

The first ensures dlt has somewhere to land — MotherDuck will not auto-create
the database, only schemas within it. 

Database names come from the env vars set in step 3 — substitute them
into the SQL string before sending to MCP. If either call fails (database
doesn’t exist, snapshot name collision), stop and surface the error. Don’t
proceed to a load that has no rollback point.

### 7. Trigger on the feature branch and poll
Trigger the pipeline against the branch from step 1, then poll until terminal.

```
mcp__orchestramcp__start_pipeline(alias=”<pipeline-uuid>”, branch=”<feature-branch>”, snapshot_name=”snapshot_name”)
mcp__orchestramcp__get_pipeline_run_status(pipeline_run_id=”<run-uuid>”)
```

Poll `get_pipeline_run_status` until the status is not `RUNNING` or `CREATED`.
Report the terminal status and surface the Orchestra UI link:

```
https://app.getorchestra.io/pipeline-runs/<run-uuid>/lineage
```

On failure, call `list_task_runs` (filtered by the run UUID) to find the FAILED
task, then `download_task_run_log` for its log. The most common causes:
- `requirements.txt` missing the new extra (e.g. `dlt[motherduck]`) → the
  `motherduck` destination import errors out. Fix step 4b and retrigger.
- Env var name mismatch between the script and the connection’s env config →
  rename in `<source>_pipeline.py` to match the connection. Applies to credential
  vars and `<SOURCE>_MD_STAGING_DATABASE` / `<SOURCE>_MD_PROD_DATABASE` /
  `<SOURCE>_MD_DATASET`.
- API rate-limit / 401 → token is missing from the connection’s env config, not
  a code bug.
- Load partially corrupted an existing dataset → `RESTORE` from the snapshot
  created in step 6, then fix the bug and re-run.

Once the run is green on the feature branch, the staging database holds the
fresh load. Prod is still untouched — promote it in step 8.

### 8. Verify the promote

First create a snapshot of the promote in staging

```
mcp__MotherDuck__query_rw(
  sql=”CREATE SNAPSHOT <source>_pre_<YYYYMMDD_HHMMSS> OF <md_staging_database>”
)
```

Once the pipeline run is green end-to-end (Python stage SUCCEEDED, promote
stage SUCCEEDED), spot-check the prod result via the MotherDuck MCP:

```
mcp__MotherDuck__query(database=”<md_prod_database>”,
  sql=”SELECT COUNT(*) FROM <md_dataset>.<main_table>”)
mcp__MotherDuck__query(database=”<md_prod_database>”,
  sql=”SELECT MIN(updated_at), MAX(updated_at) FROM <md_dataset>.<main_table>”)
```

If the promote stage failed but the Python stage succeeded, staging holds the
correct data and prod is untouched — re-run only the promote stage from the
Orchestra UI (or re-trigger the whole pipeline; `OVERWRITE` is idempotent). If
the COPY itself corrupted prod, `RESTORE` from the snapshot taken in step 6.

Once both stages are green on the feature branch, the YAML and Python code are
ready to merge to `main`.

## Notes
- The promote in step 8 is wired into the pipeline as a
  `MOTHERDUCK_EXECUTE_QUERY` task that depends on the Python stage — it runs
  automatically on every trigger, no manual MCP call required. The pre-load
  snapshot in step 6 is still a manual MCP step; consider promoting it into
  the YAML as a third `MOTHERDUCK_EXECUTE_QUERY` task (no `depends_on`, with
  the Python task gaining `depends_on: [<snapshot-stage>]`) once the pattern
  is proven on the source.
- Staging vs prod: dlt should never write directly to prod. The split exists
  so a half-failed load only corrupts staging (which is rebuilt every run),
  and prod only changes via the atomic `COPY FROM DATABASE … (OVERWRITE)` in
  step 8. If you find yourself reaching for `<SOURCE>_MD_PROD_DATABASE` in
  the Python script, stop — that’s the smell that the split is being bypassed.
- The Python task uses `shallow_clone_dirs: python/dlt` so the integration only
  pulls that subtree — adding files outside `python/dlt/` won’t slow checkout.
- `branch` on the Python task is the canonical way to scope a run to a feature
  branch. `start_pipeline`’s `branch` argument overrides the whole pipeline’s
  `inputs.branch`, including the dbt and other Git-aware tasks if you later add
  them. Don’t try to override per-task.
- See `fix-and-rerun-pipeline` for the failure-diagnosis loop once the pipeline
  exists. See `orchestra-pipeline` (built-in) for full YAML generation against
  the documented schema.

Introduction

Dashboards have never been killed because they operate within the Valley of Choice in BI.

The Valley of Choice describes the perfect ratio between the complexity of a problem and the effort we are willing to put in to solve it.

For example, a business critical question about how revenue is formed, and why different versions of revenue changed over time can warrant manual exploration. It can warrant deep discovery facilitated by multiple SQL queries and manually writing down the results and tracing through lineage.

This is High Effort, High Accuracy.

By contrast, simple questions on simple datasets like “How many sign-ups did I get yesterday?” can be satisfied by lower effort, simpler methods such as basic Text to SQL, as seen in tools like Thoughtspot.

In the centre are dashboards. Dashboards provide a moderate effort solution to a moderate complexity problem. They are capable of answering multiple questions and pivoting data to explain anomalies and forecast trends. They do not help discover insights, but help you deliver insights you have already found.

When something becomes asked frequently enough, it becomes worthy of a dashboard, and immediately graduates from a tricky, one-off, manually provided insight to something simpler and more repatable.

Agentic Analytics offers a new way to discover, and to serve insights. That is Text-to-SQL. This new way means that hitherto moderately complex queries can now be answered instantly.

Tools like Hex incorporate memory into their platforms, that mean that even complex queries, when combined with memory, allow Text-to-SQL to solve complex problems.

This changes the necessary ratio of effort to complexity and will reshape how we think of analytics. Let’s see how.

Explaining the Valley of Choice in BI

The Valley of Choice can be pictured below.

It describes a tradeoff between speed and accuracy which is determined by the nature of the demand or insight. Holding all things constant, the most important questions demand the highest degree of accuracy, which in turn justifies more time-consuming (High effort) methods.

It is a bit like saying that you can buy a delicious sandwich from your deli for $20 or go to 11 Madison Park for $500. Two different products, two different options, two different price points. Dashboards are like a meal out at a normal restaurant. $100 per person, but not suitable if you’re an eligible bachelor in New York.

Now: Imagine there is a new restuarant. The New Restuarant is very exclusive. The food is incredible; everything is organic. The service excellent. And lots of celebrities go there. It is very desirable. You can get a delicious 3 course meal for $40. We can call it Hugo’s Diner.

In fact it is part of a new trend. Hugo has a vast restaurant empire. There is Hugo’s Diner, but there is also Casa Hugo, Chez Hugo, Hugo’s table, and even Hugo-San. Hugo-San offers the best omakase in the city for $39.50.

What do you do? Where do you dine? 11 Madison Park is suddenly not so busy anymore. And this is what is happening with BI.

Simply put, the cost, and speed of running a query through new-age text-to-sql is incredibly low. It means there is essentially no reason to ever do anything else, unless of course you absolutely have to. In any case, you certainly aren’t moving downmarket.

Yes — sometimes you might need to do something manually. Just to be sure. And in the same way, nothing quite hits date (data?) night like Dorsia.

The Shifting Valley: The bar for Analytics just got higher

Imagine you don’t need to work hard to get the semantic layer going and Text to SQL can solve, reliably, repeatably, 95% of your questions.

An analyst becomes almost like a super analyst. They are only going to be asked the really gnarly questions. They are probably the person that designed the semantic layer in the first place.

The price of accuracy has become much higher. To do something manually, you had better be asking an extremely important question.

What does this mean for Analysts? Well, much the same as it means for software engineers. It now means that the proportion of folks actually being useful just got a lot smaller — in a meritocratic way, of course.

An Analyst that cannot be a semantic architect will be useless in 12 months. They will be replaced by insights agents. And if they persist, then CTOs are doing something wrong and should probably bring in Service Partners to help them implement their semantic layer.

The bar for working in Analytics just got higher. There’s a new competitor on the block and their name is A. Gent. Just like software engineering — juniors are out, and seniors are in.

What about token costs?

Token costs is an interesting point. Token costs have generally decreased by 95% since the first models. The cost of inference continues to fall, and is expected to be reduced by a further 95% over the next few years.

Despite this, companies like Uber are steaming through their credit bills. After all, credits are fun. Claude is addictive. Tokenmaxxing is encouraged.

Indeed, the price of frontier models has not fallen dramatically. Claude costs in 2023 were c.$32/million output tokens. Now it is still $25.

In fact, token use is expected to rise exponentially as systems get more complicated— it is not clear when, if at all, benefits will arise. For example 1 million output tokens for Claude is about $25. Let’s say running a few queries through an agent is 1,000 tokens. This means the cost of a query is 0.025 of 2.5 cents.

10,000 people doing one of these a day for 365 days a year is $91k. Not bad.

If the cost of analysts was say 10 analysts at $100k each, then again the ROI starts to look good. It looks even better if inference costs reduce, which they must.

I have a suspicion that assumption (1,000 tokens per “insight”) is too low. However, reducing inference costs and improvements in technology will inevitably continue to reduce it.

For example, suppose initially it takes an agent a long time to learn that when marketing says “show me revenue” it is a different set of queries to when finance say “show me revenue”.

Once this is codified, and done repeatably, this should be minimal tokens. It should become a dashboard, or persistent query, or perhaps even a skill of the agent. We do not need a million output tokens when people ask the same questions every week.

And of course generally, people do ask similar questions as time goes by.

I am therefore, bullish on token costs. While there is an inevitably strange moment in the market at the moment, whereby the cost of inference is probably too high, I do believe the cost of inference will decrease. The number of tokens required to produce insights reliably and repeatably will also decrease as people build more reliable and repeatable systems.

Which BI vendors win and lose?

There are a few factors at play

1. Strength of agent

2. Market Positioning

3. Size and Estate

Strength of Agent (Agentic Analytics)

It should go without saying the strength of the agent is imperative. There are currently a handful of companies with powerful analytics agents.

1. Lightdash

2. Omni

3. Hex

4. TextQL

These companies are all succeeding for different reasons. Lightdash have a tight integration with the dbt semantic layer. Omni have their own, being ex-Looker, and understand the space very well. Hex have an incredible agent infrastructure and memory, while TextQL is simply one of those things that has to be used to be believed.

It is worth noting data warehouses have their own versions of this too. Snowflake Cortex Intelligence and Databricks Genie are both products with a lot of investment behind them. If good enough, customers will use these.

Market Positioning

Not every BI vendor is really in a position to care, or worry about these factors. For example, Tableau will continue to accrue revenue through its livelihood through Salesforce. Its success will be determined by the stickiness of Salesforce.

Standalone BI tools without an edge are in trouble here. The Sisenses, the Qliks and the Lookers have very little reason to continue being chosen if they cannot develop a better agentic analytics layer — and they do not have a big brother in the Boardroom to look after them.

Companies like Sigma, who have historically been very close to companies like Snowflake, now compete with them directly. So they are in a tricky spot to see who can build the best AI Data Analyst.

Size and Estate

Some companies may simply have too much size and estate to be fully displaced. For example, both Tableau, Sigma and Power BI have enormous estates that cannot be removed easily.

If dashboards are serving their purpose, reliably, they will remain. Remember — dashboards exist in the Valley of Choice. This is because clicking into a known, reliable dashboard will still be lower effort than formulating a question, typing it into a chat window, waiting, asking for graphs to be drawn, and so on.

This means that companies that have implemented dashboards will likely keep them. However, expect there to be heavy pricing pressure as agentic BI forces out user-based pricing models.

Conclusion — BI will consolidate in 3 years

The logical conclusion is more consolidation.

AI means building Dashboards becomes exceptionally easy.

The shift of the BI Valley from uncanny to canny means the cost tradeoff between agentic analytics, dashboards, and manual work has shifted dramatically.

Practically, usage-based models of consumption directly counter legacy user-based pricing models, if not on structure then at minimum on cost. While companies may be willing to pay a small user fee for an agentic BI tool like Hex, they are certainly not willing to pay $5,000 for the privilege of making some Tableau Dashboards.

The Final Voyage of the Modern Data Stack
Can the Context Layer for AI provide catalogs with the last chopper out of Saigon?medium.com

All the while, companies will compete directly with the largest, most well-capitalised, and most research-heavy companies in the space: hyperscalers and data warehose providers. There is no reason not to leverage tools like Snowflake Cortex Intelligence and Databricks Genie if they are sufficiently good.

This means the industry will continue to consolidate around BI and Analytics agents. The space will become increasingly commoditised unless the standalone vendors can continue to make better and better products faster than the warehouses can catch-up.

At Orchestra, we’re building the Agentic Layer for the engineers. Follow-us to find out more.

Introduction to dbt Orchestration

dbt core has been downloaded over 1 billion times. Over 30,000 organisations use dbt in production, the number grows every day. Around 5% of the install base have over 5,000 models.

While learning dbt is easy, maintaining it is not. Analytics Engineering is a new profession and with it there is more dbt code being generated than ever before — taming spaghetti lineage has become a full time job for many, and with AI rising faster than you can say “dbt run”, this problem only seems to be getting worse.

One key area of concern is scheduling. Where a simple daily “dbt run” suffices to begin with, as companies and data teams grow quickly, the dbt estate gets more complicated. Before long, models must be tagged in different ways such as “tag:hourly” or “tag:daily”, which quickly becomes difficult to maintain.

Furthermore, as many growth stage companies expand, monitoring and cleaning up old models can be painful. What should data engineers do with contacts_hubspot and opportunities_salesforce as the company migrates to contacts_new_ai_powered_crm? It is difficult to balance the needs of the business while retaining sanity within a dbt project.

This problem is not unique to growth-stage companies, either. Large enterprises who feel the pain of decades of technical debt may and “self-serve analytics” may observe that the estate of SQL models is simply too large, but that refactoring is not a plausible option.

One solution to this is state-aware orchestration for dbt or “SAO”. While dbt fusion already has an impressive 500+ Github Stars, the traction for dbt core is unparalleled. We wanted to help solve these gnarly problems for dbt practitioners in a way that finally contributed back to the dbt community.

In this guide, we’ll cover some common dbt FAQs

Common dbt FAQs

• What is State-Aware Orchestration and what does it do?

• Why State-Aware Orchestration matters for data teams

• How dbt State-Aware Orchestration works

• How to configure dbt State-Aware Orchestration locally

• How to configure dbt State-Aware Orchestration in Orchestra

• Managing state in Orchestra

• Monitoring the impact of State-Aware Orchestration

By the end of this guide, you’ll be a certified dbt SAO practitioner!

Let’s dive in.

Want to get started? Check-out the resources below.

📚 Read the Docs

🚀 Create an Orchestra account

What does dbt State-Aware Orchestration do?

By default, dbt projects take a fairly unopinionated approach to transforming your data, with “run everything every hour and pray” being a popular approach to keeping analytics fresh within a business.
But, with time and scale, this translates into run times creeping up, larger warehouse instances, and days spent refactoring your projects.
Perhaps a better way would be to:

Transform the data that’s needed, when it’s needed

That’s where state-aware orchestration comes in.

Your orchestrator already sees everything that’s going on in your DAG, so tracking the state of each model — when it was last built, whether anything upstream has actually changed — is very much within its wheelhouse, and that information can be used to decide what needs building and what can be skipped entirely.

dbt freshness checks tell us when new data is available in our sources, but why not extend this to models? When is data available at the end of my pipeline?
Conversely, our stakeholders have processes designed around consuming that data, so why not ensure we’re updating those tables for when they’re needed? And why update them when they’re not needed?

This is state aware orchestration.

No new data? Don’t update that staging table, use it as is.

Upstream models haven’t changed? Neither will this one.

Daily mart table already built this morning? If a dbt model updates, and nobody’s around to use it…

Why does state-aware orchestration matter for data teams?

Faster, more predicatable, less computationally intensive job runs are an easy sell to any data engineering team, but where else do we gain by dbt’s saving?We’ll start with the headliner, then check out some other benefits.

Less reprocessed data
Not reprocessing the same data again and again is a clear win here, freeing up time in all services, reducing the amount of compute required each time, and ultimately taking the edge off the bills that keep Platform Engineers and Managers awake at night.

Incremental Improvements
Increments in dbt are extremely powerful, and one of the top ways to make a dbt project more efficient, but it’s not always a case of adding a simple Jinja block.
State aware orchestration doesn’t eliminate the need for them either, but rather than executing a huge query only to find out in the last filter that there’s actually no data to insert isn’t the most efficient use of your warehouse, whereas state aware would have skipped the query entirely. Although this doesn’t eliminate increments, it helps teams be more efficient with their dbt runs, before getting into the nitty-gritty.

Cleanup Days and Fire Drills
If you’re lucky enough not to have done this, congrats. When bills start edging up and it’s getting closer to the end of a financial period, dbt spend can be suspect number one for interrogation.
This results in teams setting aside hours or even days to “reduce spend” by combing through their project, deciding what to run when, where an increment can go, how to rejig the scheduling, etc.
State aware orchestration provides a bunch of that out of the box, meaning dbt runs are just more inline with the amount of data you’re receiving, processing what you need, and ultimately leading to less time spent combing your IDE.

Defined Deliveries
Even when you’re running an hourly dbt project with increments everywhere, sometimes you just have models that need running at given times to meet the needs of the consumer, or conversely, you’re running a model all the time for that Tableau dashboard referenced in a meeting once.
By bringing SLAs into your dbt project, state aware orchestration limits the need for “special runs for special data” or needlessly refreshing the enormous table nobody uses, which brings more predictability to dbt’s delivery times, and ultimately to those who use your data.
On top of this, the data function now has a data point that provides an alignment with another part of the business, and that’s a win-win.

But this is just our opinion, the proof is in the pudding, so why not try it out for yourself and see how state aware orchestration benefits your team?!

How does it work

There is a worked example in the docs, but let’s look at some common dbt scenarios you may encounter in your pipelines, and how they would look with state aware orchestration enabled.

Common Scenarios

1. No changes — The easiest scenario, all models and sources are stale, seeds and snapshots still run, triggering the final_model and reusing the existing model data.

2. Source receives data and refreshes — A source refreshes and rebuilds its stage, but not all dim_orders upstreams refreshed, so it does not build.

Scenario 1: Nothing to transform

Scenario 2: Single source receives data

3. Model code changed — stg_customers had a code change, which may indicate new logic, so was rebuilt, along with the downstream final_model , dim_orders still skipped.

4. Orders source has new data, dim_orders within build_after SLA — The orders source received new data, but the inherited SLA of 1 hour from dim_orders tells stg_orders not to rebuild.

Scenario 3: Model code changed

Scenario 4: New data with inherited SLA

5. Data is stale, SLA breached — Over 1 hours has passed since dim_orders last ran, no new data has been received, but models are rebuilt based on inherited SLAs.

6. Seed file updated — Seeds always run, behaves as usual.

Scenario 5: Stale data but SLA has passed

Scenario 6: Seed file update

7. Seed file unchanged, final_model outside SLA — Nothing happened to the seed, but final_model has passed it’s 1 hour SLA, so is rebuilt.

8. Seed file updated, final_model inside SLA — The seed file was changed and rebuilt, but final_model is inside it’s 1 hour SLA, so is not rebuilt.

Scenario 7: Seed unchanged, final_model outside SLA

Scenario 8: Seed changed, final_model inside SLA

What about in Slim CI or with state:modified?

When running Slim CI or any state:modified style dbt command in Orchestra, you may be wondering how State Aware Orchestration might impact your builds?

Well, the short answer is, it doesn’t (mostly).

For the state comparison, Orchestra will still fetch the actual dbt artifacts from the run you wish to defer to, meaning the state comparison still happens in dbt all the same, and the difference as calculated as usual, so no updates are required here.

But, what if this is in CI, and we’re making commits after opening a change request?
As above, dbt will still be using the actual manifest.json to determine your changes, but when those are sent to built, Orchestra will compare the models from your branch to those it has a current state for, which will be those built from the previous commit, not production tables (as they’re stored with their full location, i.e. MY_CI_DB.PR_123.TABLE_NAME).

So, the above scenarios apply again, if the model code has changed, it will be rebuilt, otherwise Orchestra will try to reuse the data asset built in the previous CI run — there’s performance gains to be had there too!

Configuring it for yourself

Orchestra needs a few things from dbt to be able to run state aware orchestration, so let’s get those configured, this consists of:

• Source Freshness (Required)

• dbt Core Task Setting in Orchestra (Required)

• Model Build After (Optional)

Source freshness config (Required)

Check-out how to configure source freshness here.

Likely the one you’ve seen before, or perhaps already have in your dbt project, this lives in your sources YAML under a freshness block, and is used to monitor when your source tables were last updated, compared to your defined expectations.
For this, our options are:

loaded_at_field — the timestamp column dbt inspects.
Sits on the table config, not inside the freshness block. Typically a metadata column such as _fivetran_synced for ingestion tooling, or a datetime like created_at or order_date for source-system data.

warn_after / error_after — tolerances for freshness.
Each takes a count and a period (minute, hour, day). Both are optional independently — you can set just one. Setting only error_after skips the warning and fails immediately once past the threshold.

filter — a SQL predicate applied before dbt evaluates the max timestamp. Useful for excluding soft-deleted rows that would skew the reading, i.e. filter: _fivetran_deleted = false .

loaded_at_query — replaces the default freshness check with a fully custom query. Must return a timestamp in the expected format.

Metadata-based freshness — on Snowflake, Redshift, BigQuery, and Databricks, you can omit loaded_at_field entirely and let dbt infer freshness from the warehouse’s own table metadata. Platform-specific caveats apply

Full Example

So, to give a full example of this, we have a Fivetran-managed source syncing every 30 minutes:

sources:
  - name: my_source
    tables:
      - name: oil_data
        config:
          loaded_at_field: _fivetran_synced
          freshness:
            warn_after: {count: 30, period: minute}
            error_after: {count: 60, period: minute}
            filter: _fivetran_deleted = false

The thresholds here map directly to the ingestion schedule — one missed cycle warns, two missed cycles errors.
You can adjust these to match the actual SLA of the source: tighter for real-time feeds, looser for daily batch loads.
For non-critical sources, configuring only warn_after avoids hard failures.

Task setting on dbt Core (Required)

Once your dbt project has all the configuration it needs, it’s time to enable this on your Orchestra pipelines.
To do this:

• Navigate to a dbt Core Task in Pipeline Edit mode

• Open the Task Parameters window, and toggle on “Use state orchestration”

As simple as toggling it on, as it should be. YOU DO NOT NEED TO UPGRADE FROM DBT CORE TO ANOTHER PACKAGE.

That’s it!

Orchestra will now begin monitoring the state of the data assets in this task when the next run kicks off, and from then on, it’s a case of letting the magic go to work! ✨

build_after config (Optional)

To learn more about the build_after config, check out this article.

Source freshness covers your ingestion layer, but what about the models further down your DAG?That incremental staging model can run every 30 minutes just fine, but if mart_customers takes 5 minutes to build and the CRM team only pulls it twice a day, rebuilding it on every dbt run is just burning compute for nothing!

build_after lets you define model-level SLAs — a minimum time that must elapse before dbt considers rebuilding a model.
This sits within a model’s config.freshness block and takes three arguments:

count and period — the minimum time since the model was last built. period accepts minute, hour, or day.

updates_on — any or all. Controls whether the model rebuilds when any upstream dependency has new data, or only when all of them do.

The important thing to understand is that both conditions must be true simultaneously: the time window has elapsed and new upstream data is present. If either is false, the model sits tight and the existing table gets reused.
Let’s check out a worked example:

models:
  - name: stg_customers
    config:
      freshness:
        build_after:
          count: 2
          period: hour
          updates_on: any

  - name: mart_customers
      config:
        freshness:
          build_after:
            count: 6
            period: hour
            updates_on: all

stg_customers rebuilds after two hours as soon as any upstream has something new.
mart_customers waits six hours and requires all upstreams to be fresh — a reasonable trade-off for an expensive full refresh that’s only consumed a couple of times a day.

One thing to keep in mind: build_after configs tend to sit towards the right-hand side of the DAG, but they have implications moving left. If mart_customers needs fresh data every six hours, everything upstream of it needs to be feeding it data at least that often — otherwise the window will arrive with nothing new to process.

Combined with source freshness, this gives you SLAs at both ends of the pipeline — the ingestion layer and the model layer — which is the foundation state-aware orchestration is built on.

Managing state in Orchestra

Here is how we manage state in Orchestra (docs).

Once state orchestration has been enabled on a task within a workspace, Orchestra will begin tracking the data assets as they’re built.

Mostly, this is going to be for dbt’s benefit, but sometimes we may want to peek behind the curtain and see what’s going on.

The Orchestra State

Orchestra manages this state at the Workspace level

In each workspace, Orchestra stores a minimal state of when each asset was last updated, where each asset is represented by its qualified location, i.e. database.schema.table, in most cases, like below.

Last update times of “model” in production and staging

This allows the different dbt pipelines in the same workspace to have a unified understanding of the state of a dbt model, so Pipeline A won’t try and update a model Pipeline B finished 2 minutes ago, unless specified.

Resetting State

But what if we want to do the equivalent of a “full refresh”?

Well, we’d either need to get Orchestra to ignore the state, or delete the state itself, which is as follows:

1. Navigate to State Management in Orchestra, via Settings > Workspace > Asset management
   Here, you can see how many models are currently being tracked.

2. To reset state, click the Purge state button. The next run will rebuild all models.

Purging dbt state for a workspace

After doing this, any dbt pipelines in the workspace will begin rebuilding all of their models from scratch on the next run.
If this was just for one dbt pipeline in the workspace, it would be equivalent to running with --full-refresh .

Exploring dbt Metadata

Orchestra Assets already contains information from monitoring all your data assets, and state aware orchestration enriches that with further details about the asset state.

1. Navigate to Assets in Orchestra

2. Search for one a model, i.e. dim_orders

Which will lead to 2 tabs containing rich details about the asset, as in the display below.

Summary data for a table in a state aware dbt pipeline

Metrics tab — Analytics related to this asset, and the latest operations and pipeline runs linked to it, useful for quickly seeing when and how an asset was last materialised

Configuration tab — dbt-specific configuration collected for this asset, such as its freshness configuration, SQL, and associated tests

Extra Debug Logs

Sometimes it’s more useful to inspect a particular job or run, for this it’s possible to enable additional logging for debugging.

Enabling this is simple, just update your dbt task to run with the environment variable ORCHESTRA_DBT_DEBUG=true .

Monitoring the impact of state-aware orchestration

The impact of state-aware Orchestration can be put into three buckets.

1. Fresher Data

2. Saved costs

3. Saved time

Let’s deal with these in turn.

Saved Time

Before State-Aware Orchestration

Let’s suppose you have a small dbt project that starts out as a single dbt build once a day.

Over time this turns into a thousand model behemoth with dozens of different tags. Taking an example below, if you want to change the schedule for the DEALS_DAILY_ATTIO table you will need to edit two schema.yml files and 5 models.

After State-Aware Orchestration

With State-Aware Orchestration in Orchestra, you only need to be make one code change.

sources:
  - name: raw
    schema: raw
    tables:
      - name: src_orders
        config:
          freshness:
            warn_after: { count: 24, period: hour }
          loaded_at_field: order_date
      - name: src_customers

sources:
  - name: raw
    schema: raw
    tables:
      - name: src_orders
        config:
          freshness:
            warn_after: { count: 24, period: hour }
          loaded_at_query: “select max(order_date) from table where order_date > dateadd(DAY, -7, current_date())”
      - name: src_customers

You should evaluate how long your team spends tweaking schedules and orchestrating dbt and assign a value to it.

Fresher Data

Before State-Aware Orchestration

dbt runs on separate schedules; some processes run daily and some may run hourly. This means that if anything lands in between the fifteen minute and hourly runs, data gets stale. If an hourly process fails, stakeholders need to wait another hour to get data.

After State-Aware Orchestration

dbt runs every fifteen minutes or perhaps even more frequently at a scheduling frequency. The data is processed as soon as it arrives. Transformations take time; each model has a processing duration (the time it takes to process the model). This means that the freshness lag equals the scheduling frequency + processing duration, greatly simplifying the formula for data freshness.

How Orchestra and Estuary make activation faster

In the example above, we can see that different data feeds from a real-time streaming too like Estuary result in data landing in a data warehouse environment. Rather than scheduling separate dbt jobs, dbt simply runs on a regular cadence, processing data as it arrives.

You should evaluate the benefits of fresher and more reliable data, as well as the potential gains to stakeholder trust from implementing SAO.

Reduced cost

We save the best until last! With Orchestra SAO, you can reduce data warehouse costs substantially in the correct conditions.

Best Case Scenario: many unused sources, many schedules, minimal SLAs.

Before State-Aware Orchestration, the company runs everything hourly. This results in 1,000 model builds an hour an 24,000 builds a day.

Minimal SLAs: It turns out that key models only need to be run daily. Build_after configurations are added to models. This reduces the number of hourly models from 1000 to 500.

Many schedules: data lands in the day at different times. By running everything hourly, everything gets picked up, but models are run unnecessarily. By moving to SAO, an additional 100 models are skipped in every hourly run.

Unused sources: it turns out that there are many sources that are not updated. These relate to an incremental 200 additional models every hour.

Result: the hourly job remains, but only 200 models run each time. Once a day, there is a a run of 500 models. This results in 23*200 + 500 = 5,100 model runs vs. 24,000 previously. Assuming an even distribution of models and serverless compute under-the-hood, this will reduce costs by over 75%

Worst Case Scenario: regular, reliable, infrequent data

Before SAO, data lands daily and regularly. It is reliable and almost never breaks. A single dbt build is run once a day, and it rarely fails. There are no plans to change the structuring or scheduling of dbt jobs.

After SAO, this will remain broadly the same. Adding in build_after configurations is unlikely to impact an already-infrequently running pipeline. Data is always fresh when it is processed, so this aspect does not change either.

However, what SAO opens up is the possibility of going faster. What would the Data Function be able to do if they didn’t need to promise only daily data? What if they could promise more or less real-time data?

How to find the reduced cost using the Metadata API

Using the Orchestra Metadata API, we can fetch and analyse the improvements in both the duration of tasks and warehouse cost savings. To get started, review our ‘blueprints’ GitHub repository, and follow the instructions in the README provided.

Reductions in cost from Orchestra SAO

Next Steps for SAO

Congratulations! You’ve completed our guide on SAO. You learned:

• What is State-Aware Orchestration and what does it do?

• Why State-Aware Orchestration matters for data teams

• How dbt State-Aware Orchestration works

• How to configure dbt State-Aware Orchestration locally

• How to configure dbt State-Aware Orchestration in Orchestra

• Managing state in Orchestra

• Monitoring the impact of State-Aware Orchestration

In terms of helpful resources, we recommend getting started with what’s below:

• Create a Free Account with SAO Now ⭐️

• Read the Documentation 📚

• Follow us on Linkedin 👴

We can’t wait to see what you build!

Orchestra Team

Introduction

Today we’re excited to announce the official launch of State-Aware Orchestration (“SAO”) in Orchestra.

SAO is an Orchestra-managed python package built for dbt Core™ that allows users to store and use the state of models. This helps to orchestrate dbt Core™ more intelligently and only run models when certain conditions are met.

With Orchestra SAO, Orchestra users can:

• Save cost and speed-up runs: the state of dbt models is stored using source_freshness checks which means models are skipped if there is no new data and no code has changed

• Stop wasting time tagging models: adding build_after configurations allow for declarative scheduling. There is no need to manual tagging

• Move to real-time: dbt state-aware can run every 10 or 5 minutes, enabling close-to-realtime patterns with leading ELT tools like Estuary

Users of dbt do not need to upgrade to a proprietary engine. Orchestra leverages the Sao Paolo repo. Simply enable Orchestra SAO with one line of code:

tasks:
      execute_dbt:
        integration: DBT_CORE
        integration_job: DBT_CORE_EXECUTE
        parameters:
          commands: dbt  build
          package_manager: PIP
          python_version: ‘3.12’
          use_state_orchestration: true // <-- this line
.
.
.

Some of the biggest and most advanced data teams are already leveraging State-Aware Orchestra in Orchesta. Experian use Orchestra State-Aware Orchestration to power dozens of analysts across multiple domains, to easily build SQL models powering marketing analytics.

The largest energy provider in the UK, Octopus Energy, used to maintain a custom fork of Airflow 2.0. Rather than spend resources upgrading to 3.0 and training their 10+ business units to learn an antiquated python framework, they leverage Orchestra SAO on Databricks to easily power a multi-tenanted project with over 5,000 dbt models.

We can’t wait to see what you build! To learn more, check-out the resources below. While SAO is GA, we need to turn it on for you. So if you want to try it, message us at support@getorchestra.io.

Video Resources

📚 Docs

🚀 Try State-Aware Orchestration

⭐️ Learn more about Orchestra

Awesome Blogs

• Configuring dbt Source Freshness (read here)

• Configuring the build_after configurations (read here)

The Problem with dbt

I remember my first dbt run I really do. It was a magical thing. Within the week I had alread submitted multiple queries with multiple union statements and cost the analytics team $1053.21 dollars in bigquery — which I was swiftly reprimanded for.

You see, the Problem with dbt is that it is too good. It makes things too easy. Indeed, “dbt spaghetti” is a common term used to describe the lineage people see.

I used to believe this was down to poor domain design, and poor organisational practices. However, as I have started working more and more with fast-moving organisations at scale — I have realised many of these things can sometimes be inevitable.

Therefore, dbt SAO in Orchestra is a way to solve many problems at once.

The Problem with dbt Project Growth

dbt Project GRowth becomes complicated as organisations grow:

• SAAS migrations: when organisations change SAAS tools like CRMs, Analytics Folks need to remember to remove and turn off old models

• First versions: when Data Teams are building the first versions of data products, models may swiftly become out of date as requirements change quickly. Old models get deprecated

• High turnover: companies may turn over staff quickly, leading to a lack of documentation and scheduling awareness of dbt models

This means that keeping track of models that need to run and models that don’t.

The Problem with dbt Scheduling

As dbt projects get larger, people have different data needs. For example yu may run dbt on three schedules

• Business Critical: dbt run --select models tag:fifteen_minutes

• Important: dbt run --select models tag:hourly

• Daily: dbt run --select models tag:daily

This becomes difficult to maintain. What if someone adds a new model that should be daily? What if it changes to Every fifteen minutes? You must remember to change the schema.yml files for all upstream dependencies and manually add those. This is arduous, time-consuming and annoying

The Problem with dbt cost

What happens if you do not have well-architected, reliable pipelines? Data sometimes arrives, but sometimes it doesn’t. dbt is not aware of whether or not data has arrived.

While end-to-end Orchestration solves some of this problem (Orchestra only triggers dbt if all upstream processes succeed) it may not be suitable for larger organisations with multiple teams.

For example, what if the Data Engineering Team look after landing data, while the Analytics Team look after it in the warehouse? What if there is a failure?

The result is many models are re-run despite there being no new data. This makes projects run slower, and become more expensive

The solution: dbt State-Aware Orchestration in Orchestra

Orchestra’s dbt Core™ integration allows users to store the state of models from their DAGs, allowing for more optimised runs that avoid rebuilding models that have no upstream changes.

This is very useful for large workflows, and for organisations where additional configuration can be added to generate further warehouse cost savings.A standard dbt Core™ run will not take into account the last time upstream data was refreshed, or when the models were last built.

This means that if a model has not changed, and the upstream data is stale, it will be rebuilt unnecessarily.​

Features

• Orchestra dbt Core™ runs will load in the previous known state of models in the DAG.

• Users can define ‘freshness’ configurations on models, and have them propagated to parent models.

• Orchestra will only rebuild models that have been edited since the last run, or have fresh upstream data (respecting any additional configuration in the model defined by the user).

• Models that are reused will be clearly highlighted in the resulting lineage view, in the logs, and in the analytics pages.

• The state will be updated after the dbt Core™ run has successfully completed. This state can be reset by users if required via the Orchestra UI. Only models built successfully will be updated.

• State is managed at the Orchestra workspace level. For details on how state is stored and managed, see How state is stored.​

How state is fetched

Stateful reuse uses dbt source freshness results. When a source defines loaded_at_field or loaded_at_query, dbt’s normal freshness logic runs on every adapter Orchestra supports through dbt Core™.When both are omitted, Orchestra can still run adapter-specific SQL to infer max_loaded_at for some warehouses. Only the adapters below register that path today.

Warehouse Adapters

For adapters without a registered fallback, if both loaded_at settings are missing, Orchestra follows dbt’s FreshnessRunner behavior (which may surface as warnings or a non-actionable result depending on dbt and the warehouse).State is fetched for the entire project on every run to keep the state cache fresh.​

How state is stored

State is managed at the Orchestra workspace level — a single state store is shared across all pipelines and dbt Core™ tasks within a workspace. This means that if the same model is built by multiple pipelines in the same workspace, they all read from and write to the same state.Within that store, state is tracked at the asset level, using the fully-qualified name of each materialised model. For example:

An example ofthe State Table

Two tasks writing to different databases or schemas are tracked independently, even if they build the same dbt models. This is how Orchestra distinguishes between environments.

Tables should not be shared across environmentsBecause state is keyed by fully-qualified table name, each environment must write to its own distinct tables. If a production and a staging task both materialise models to the same database and schema (e.g. both writing to db.schema.model), they share a single state entry — and a run in one environment could incorrectly skip a rebuild in the other.​

Orchestra State-Aware Orchestration Example

Consider the DAG below with the following items:

• sources: orders and customers.

• seeds: locations.

• models: stg_orders, stg_customers, dim_orders, and final_model. There are some linked tests on selected models.

• snapshots: snapshot_x.

On a standard dbt Core™ run, all of the nodes will run as usual. Once state management is enabled for the task, it is possible to begin reusing models. Currently, seeds, snapshots, and tests will always run regardless of any upstream changes. The graph below shows some example configs attached to the models.

Visualisation of the DAG outlined above.

Based on the configs in the diagram, the following table illustrates which models get rebuilt based on different scenarios:

The result of State

​

Setting up in SAO in Orchestra for running dbt

Detailed tutorial
For a step by step guide to setting up and using state management in Orchestra, please see our tutorial.

To configure this in Orchestra’s dbt Core™ integration:

1. Go to any dbt Core™ task in any Orchestra pipeline

2. Open the task parameters, and enable Use state orchestration

You may wish to update your dbt™ code to enhance the capabilities of this system. For example, additional configuration can be defined on any model. Likewise, source freshness accuracy depends on the warehouse being utilised, and if any freshness config has been applied or not.

By default, if no config is added to a source, Orchestra assumes it always has new data.If you are seeing any issues during a run, you can enable additional debug logging by setting the ORCHESTRA_DBT_DEBUG environment variable to the string true, or set debug = true under [tool.orchestra_dbt] in pyproject.toml when using the dbt-orchestra CLI.​

Analysing warehouse savings

Warehouse savings analysis We are looking to natively provide this information in Orchestra in Q1 2026. Please use the information below in the meantime to analyse your metadata.

Using the Orchestra Metadata API, we can fetch and analyse the improvements in both the duration of tasks and warehouse cost savings. To get started, review our ‘blueprints’ GitHub repository, and follow the instructions in the README provided.

Notice the decrease in total duration of operations for a task run when models are reused.​

Viewing and resetting state

If there are some issues running dbt Core™ with state orchestration, such as models not being rebuilt when they should be, you can reset the state. Currently, this can only be done at the workspace level, but per model support should be introduced by Q1 2026. Click the Purge state button when viewing state management.

dbt Core, dbt and dbt Labs are all Trademarks of dbt Labs, Inc.

I cover this in the video but I think Fivetran is far from cooked and there are sound technical reasons for this.

1. The scripts are hard, but not that hard

Writing good ELT means you need a pattern like the below, at a minimum, at scale:

from datetime import date
from typing import Literal, Dict, Any, Optional


def extract_from_api(
    endpoint: str,
    start_date: date,
    end_date: date,
    params: Optional[Dict[str, Any]] = None
) -> Dict[str, Any]:
    """
    Dummy API extraction function.
    """
    return {
        "endpoint": endpoint,
        "start_date": str(start_date),
        "end_date": str(end_date),
        "params": params or {},
        "records": [
            {"id": 1, "value": "example"},
            {"id": 2, "value": "example"}
        ]
    }


def build_s3_file_path(
    bucket: str,
    prefix: str,
    file_path_type: Literal["date_partitioned", "flat", "custom"],
    start_date: date,
    custom_path: Optional[str] = None
) -> str:
    """
    Builds destination S3 file path.
    """
    if file_path_type == "date_partitioned":
        return f"s3://{bucket}/{prefix}/year={start_date.year}/month={start_date.month}/day={start_date.day}/data.json"

    if file_path_type == "flat":
        return f"s3://{bucket}/{prefix}/data.json"

    if file_path_type == "custom" and custom_path:
        return f"s3://{bucket}/{custom_path}"

    raise ValueError("Invalid file_path_type or missing custom_path")


def load_to_s3_staging(
    data: Dict[str, Any],
    s3_path: str,
    write_mode: Literal["incremental", "full_refresh"]
) -> None:
    """
    Dummy S3 load function.
    """
    if write_mode == "full_refresh":
        print(f"Deleting existing data at {s3_path}")
        print(f"Writing full refresh data to {s3_path}")

    elif write_mode == "incremental":
        print(f"Appending incremental data to {s3_path}")

    else:
        raise ValueError("write_mode must be incremental or full_refresh")

    print(f"Loaded {len(data['records'])} records")


def run_api_to_s3_staging_elt(
    endpoint: str,
    bucket: str,
    prefix: str,
    start_date: date,
    end_date: date,
    arbitrary_json: Dict[str, Any],
    refresh_type: Literal["incremental", "full_refresh"],
    file_path_type: Literal["date_partitioned", "flat", "custom"],
    custom_path: Optional[str] = None
) -> None:
    """
    Main ELT orchestration function:
    API → S3 staging area
    """

    data = extract_from_api(
        endpoint=endpoint,
        start_date=start_date,
        end_date=end_date,
        params=arbitrary_json
    )

    s3_path = build_s3_file_path(
        bucket=bucket,
        prefix=prefix,
        file_path_type=file_path_type,
        start_date=start_date,
        custom_path=custom_path
    )

    load_to_s3_staging(
        data=data,
        s3_path=s3_path,
        write_mode=refresh_type
    )


# Example usage
run_api_to_s3_staging_elt(
    endpoint="https://api.example.com/orders",
    bucket="my-staging-bucket",
    prefix="staging/orders",
    start_date=date(2025, 1, 1),
    end_date=date(2025, 1, 31),
    arbitrary_json={
        "region": "EU",
        "status": ["completed", "refunded"],
        "include_metadata": True
    },
    refresh_type="incremental",
    file_path_type="date_partitioned"
)

This is actually quite hard to do and hard to manage. It is not as simple as “AI one-shots this and you’re done”. AI can perhaps one-shot the logic, but you need to maintain the framework, and the UI.

2. Infrastructure

Moving data takes a lot of infrastructure! not everyone wants to manage it

3. CDC

Fivetran does CDC too (like all the other providers) which again is not just a python script AI can one-shot

4. Destination types

Adding types of destination like iceberg is non-trivial. There is basically a new connector for every iceberg catalog you use.

5. Networking permutations

This is the real killer. There are so many options

1. Fivetran all in cloud

2. Fivetran control plane in cloud, worker in your network

3. [2] but the network is a VPC so you connect via Private Link

   1. x3 for GCP, AWS, and Azure

4. on-premise deployment with control planein the cloud

5. on-premise air-gapped

and then again, the destinations themselves may also have similar locations. Which creates quite a lot of permutations

Massive pain to configure yourself…

Conclusion: moving data is here to stay!

I believe moving data is here to stay.

However - there are signs that Fivetran and the old pattern could be disrupted

1. Clickhouse offers a native suite of tools like Clickhouse Postrges and Clickhouse joined via ClickPipe that basically means all the ELT is handled for you. We leverage this at

2. Deveoper-first: people do not want to have to spin up a SAAS tool to sync data between places. It would be much nicer if there was just a package that could be run “somewhere” that moved data from place to place. The thing I’ve seen that is closest to this is dlthub, though I wonder if there is yet an even smoother abstraction for moving data that could exist

Beyond this we do not believe AI can disrupt players like Fivetran and that managed ELT remains one of the most cost-effective ways for data teams to buy back their time.

To read more about how to manage Fivetran syncs from Orchestra, check-out our Fivetran page here.

I’ve been astonished at how many people have built overly complicated metadata-driven frameworks over the years. Unfortunately, these form the foundations of all the data in the platform, and mean that accessing data and easily identifying what data is fresh is impossible.

Despite this, some CDOs still insist on chasing AI despite not laying the foundations for it - a tale as old as time.

We recently added support for sequential tasks in the MetaEngine. It might seem like a small thing but it allows for parallelisation of thousands of data processing tasks at once. This is super powerful, not just data data pipelining but also AI use-cases.

In this article we cover some approaches to making the most of this feature in Snowflake. Hopefully it gives an insight into how simple things has become - and provides a deterrent to anyone considering using an “Accelerator”!

Introduction

Having a simple set of pipelines works until it doesn’t. Managing a small sample set of sources and destinations is something a one or two person data team deal with comfortably. It’s scale that changes this.

When hundreds of tables become thousands, a “Lean team” operating “super efficiently” suddenly becomes “tech debt” and “tribal knowledge”. Work leads to cancelled vacations which leads to burn-out. Eventually, Kevin isn’t on holiday — Kevin’s quit his job.

One approach to curtail some of this from the technical angle are Metadata driven frameworks.

In a metadata-driven framework, modular components are assembled that are parameterised at runtime. Instead of having a pipeline for moving table_a and a pipeline for moving table_b , there is a single pipeline capable of moving both tables. At runtime, the pipeline is parameterised with both table_a and table_b , and is therefore metadata-driven.

However, metadata-driven frameworks are frought with complexity. Many consultants have been paid millions of dollars to implement layers, and layers of metadata. Typically this involves Azure Data Factory pipelines calling pipelines, which in turn call more pipelines and more pipelines.

This endless sea of recursion is the enemy of metadata-driven frameworks, as such patterns are complex to build, maintain, and unravel. They are emblematic of the “Fortress” that the “Data person” may seek to build around themselves.

In this article, I’ll walk through a practical example of building the ultimate metadata-driven ELT Pipeline on Snowflake.

Step 1: Identify your sources

The first step involves identifying your sources and identifying common data ingestion patterns. As a rule of thumb, one API means one script. Where APIs are complex, you may have a separate script for the API (for example if one set of endpoints supports incrementality but another doesn’t).

There is always a tradeoff between abstraction and complexity. Of course, you could have one “Master script” that triggers any job.

This is illustrated above. On the far left hand side you have a completely abstracted entrypoint that accepts a wide range of parameters. Depending on the parameters, it runs different scripts. These scripts could range from pulling incremental structure data from an API to pulling unstructured one-off .wav files from an SFTP Server.

• Complex

• Very scalable

The far right example is the opposite. There is no abstraction at all, and no metadata drives the pipelines. There is a pipeline for every script.

• Not complex and easy to edit code on a case-by-case basis

• Can be difficult to scale / standardise

The middle example is the best. There is a set of common scripts that understand how to move data for different sources and types of data. For example, one for SQL Server, one for MYSQL and one for the Workday API.

These individual Tasks can be parameterised with different things like the table or object names.

• Balanced level of complexity

• Very scalable and less dependencies between jobs

Step 2 identify your destination

For Snowflake you should basically choose one of S3, GCS or Azure Data Lake Storage.

Writing in Iceberg format is not a bad idea either, though it is not required.

Example

Writing these types of scripts is easy now. Just use a prompt like this:

Write me a basic script that takes data from workday and drops it in S3 in python. The script should be parameterisable for the following parameters
1. Object name or API endpoint
2. The date from and date to
3. Should handle pagination
4. S3 bucket syntax should be /workday/object/date_from_date_to.json
5. Should acept a param for full or incremental loading.

Workday was not a good example. The API seems quite gross. but:

import os
import json
import argparse
from typing import Any, Dict, List, Optional

import requests
import boto3


def parse_args() -> argparse.Namespace:
    parser = argparse.ArgumentParser(description=”Extract data from Workday API and load to S3”)

    parser.add_argument(”--base-url”, required=True, help=”Workday API base URL”)
    parser.add_argument(”--object-name”, required=True, help=”Logical object name, e.g. workers”)
    parser.add_argument(
        “--endpoint”,
        required=True,
        help=”API endpoint path, e.g. /workers or /customreport2/tenant/report”
    )
    parser.add_argument(”--date-from”, required=True, help=”Start date in YYYY-MM-DD format”)
    parser.add_argument(”--date-to”, required=True, help=”End date in YYYY-MM-DD format”)
    parser.add_argument(
        “--load-type”,
        required=True,
        choices=[”full”, “incremental”],
        help=”Whether to run a full or incremental load”
    )
    parser.add_argument(”--s3-bucket”, required=True, help=”Target S3 bucket name”)

    # Auth parameters
    parser.add_argument(”--auth-type”, default=”bearer”, choices=[”bearer”, “basic”])
    parser.add_argument(”--token”, default=os.getenv(”WORKDAY_TOKEN”), help=”Bearer token”)
    parser.add_argument(”--username”, default=os.getenv(”WORKDAY_USERNAME”), help=”Basic auth username”)
    parser.add_argument(”--password”, default=os.getenv(”WORKDAY_PASSWORD”), help=”Basic auth password”)

    # Optional pagination tuning
    parser.add_argument(”--page-size”, type=int, default=100, help=”Number of records per page”)
    parser.add_argument(”--timeout”, type=int, default=60, help=”HTTP timeout in seconds”)

    return parser.parse_args()


def build_auth(args: argparse.Namespace):
    if args.auth_type == “bearer”:
        if not args.token:
            raise ValueError(”Bearer auth selected but no token provided”)
        return None, {”Authorization”: f”Bearer {args.token}”}

    if args.auth_type == “basic”:
        if not args.username or not args.password:
            raise ValueError(”Basic auth selected but username/password not provided”)
        return (args.username, args.password), {}

    raise ValueError(f”Unsupported auth type: {args.auth_type}”)


def build_initial_params(args: argparse.Namespace) -> Dict[str, Any]:
    “”“
    Generic params.
    You may need to rename these to match your Workday endpoint,
    for example updatedFrom / updatedTo or effectiveDate, etc.
    “”“
    params = {
        “limit”: args.page_size
    }

    if args.load_type == “incremental”:
        params[”dateFrom”] = args.date_from
        params[”dateTo”] = args.date_to

    return params


def extract_records_from_response(payload: Dict[str, Any]) -> List[Dict[str, Any]]:
    “”“
    Tries common response structures.
    Adjust this for your specific Workday response shape.
    “”“
    if isinstance(payload, list):
        return payload

    for key in [”data”, “items”, “records”, “results”]:
        if key in payload and isinstance(payload[key], list):
            return payload[key]

    return []


def get_next_page_info(payload: Dict[str, Any]) -> Optional[Dict[str, Any]]:
    “”“
    Generic pagination handler.

    Supports:
    - next page URL in payload[’next’]
    - next page token in payload[’nextPage’]
    - offset/total/limit style paging

    Adjust as needed for your Workday endpoint.
    “”“
    if not isinstance(payload, dict):
        return None

    if payload.get(”next”):
        return {”type”: “url”, “value”: payload[”next”]}

    if payload.get(”nextPage”):
        return {”type”: “token”, “value”: payload[”nextPage”]}

    if all(k in payload for k in [”offset”, “total”, “limit”]):
        offset = int(payload[”offset”])
        total = int(payload[”total”])
        limit = int(payload[”limit”])

        next_offset = offset + limit
        if next_offset < total:
            return {”type”: “offset”, “value”: next_offset}

    return None


def fetch_all_pages(args: argparse.Namespace) -> List[Dict[str, Any]]:
    auth, extra_headers = build_auth(args)

    headers = {
        “Accept”: “application/json”,
        **extra_headers
    }

    base_endpoint = f”{args.base_url.rstrip(’/’)}/{args.endpoint.lstrip(’/’)}”
    params = build_initial_params(args)

    all_records: List[Dict[str, Any]] = []
    next_request_url: Optional[str] = base_endpoint
    next_request_params: Optional[Dict[str, Any]] = params.copy()

    while next_request_url:
        response = requests.get(
            next_request_url,
            headers=headers,
            auth=auth,
            params=next_request_params,
            timeout=args.timeout
        )
        response.raise_for_status()

        payload = response.json()
        records = extract_records_from_response(payload)
        all_records.extend(records)

        next_page = get_next_page_info(payload)

        if not next_page:
            break

        if next_page[”type”] == “url”:
            next_request_url = next_page[”value”]
            next_request_params = None

        elif next_page[”type”] == “token”:
            next_request_url = base_endpoint
            next_request_params = params.copy()
            next_request_params[”pageToken”] = next_page[”value”]

        elif next_page[”type”] == “offset”:
            next_request_url = base_endpoint
            next_request_params = params.copy()
            next_request_params[”offset”] = next_page[”value”]

        else:
            break

    return all_records


def upload_to_s3(bucket: str, object_name: str, date_from: str, date_to: str, records: List[Dict[str, Any]]) -> str:
    s3 = boto3.client(”s3”)

    key = f”workday/{object_name}/{date_from}_{date_to}.json”
    body = json.dumps(records, indent=2).encode(”utf-8”)

    s3.put_object(
        Bucket=bucket,
        Key=key,
        Body=body,
        ContentType=”application/json”
    )

    return f”s3://{bucket}/{key}”


def main() -> None:
    args = parse_args()

    records = fetch_all_pages(args)
    s3_path = upload_to_s3(
        bucket=args.s3_bucket,
        object_name=args.object_name,
        date_from=args.date_from,
        date_to=args.date_to,
        records=records
    )

    print(f”Loaded {len(records)} records to {s3_path}”)


if __name__ == “__main__”:
    main()

With example usage:

python workday_to_s3.py \
  --base-url https://wd2-impl-services1.workday.com/ccx/api/v1/your_tenant \
  --object-name workers \
  --endpoint /workers \
  --date-from 2026-04-01 \
  --date-to 2026-04-20 \
  --load-type incremental \
  --s3-bucket my-raw-bucket \
  --auth-type bearer \
  --token YOUR_WORKDAY_TOKEN

The alternative here would be to have those parameters above set as environment variables in the code.

Step 3: Create and set-up an external stage

You can actually wrap the entire creation and incremental execution in a single set of statements.

USE DATABASE WORKDAY;
USE SCHEMA RAW;

-- =========================================================
-- File format
-- =========================================================
CREATE FILE FORMAT IF NOT EXISTS WORKDAY_JSON_FF
  TYPE = JSON
  STRIP_OUTER_ARRAY = TRUE;

-- =========================================================
-- External stage
-- =========================================================
CREATE STAGE IF NOT EXISTS WORKDAY_WORKERS_STAGE
  URL = ‘s3://my-raw-bucket/workday/workers/’
  STORAGE_INTEGRATION = MY_S3_INTEGRATION
  FILE_FORMAT = WORKDAY_JSON_FF;

-- =========================================================
-- Table
-- =========================================================
CREATE TABLE IF NOT EXISTS WORKDAY.RAW.WORKERS
(
    RAW_PAYLOAD VARIANT,
    SOURCE_FILE STRING,
    LOAD_TYPE   STRING,
    DATE_FROM   DATE,
    DATE_TO     DATE,
    LOADED_AT   TIMESTAMP_NTZ DEFAULT CURRENT_TIMESTAMP()
);

-- =========================================================
-- Load
-- =========================================================
COPY INTO WORKDAY.RAW.WORKERS
  (RAW_PAYLOAD, SOURCE_FILE, LOAD_TYPE, DATE_FROM, DATE_TO)
FROM
(
  SELECT
    $1,
    METADATA$FILENAME,
    ‘incremental’,
    TO_DATE(’2026-04-01’),
    TO_DATE(’2026-04-20’)
  FROM @WORKDAY_WORKERS_STAGE
)
FILES = (’2026-04-01_2026-04-20.json’)
ON_ERROR = ‘ABORT_STATEMENT’;

This will load the data into the stage. The above should be parameterised, obviously

Step 4: Test the data

The next step should be to test the data. You can confirm there are at least >0 rows in the table. This is a straightforward check.

Select * from table
where date_from = “<date_from>”
and date_to= “<date_to>”

Optional Step — cleaning transformations

Every raw table may want to run a view that cleans the data. For example, something like:

-- table_a_clean.sql

select
badly_named____column as badly_named_column,
cast(timestamp_column as date) as data_column,
case when bad_values = ‘good’ as ‘bad’ else bad_values end as good_values
from table

Every model should ideally have a standard file name to ensure consistency. You could even wrap this as a python task, or perhaps a stored procedure or Snowflake Task called table_name_clean_task or similar.

The flow is summarised in this nice diagram above.

Step 5: Data Transformations

Now that the data is landing, it’s time to transform data and break the metadata-driven approach in favour of something more declarative.

When there are many tables with multiple dependencies, managing these dependencies can be hard.

Furthermore, what happens when different tables land at different times, but dependencies depend on multiple pieces of data? At some point — the linearity of metadata-driven frameworks breaks.

At this point, you should leverage a state-orchestration framework. A state-orchestration framework only runs data when new data arrives. This means it can be run continuously and relied upon to update the data that is required.

This could ideally run as a separate pipeline, or within the same pipeline. Sensors can be enabled in between the data landing and transformation steps, depending on how quickly data needs to land.

Step 6: piecing it together in your orchestrator

This is perhaps the hardest part if you’re not using a declarative Orchestrator, since piecing this together in a tool like ADF Is a fool’s errand — the reason “Databricks Accelerators” are so expensive and time-consuming.

However if you are, the process is really quite simple. Here is an example using Orchestra:

version: v1
name: ‘#snowflake #dlt #matrix’
pipeline:
  3459fb19-f081-43ca-8997-f7f2d54d331d:
    tasks:
      c581252f-2975-4af6-8caf-a42e30527088:
        integration: PYTHON
        integration_job: PYTHON_EXECUTE_SCRIPT
        parameters:
          package_manager: PIP
          python_version: ‘3.12’
          build_command: pip install -r requirements.txt
          environment_variables: ‘{

            “SHEET_NAME”: “${{ MATRIX.gsheets[’‘SHEET_NAME’‘] }}”,


            “TABLE_NAME”:”${{ MATRIX.gsheets[’‘TABLE_NAME’‘] }}”,

            “RANGE_NAME”:”dlt_range”


            }’
          set_outputs: true
          source: GIT
          command: python -m run_dlt_pipelines_snowflake
          project_dir: dlt
          shallow_clone_dirs: dlt
        depends_on: []
        name: Execute parallel dlt
        connection: python__production__blueprints__19239
        treat_failure_as_warning: true
      00a86786-65c4-48df-8074-a2dfdd4e7dbb:
        integration: SNOWFLAKE
        integration_job: SNOWFLAKE_RUN_TEST
        parameters:
          statement: ‘select * from ${{ MATRIX.gsheets[’‘TABLE_NAME’‘] }}

            limit 1’
          error_threshold_expression: <0
          warn_threshold_expression: <1
          database: BIGQUERY_SAMPLE
          schema: PUBLIC
        depends_on:
        - c581252f-2975-4af6-8caf-a42e30527088
        name: Check data
        connection: snowflake__prod__80122
        treat_failure_as_warning: true
      2eb82923-5892-4c14-921d-a8a85afde545:
        integration: SNOWFLAKE
        integration_job: SNOWFLAKE_RUN_QUERY
        parameters:
          statement: execute task ${{ MATRIX.gsheets[’TABLE_NAME’] }}_TASK;
          set_outputs: false
        depends_on:
        - 00a86786-65c4-48df-8074-a2dfdd4e7dbb
        name: Insert Data
        treat_failure_as_warning: true
    depends_on: []
    name: ‘’
    matrix:
      inputs:
        gsheets:
        - SHEET_NAME: 1H0UnZ1vJ6WSsZgiVkg96zq52p7qaXkhvodlO1Mzoj6s
          TABLE_NAME: dbt_leads
        - SHEET_NAME: 1TMNhQFZbmaBsa1vIehQzh3vdyfc05damwJ_PIKDF14A
          TABLE_NAME: social_leads
        - SHEET_NAME: 1tImqLq_zLNthL7DDUkTUjU0xHoV_PhdM1zpuJ-TB0kk
          TABLE_NAME: unstructured_feedback
  e3e7d5bf-8c8c-400e-94d8-28a584fc638e:
    tasks:
      0f480822-0044-49b5-9635-eb9987978eab:
        integration: DBT_CORE
        integration_job: DBT_CORE_EXECUTE
        parameters:
          commands: dbt build
          package_manager: PIP
          python_version: ‘3.12’
          use_state_orchestration: true
          project_dir: dbt_projects/snowflake
          shallow_clone_dirs: dbt_projects/snowflake
        depends_on: []
        name: run dbt
        connection: dbt_snowflake_blueprints_prod_07025
    depends_on:
    - 3459fb19-f081-43ca-8997-f7f2d54d331d
    name: group
schedule:
- name: Daily at 8am
  cron: 0 8 ? * * *
  timezone: Europe/London
webhook:
  enabled: false
inputs:
  dbt_command:
    type: string
    default: build
  dbt_branch:
    type: string
    default: main
alerts:
- name: Failures
  statuses:
  - FAILED
  destinations:
  - integration: SLACK
    destination: alert-testing

This is less than 100 lines of code and event accounts for things such as alerting.

This is what it looks like when it runs:

This an incredibly simple but an incredibly powerful way to structure your data pipelines on Snowflake. Notice how all the business logic for moving data, transforming etc. is completely separate from the orchestration code, meaning domain experts can do what they do best without getting bogged down in complex tools like Apache Airflow or other python based orchestrators.

Step 7: Store data in a database (Bonus)

What happens if your database configurations are too large to be stored in a .yml file?

No problem! You can store the data in a database and pass these to your metadata framework instead.

The code will get shorter, as all the configs are stored elsewhere in a Snowflake table.

version: v1
name: ‘#snowflake #dlt #matrix’
pipeline:
  3459fb19-f081-43ca-8997-f7f2d54d331d:
    tasks:
      c581252f-2975-4af6-8caf-a42e30527088:
        integration: PYTHON
        integration_job: PYTHON_EXECUTE_SCRIPT
        parameters:
          package_manager: PIP
          python_version: ‘3.12’
          build_command: pip install -r requirements.txt
          environment_variables: ‘{

            “SHEET_NAME”: “${{ MATRIX.gsheets[’‘SHEET_NAME’‘] }}”,


            “TABLE_NAME”:”${{ MATRIX.gsheets[’‘TABLE_NAME’‘] }}”,

            “RANGE_NAME”:”dlt_range”


            }’
          set_outputs: true
          source: GIT
          command: python -m run_dlt_pipelines_snowflake
          project_dir: dlt
          shallow_clone_dirs: dlt
        depends_on: []
        name: Execute parallel dlt
        connection: python__production__blueprints__19239
        treat_failure_as_warning: true
      00a86786-65c4-48df-8074-a2dfdd4e7dbb:
        integration: SNOWFLAKE
        integration_job: SNOWFLAKE_RUN_TEST
        parameters:
          statement: ‘select * from ${{ MATRIX.gsheets[’‘TABLE_NAME’‘] }}

            limit 1’
          error_threshold_expression: <0
          warn_threshold_expression: <1
          database: BIGQUERY_SAMPLE
          schema: PUBLIC
        depends_on:
        - c581252f-2975-4af6-8caf-a42e30527088
        name: Check data
        connection: snowflake__prod__80122
        treat_failure_as_warning: true
      2eb82923-5892-4c14-921d-a8a85afde545:
        integration: SNOWFLAKE
        integration_job: SNOWFLAKE_RUN_QUERY
        parameters:
          statement: execute task ${{ MATRIX.gsheets[’TABLE_NAME’] }}_TASK;
          set_outputs: false
        depends_on:
        - 00a86786-65c4-48df-8074-a2dfdd4e7dbb
        name: Insert Data
        treat_failure_as_warning: true
    depends_on: []
    name: ‘’
    matrix:
      inputs:
        gsheets:
        - SHEET_NAME: 1H0UnZ1vJ6WSsZgiVkg96zq52p7qaXkhvodlO1Mzoj6s
          TABLE_NAME: dbt_leads
        - SHEET_NAME: 1TMNhQFZbmaBsa1vIehQzh3vdyfc05damwJ_PIKDF14A
          TABLE_NAME: social_leads
        - SHEET_NAME: 1tImqLq_zLNthL7DDUkTUjU0xHoV_PhdM1zpuJ-TB0kk
          TABLE_NAME: unstructured_feedback
  e3e7d5bf-8c8c-400e-94d8-28a584fc638e:
    tasks:
      0f480822-0044-49b5-9635-eb9987978eab:
        integration: DBT_CORE
        integration_job: DBT_CORE_EXECUTE
        parameters:
          commands: dbt build
          package_manager: PIP
          python_version: ‘3.12’
          use_state_orchestration: true
          project_dir: dbt_projects/snowflake
          shallow_clone_dirs: dbt_projects/snowflake
        depends_on: []
        name: run dbt
        connection: dbt_snowflake_blueprints_prod_07025
    depends_on:
    - 3459fb19-f081-43ca-8997-f7f2d54d331d
    name: group
schedule:
- name: Daily at 8am
  cron: 0 8 ? * * *
  timezone: Europe/London
webhook:
  enabled: false
inputs:
  dbt_command:
    type: string
    default: build
  dbt_branch:
    type: string
    default: main
alerts:
- name: Failures
  statuses:
  - FAILED
  destinations:
  - integration: SLACK
    destination: alert-testing

Conclusion: Metadata Frameworks on Snowflake easier to use than ever before

It is worth noting that throughout this process, I didn’t actually write a single line of code. I just got AI to do it for me.

Everything must be code in the age of AI, which means that having orchestration pipelines as code as well opens up a ton of new possibilities. When paired with a simple declarative framework built for scale like a MetaEngine, this is a breeze for AI to take control of.

This also avoids complex set-ups with multiple layers of nesting that data engineers struggle to break free from — the only things Databricks accelerators do is accelerate technical debt!

This pattern of load → object storage → test → possible clean → transformation is the ultimate metadata-driven approach on Snowflake.

It provides for any number of patterns. It is clean, uniform, has the perfect level of abstraction, and above all — it scales. This is not a system you implement that is complex to understand. The architecture is simple — but there is beauty in simplicity.

I hope you think so too!

Hugo Lu is the Founder of Orchestra, the Single Pane of Glass for AI and Data.

Been having a lot of fun recenty with Claude, and this weekend I thought I’d try something new.

What if we used Claude and our existing Data Stack to do all the discovery that we do with data quality testing and observability.

Rather than rely on a complex testing framework and separate tool, we can leverage Claude and documentation to build a simple system that surfaces observability.

The results were surprisingly good. Claude successfully inspected the underlying data and recommended appropriate tests with little input.

The full video is below, but I’ve written it up too. If you’re a Head of Data constantly getting bothered by people like me trying to sell you software, this one’s for you.

Methodology for creating your observability | Claude Tester

The methodology is actually really simple.

1. Create a Data Quality Node

A data quality node or Task is a job in your orchestrator that knows how to run a SQL Query with some parameters.

You could just trigger Lambdas.

Here is an example in Apache Airflow, which uses the SQLThreshldHoldCheck Task.

column_mapping = {
    “col_name”: {
        “null_check”: {”equal_to”: 0, “partition_clause”: “other_col LIKE ‘this’”},
        “min”: {
            “greater_than”: 5,
            “leq_to”: 10,
            “tolerance”: 0.2,
        },
        “max”: {”less_than”: 1000, “geq_to”: 10, “tolerance”: 0.01},
    }
}

2. Create your Claude Skill | Knowledge dump

A common objection I’ve recevied is:

“Oh but Claude knows nothing about Governance”

No you’re right, it doesn’t. So why don’t you write it down?

It turns out that when you write down

“The Data is more or less uniform and as you expect it, but so and so column has this one particular quirk that means when you [check daily data] you need to [add 5 minus 10 and times it by a million]” Claude is actually pretty good at interpreting this arbitrary logic.

What’s best is that by writing it down somewhere you actually document it — magic!

It could be something as simple as:

----- name:data-quality-test-writer ------
----- description: you are an agent designed to inspect data, work out errors in it, and write tests to catch errors 


1. You should expect to see credentials in your evnironment for <WAREHOUSE>
2. Find the tables, schema, and databases in the warehouse. For each table figure out what data is in there
(You can sample the data) and devise quality tests to assess the DQ
3. <YOUR CUSTOM GNARLY LOGIC HERE>
4. Below is an example of a pipeline you can PR into the repo for running the DQ Tests
5. When you PR it, create new branch and add relevant comments


```yml
version: v1
name: ‘Snowflake Timestamp Data Quality Tests #snowflake #dataquality’
pipeline:
  timestamp_not_null_tests:
    tasks:
      check_timestamp_not_null:
        integration: SNOWFLAKE
        integration_job: SNOWFLAKE_RUN_TEST
        parameters:
          statement: ‘select * from BIGQUERY_SAMPLE.${{ MATRIX.dq_inputs[’‘schema’‘]
            }}.${{ MATRIX.dq_inputs[’‘table’‘] }}

            where ${{ MATRIX.dq_inputs[’‘column’‘] }} is null

            limit 100’
          error_threshold_expression: ‘> 0’
          warn_threshold_expression: ‘> 0’
        depends_on: []
        name: Timestamp Not Null Check
        connection: ${{ ENV.SNOWFLAKE_CONNECTION }}
    depends_on: []
    name: Timestamp Not Null Tests
    matrix:
      inputs:
        dq_inputs:
        - schema: ORCHESTRA_METADATA
          table: PIPELINE_RUNS
          column: CREATED_AT
        - schema: ORCHESTRA_METADATA
          table: TASK_RUNS
          column: CREATED_AT
        - schema: PUBLIC
          table: ISSUES
          column: CREATED_AT
  timestamp_year_bounds_tests:
    tasks:
      check_timestamp_year_bounds:
        integration: SNOWFLAKE
        integration_job: SNOWFLAKE_RUN_TEST
        parameters:
          statement: ‘select * from BIGQUERY_SAMPLE.${{ MATRIX.dq_inputs[’‘schema’‘]
            }}.${{ MATRIX.dq_inputs[’‘table’‘] }}

            where year(${{ MATRIX.dq_inputs[’‘column’‘] }}) < 2000

            or year(${{ MATRIX.dq_inputs[’‘column’‘] }}) > 2030

            limit 100’
          error_threshold_expression: ‘> 0’
          warn_threshold_expression: ‘> 0’
        depends_on: []
        name: Timestamp Anomalous Year Check
        connection: ${{ ENV.SNOWFLAKE_CONNECTION }}
    depends_on:
    - timestamp_not_null_tests
    name: Timestamp Year Bounds Tests
    matrix:
      inputs:
        dq_inputs:
        - schema: ORCHESTRA_METADATA
          table: PIPELINE_RUNS
          column: CREATED_AT
        - schema: ORCHESTRA_METADATA
          table: TASK_RUNS
          column: CREATED_AT
        - schema: PUBLIC
          table: ISSUES
          column: CREATED_AT
schedule:
- name: Daily at 8am
  cron: 0 8 ? * MON-FRI *
  timezone: Europe/London
webhook:
  enabled: false
alerts:
- name: Failures
  statuses:
  - FAILED
  destinations:
  - integration: SLACK
    destination: alert-testing
```

3. Run your Claude Skill | Let the magic happen

You might want to play around with the Skill a bit first.

For example, the first time I ran it, all my tests were green.

I want to find errors! Not be the emperor with no clothes!

But more or less, you can run this and perhaps even run it in a staging evironment as part of your testing.

As part of my Claude Skill, it generated the yml included in the skill above.

I then Pr’d it and ran it and this happened.

Running Snowflake Data Quality Tests written by Claude in Orchestra

Pretty cool right?

Conclusion | Let Claude be your leverage

This is by far the biggest tip I can give to you — but it’s to start thinking of Claude not as the end but as the leverage.

We all have data architectures out there. How can we “sweat” these assets and make them even more powerful?

In this context, it may a case of getting Claude to run things in the orchestrator. It may be getting it to configure Snowflake Data Quality Checks or Databricks Quality Monitors. The point is, AI Agents remove the friction that once existed around this particular problem.

They also offer a fantastic way to finally capture the tribal knowledge people have about how the data is the way it is.

As I wrote in the Youtube Video:

Data Observability Tools always occupied a difficult space in the stack. Often thought of as “nice to have” and blindingly expensive, the great SAASPocalypse of 2026 (as well as tightening IT budgets globally) have Heads of Data considering what is critical and what is “nice to have”.

The most compelling reason was simply that many “data quality issues” are caused by things outside of a tool’s control. Poor processes and inputs at the source (human problems), failures in ingestion or orchestration code, or perhaps a simple lack of education for the end user all result in “data quality issues” — albeit ones that cannot be solved or even improved by ex-post testing.

Despite this, Data Observability Tools’ customers are staunch, often due to their success in “identifying” the issue. A company with thousands or hundreds of thousands of tables simply needs a consolidated view, a job that would otherwise take multiple engineers. I believe AI is the straw that breaks the camel’s back. AI makes it easier than ever to construct systems of data quality testing.

The problem has never been the infrastructure. Other orchestrators, including Orchestra, Airflow and others all offer ways to run data quality tests. The issue has been in finding the time to write these tests.

With an Observability Tool, Data Governance Professionals could go in, decide what tests need to be run, and press play. AI makes it easier than ever to do this. In fact, it makes it so easy that perhaps, the Data Governance Professional isn’t needed either.

To learn more about me and Orchestra, head to getorchestra dot io.